Blog

Data over here, Data over there, Data everywhere...

We live in an an age where there is probably more data compiled about a newborn child in it's first 1yr of life than was compiled about their great grand parent in their entire lives! Imagine collecting health data for the nearly 4.25M kids that are born in the US a year and then doing absolutely nothing with it. No studies on mortality rates, no studies on child development, no alerts on outbreaks effecting this new population...Nada. Preposterous? Yup. Ludicrous? U huh. Down right dumb? "It is decidedly so". Well you'd be surprised how many private companies and public organizations are sitting on a gold mine of data and doing absolutely nothing with it. It's time to put down that nifty Magic 8-Ball you're using to run your business and start mining the wealth of information your organization already possesses.

Understand your business...

Don't get me wrong, many companies are in fact leveraging their large reservoirs of business data in the form of a Business Intelligence (BI) function, Marketing Database or Data Warehouse (DW). These vast data stores generally hold human generated information including demographic data points on customers, things they've purchased from your organization, which marketing campaigns they've responded to, customer service touch points including information such as calls into the service center as well as complaints, whether or not a customer has canceled or renewed their subscription to your service etc. It could and probably does include information on customer satisfaction and other self reported feedback from the customer base. This sort of data store is usually outfitted with fancy report generating facilities and allows the user to slice and dice along the various dimensions of the DW and look at trends among their customer segments. This allows companies to "listen" to the individuals who fall within very particular groups or marketing segments. Unfortunately, DWs are generally big bulky systems and the processes that feed these systems often times lag well behind emerging trends at the edge of the the sonic boom. Thus DWs are quite useful for long term trends but fail to capture the day to week level trends which allow an organization to be agile. It's quite obvious to see the Business Manager's love for what the DW can provide and the technologist's disdain for it a well. Generally, any additional dimensions added to such a warehouse include lots of planning, manpower and expensive ETL technologies like IBM's Datastage to prep and get the right data in the right format to the target system. The other problem experienced by such systems relates to the relational database at it's heart and the big data problem associated with it.

Understand how machines impact your business...

The other broad category of data that your company owns is machine generated data (MGD). This is comprised of all data that is automatically generated without direct human action. Machine generated data includes web server logs, application logs, network appliance data, packet data, call records, OS event data, and the list goes on and on. In this ever increasingly technological age, it easy to see that in your daily life you generate reams of data. Emerging technologies adept at handling the big data problem are now becoming more and more accessible to the average technologist. Companies are now beginning to incorporate the use of this machine generated data into their day to day operations and a product like Splunk is at the heart of this trend. "Operational Intelligence" is best described as discipline which utilizes organizational knowledge and machine generated data to quickly understand the cause and nature of business and system events, quickly identify trends within and the ability of it's practitioners to take corrective action in near real time to effect a positive outcome. Yup I just pulled that one outta thin air, sounds about right though? Unlike other log analysis tools Splunk provides it users with the ability to quickly and easily digest all forms of MGD without having to labor over ETL processes. It relates all of these sources through the one thing common to all MGD which is a time-stamp (yes even in European format). Using a simple yet powerful search interface the product enables the user to quickly perform complex event correlations. If you're a systems integrator then you've probably dreamed of a system like this. As someone who's been waist deep in WCI, I know what it means to "troubleshoot" a system or component outage to determine root cause. "Drilling down" means logging into a remote machine and opening up large log files in a simple text editor (or hundreds of small ones...which ever makes you want to jump off a bridge faster) to then have to remember a log entry and log into another machine and open another log file to then search for the period of time the event occurred and so on and so on. After a while you think to yourself, "There has got to be a better way". Having real-time feeds of log data indexed and searchable all in one place translates into is less down time and more reliable and more secure systems. The beauty of the system is it's collaborative nature for knowledge building. As a user you create relevant queries in your user group (Security Dept., Web Analytics Group, etc.), annotate parts of logs via regular expressions and save them for later use by others. For example using Splunk's transactional aggregation operator you can quickly create queries to sift through application server logs aggregating sessions by ip to determine if session spoofing is occurring. You can then save these and create scheduled runs with email alerts. Splunk has plenty of other applications to security, compliance and web analytics as well.

Survey says?

So what was your answer to the question in the title? If you responded "yes", then congratulations your company certainly has a competitive advantage in this ever challenging business climate. If you answered "no" or even "I don't know, I'll get back to you", then it's time to put down that Magic 8-Ball and to start asking yourself why that is.