Blog

Big Data surrounds us all, in some shape or form. Typically Big Data (billions or trillions of vast and complex records) is so large, that it requires new and powerful computational resources to process and store. These gigantic sets of data can be analyzed to comprehend patterns, associations, trends, and statistics that help better understand user experience, human behavior, interactions, engagement, etc.

Big Data analysis, such as the services offered by our Function1 Operational Intelligence team, can be provided for a range of industries including but not limited to: financial, transportation, social media, healthcare, customer service, insurance, academia, government, and many more sectors. Indeed, Big Data and Big Data analysis surround us all.

So, how do we better recognize and analyze this data? We must make valuable proactive decisions to uncover the answers tucked away inside the mass of information that is Big Data.  We use products, such as Splunk, to help resolve and reveal the answers to this question. Any of our Function1 Splunk Consultants can tell you that trending over more data will give you better pattern recognition and anomaly detection; however, what can we do when there are 1 or more terabytes of incoming data everyday, and there is a need to keep this data searchable for a year, 3 years, or even more than 7 years? 

The answer is Red Hat Storage. Red Hat Storage Server is an open, software-defined storage platform that provides cost-effective storage solutions for extremely large, historical data sets. 

Let’s analyze an simple example - if you have some knowledge of Big Data and the ongoing need for storage solutions. Let’s say we are ingesting 2 TB of data daily, and need to keep that data set searchable for 3 years. Even at optimal compression, you’ll need 1.5 PB (or 1,533 TB). How do you store this data in an affordable way?

Red Hat Storage Server is the solution to this problem. Red Hat Storage meets the demand by providing cost-effective storage for extremely rare, historical data sets for enterprise class Splunk deployments. Red Hat maintains storage performance, capacity, and availability to meet the most demanding enterprise storage requirements. As a hybrid storage model, Red Hat Storage suggests to customers to leverage this tool for their “cold” Splunk data (often, the less actively searched data that is much larger in volume), while leaving “hot/warm” data on Direct Attached Storage.

Click here to download and explore Function1’s White Paper about architectures and operational scenarios to validate the integration between Splunk and Red Hat Storage. This testing and validation was conducted in a lab environment. The following scenarios were tested and outcomes were recorded:

• Scenario 1: Searching availability during system failure;

• Scenario 2: Uninterrupted scaling of Splunk indexers;

• Scenario 3: Configuring storage to be non-disruptive to application workload;

• Scenario 4: Splunk bucket management;

• Scenario 5: Achieving non-disruptive storage expansion; and

• Scenario 6: Warm and cold data searching availability throughout different stages.

Through these six scenarios, Red Hat Storage Server proved to provide: 1) uninterrupted searching and indexing, 2) continuous scaling of Splunk indexers through simulated maintenance or unexpected restart of service, 3) fully replicating prolonged outages through the self-heal process, 4) holding and managing upwards of dozens of buckets scheduled for cold migration, without any performance impact, 5) remaining fully functional through administrative changes and automatically replicating to resized bricks with zero data lost both during and after the testing, and 6) remaining fully functional with uninterrupted searching throughout the various manipulated stages.

Revisiting our earlier question, how do we store these massive amounts of data without compromising on scale or performance? Red Hat Storage Server is the answer, proven to and capable of performing the role as a hybrid storage solution for enterprise class Splunk deployments. Red Hat Storage Server continues to be a cost-effective solution to help manage and maintain the tremendous increase in large-scale data growth, surpassing the most demanding enterprise storage requirements.

Join us at the upcoming Red Hat Summit with Function1’s Director of Data Security & our Vice President of Business Development and WW Sales.