Blog

Splunk is a journey. Whether you are a newbie playing around with Splunk on your local machine or have a multi-instance distributed Splunk deployment, your knowledge of Splunk is always evolving. Typically, the more you know about Splunk, the more you want to do with Splunk. Often, proof of concepts (POCs) turn into production environments and soon enough you’re increasing your license and looking into new architecture. Congratulations, you have become a full on Splunk ninja! Now, how do we migrate your current Splunk deployment to ‘beefier’ hardware, minimizing downtime, and maximizing value? There are a few things to keep in mind:

·       Are you expanding your architecture or moving to completely new hardware?

o   As expected, there is greater complexity involved when migrating to a completely new set of hardware versus adding a few Indexers into your environment. Ensure you read over Splunk documentation for the best course of action and through the examples below.

·       Is the new hardware in the same data center or a new data center?

o   The Splunk Search Head queries Search Peers (Indexers) for data. Your search latency may be adversely affected if your Search Head and/or Indexers reside in a separate data center from one another. This can also affect traffic flowing through your internal networks.

·       Are you using a deployment server to pass out configuration files? Or are you using a 3^rd party configuration tool like Puppet?

o   The great thing about utilizing Splunk’s Deployment Server is that you have one central location of what configuration files are being passed to the different Splunk instances. It also makes migrating Splunk instances much easier because you can pass out multiple configurations with one deploymentclient.conf file. By placing a deploymentclient.conf file within /etc/apps or /etc/system/local, you direct Splunk to ‘call home’ and receive the new apps. If you are migrating using a 3^rd party tool like Puppet, you may have several iterations to go through when updating configuration files.

·       What type of OS are you running Splunk on?

o   It is very important to refer to Splunk documentation for any migrations or upgrades pertinent to your Operating System, especially for you Windows users.  Search Head Clustering, for example, cannot be deployed on Windows machines yet for Splunk 6.2. Make sure you are aware of any limitations you may have on an OS level.

Now, let’s examine a few examples of migrations performed over the past few months and each company’s course of action. For privacy purposes, the names of the companies have been changed.

Scenario 1: Dasher Company (Expansion of Environment)

Dasher Company had a single Search Head/Indexer Splunk instance. They were looking to expand their license and hardware simultaneously as they moved to monitor their entire production environment. We had 2 new servers to play with during this migration, we took the following approach:

Dasher Company’s Original Architecture: 1 Dual Search Head/Indexer

Dasher Company’s New Architecture: Reconfigure the Dual Search Head/Indexer as a Sole Indexer, Configure 1 Indexer with a new machine, Configure 1 Search Head/ Deployment Server with the other server.

1.     Built out the new 6.2 environment on the new servers (2 Indexers and 1 Search Head/Deployment Server).

2.     Copied over App configurations pertinent to the Search Head from Dasher’s original instance to new search head instance.

3.     Configured Dasher’s Original Indexer as a search peer of the new Splunk Instance (Documentation Link).

4.     Modified configurations on the Deployment Server with the appropriate configuration packages for the Forwarders.

5.     Placed deploymentclient.conf files on Dasher’s forwarders to point to the Deployment Server. As Dasher’s forwarders ‘called home,’ they received their associated App packages. 

6.     Ensured load balancing was occurring between new Indexers as data was ingested into Splunk. (You can check this by looking at your licensing report on the License Master.)

Scenario 2: Cupid Company (Acquisition of New Company)

Cupid Company had acquired a company with an existing Splunk deployment (we’ll call the newly acquired company Comet.) Cupid Company wanted to build out their own deployment while migrating the newly acquired company’s deployment into their own.  Eventually the goal was to decommission Comet’s environment, as all data would be migrated into Cupid’s. We took the following course of action:

Cupid Company’s Architecture:  1 Indexer, 1 Search Head/Deployment Server/License Master

Comet Company’s Architecture:  1 Sole Splunk Instance (Search Head/Indexer)

1.     Set-up the new Splunk distributed environment for Cupid Company with 1 Indexer, 1 Search Head/Deployment Server.

2.     Configured Comets’s Indexer as a search peer of Cupid’s Splunk Instance (Documentation Link).

3.     Inform all users that they need to start using Cupid’s Search Head.

4.     Copy over any App configurations from Comet’s instance to new Cupid instance.

5.     Modify configurations on Cupid’s Deployment Server with the appropriate configuration packages for Comet’s Forwarders.

6.     Place deploymentclient.conf file on Comet’s forwarders to point to Cupid’s Deployment Server. As Comet’s forwarders ‘call home,’ they will receive their associated App packages.  

7.     Upon completion of #6, move all data buckets from Comet’s Indexer to Cupid’s Indexer.

8.     Validate that all data from Comet’s Instance is searchable in Cupid’s Instance.

9.     Decommission Comet’s Environment.

Scenario 3: Prancer Company (Expansion of Environment with Puppet to Deployment Server Component)

Like discussed above, Prancer Company had so much success with Splunk that their proof of concept (POC) turned into a full on production environment. They were experiencing some performance issues and wanted to move to ‘beefier’ hardware to onboard some new data and explore other use cases. We completed this task by following the below steps:

Prancer Company’s Original Architecture: 1 Dual Search Head/Indexer (MI Data Center), 1 Indexer (VA Data Center)

Prancer Company’s New Architecture:  Reconfigure the 1 Dual Search Head/Indexer as a Sole Search Head (No Indexing), Configure 2 New Indexers, Decommission the 1 Indexer in the VA Data Center once Retention Period is Met (as this is a virtual machine), Configure 1 Deployment Server (configurations were originally being controlled by Puppet)

1.     Built out the new 6.2 environment on the new servers (2 Indexers and 1 Deployment Server).

2.     Configured the Deployment Server with all configurations for Indexers, Forwarders (this took a bit of time as we had to ensure serverclasses were created for all the forwarders that were being controlled through puppet).

3.     Change the Forwarders to point to 6.2 indexers by using Puppet to place a deploymentclient.conf on each forwarder. We will test out a few "low priority" forwarders as our initial list to redirect.

4.     Ensure data is forwarding to the new Splunk instance and everything is functioning properly.

5.     Decommission Prancer’s VA Data Center Indexer once the retention period is met.

As you embark on migrating your Splunk deployment, please feel free to reach out to info@function1.com with any questions or concerns. Happy Splunking!