Blog

Recently one of our clients had an outdated version of Splunk and needed to upgrade to a newer version of Splunk Enterprise that incorporated additional features and security updates. This of course is a very important procedure that many Splunk customers will have to learn over their time using Splunk. In this blog I will outline the process used for upgrading distributed deployments of Splunk. 

• You can download the latest version of Splunk here:  http://www.splunk.com/download
• You can download previous releases here:  http://www.splunk.com/page/previous_releases

• Before you start your upgrade you want to check all apps for functionality with the version of Splunk you’re upgrading to.
• Next you’ll want to back up your existing environment. There are two steps to this part.
  • First you’ll want to backup all configurations under /opt/splunk/etc.  
  • Next you’ll want to ensure that your indexed data is backed up (by default the data is stored under /opt/splunk/var/lib/splunk, but this may have been customized in your deployment).

• The basic Splunk upgrade process involves the following:
  • Download the correct Splunk package for your OS.
  • Stop the current Splunk process on the instance that you wish to upgrade. 
  • Install the new Splunk package.
  • Start the Splunk process, review the list of detected changes and accept the license.
  • Test and verify the upgrade.  

Simple so far, right? The challenge at this client site was upgrading a clustered with the latest version of Splunk, while maintaining the shortest possible maintenance window. This meant zero mistakes could be made.

• To upgrade a cluster you first should disable the Deployment Server so that it does not try to update any client instances.
• Next, in this order, stop the Cluster Master, all the Indexers (peer nodes), and all the Search Heads.
• Then upgrade the Cluster Master by installing the latest Splunk package.
• Start the Cluster Master and enable maintenance mode.
• Next upgrade each Indexer, one by one.
• After the Indexers have been upgraded, upgrade all of the Search Heads.
• Now after all the Indexers and Search Heads have been upgraded start up Splunk on each of those components.
• Next start up the Deployment Server.
• Finally disable maintenance mode on the Cluster Master and confirm that it has left maintenance mode.  You can check on the Cluster Master dashboard or via CLI command.
• After upgrading you’ll want to fully test the environment. This includes testing out apps, running searches, and checking if dashboards are populating correctly.