Blog

Splunk SDK for Python: Getting Data In

Data is a pivotal part of a Splunk Enterprise deployment.  Every configuration and enhancement we make is centered on a particular dataset. As a result, Splunk provides different options for getting data into Splunk Enterprise in order to turn that data into decision-making information. The most common ways of getting data into Splunk are via UF forwarders, syslog, scripted inputs, and modular inputs.  For this post, I’m going to focus on getting data from a remote interface into Splunk via HTTP utilizing Splunk SDK for Python. This post assumes some familiarity with Python.

Splunk SDK for Python

Splunk SDK for Python allows developers to interact with Splunk utilizing python.  This allows Splunk Enterprise customers to create modular inputs for Splunk apps, develop applications that interact with Splunk, and integrate Splunk with currently existing applications in your environment. With the modules provided by the SDK for Python you can run searches, create modular inputs, manage and create indexes, create your own search commands, display search results, edit and create roles, and index data from remote interfaces all done with very few lines of code (This is demonstrated below).  To learn more about the Splunk SDK modules and what you can do with the Splunk SDK for Python please see: http://dev.splunk.com/python

Splunk SDK for Example: Getting data in from remote interfaces.

For this tutorial, I decided to utilize data from Locu since it has a very friendly and easy-to-use API that only requires developers to signup for a developers account to obtain an API key to access their data. Locu maintains local business data and allows those businesses to be found by customers and potential customers everywhere. Here, I’m searching for data on businesses in the New York and Philadelphia areas. See the link at the end of this post for more information about creating a Locu developers account.

Getting the data:

Before we can send data into Splunk, we must first obtain the data from locu by querying the locu via a GET request. Here I’m using URLLIB2 without proxy credentials. If http requests require proxy authorization and authentication, add your proxy credentials to the proxy handler or else you may encounter a connection refused.

The results of calling this function should be a JSON formatted data set that includes 25 search results:

Parsing the data received:

We could very well send the JSON formatted search results shown above to Splunk for parsing and ingestion since Splunk has the ability to handle JSON formatted data but per Splunk best practices, you should only send data that you plan to utilize and is of value to Splunk to save on licensing and performance. So, for this dataset, I created another small function to parse the JSON and return only the data that is needed for routing to Splunk. As you can see below, I’m formatting the key: values to utilize equal signs (=) since I’m have KV_MODE set to auto in props.conf. Note:  KV_MODE  = auto uses equal signs for search time field extractions.

The results of calling this function and printing it would be:

Connecting to Splunk:

Now that we have our formatted strings, we are going to utilize the Splunk SDK for Python’s connect module to connect to my Splunk instance. This requires our host, management port, username and password. Please note: I defined my credentials as global variables as noted in the Getting the data section.

Upon successful connection to Splunk Enterprise , a service object should be returned .

Pushing data to an index

After a successful connection to Splunk Enterprise, we are able to send our data to the locu index with the sourcetype of locu_api with the following function:

The result of calling this function with a successful output should be indexed data:

As for our fields:

Where to from here:

If you would like to create your own applications and inputs that interact with Splunk, you should download the Splunk SDK for Python from: http://dev.splunk.com/view/python-sdk/SP-CAAAEE4">http://dev.splunk.com/view/python-sdk/SP-CAAAEE4

The requirements are:

• Python 2.6 or later (Python 3 is not supported at this time)
• Splunk Enterprise

For more information on Locu and developer access to their API: https://dev.locu.com/documentation/">https://dev.locu.com/documentation/

The source code for this tutorial can be found here : https://github.com/ashnet16/Splunk_Mini_Scripts/blob/master/locu_api.py

Happy Splunking!