Using HTTP Event Collector and Splunk Logging Driver to Gain Visibility into Your Docker Applications

By: Jennifer February 09, 2017

Splunk is always looking for ways to make it easy for its users to collect machine generated logs to gain real-time visibility into their infrastructure and IT Operations. The traditional way of collecting logs is through the Splunk universal forwarder, which is installed at the data source to route events to the Splunk. Splunk introduced a new way to collect logs using the HTTP Event Collector (HEC), a token-based JSON API that allows you to send application events to your Splunk deployment using HTTP or Secure HTTP (HTTPS). HEC was mainly created for developers in mind as a way to send application logs to Splunk without using a universal forwarder.

In this post I will explain how to set up HEC and the Splunk Logging Driver, which is a new driver available in Docker that works alongside HEC to route events to Splunk. The scenario involves collecting logs from services running in Docker containers.

Setting Up HTTP Event Collector in Splunk

The first thing is to enable HEC in Splunk as it is not enabled by default. To do this open Splunk's Web UI and navigate to Settings - > Data Inputs -> HTTP Event Collector and click Global Settings at the top right corner. The screen below should pop up.

For All Tokens, select Enabled
Default Index drop-down allows you to select an index for all HEC tokens
To set the default forwarding output group for all HEC tokens, choose an output group from the Default Output Group drop-down.
If you wish to use deployment server to manage HEC tokens configurations, click Use Deployment Server.
For secure HEC communications, click the Enable SSL. This allows you to communicate via HTTPS instead of HTTP.
You can specify the port number that HEC listens on, it is 8088 by default
Click Save when you are done.
HEC endpoint is now enabled to listen for data on select port!

Since HEC uses token-based authentication, you are required to setup at least one token. To generate a token navigate to Settings - > Data Inputs -> HTTP Event Collector and click New Token on the top right of the page. Below wizard to create a new input/tokxtcaen will appear

Enter a descriptive Name for the new token, Source name override, and Description
Click Next when you are done

On the next page, you can specify the source type and index for the token/input.

HEC will automatically set the source type if you don't specify one.
Assign your token to an index. This is where your data will be stored for the input.
Click Review

You now have a new token! Take note of the token value that was generated as you will need this for your Docker configurations.

Setting up Splunk Logging Driver in Docker

First, in order to use the Splunk Logging Driver feature, ensure you have Docker version 1.10 or higher
Next, there are few config items you will have to set (example below)

Example:

[Service]

ExecStart=

ExecStart=/usr/bin/dockerd --log-driver=splunk --log-opt

splunk-token=HTTPEVENTCOLLECTORTOKEN --log-opt

splunk-url=https://your_splunk_instance:8088 --log-opt

splunk-token - insert the token you generated in Splunk here after the "=" sign
splunk-url - specify the Path to your Splunk instance
With these few configurations you are now able to send events from Docker to your Splunk instance
Here is a link that provides a list of all the available options in Splunk Logging Driver : https://docs.docker.com/engine/admin/logging/splunk/

Best Practices for Creating Tokens and Assigning Indexes

I recommended that you generate one token for each application/service in Docker that is routing data to Splunk. Setting it up this way gives you a lot of control in terms of managing your data in Splunk. If you ever need to send data to Splunk from a new application or service in Docker, you can simply create a new token and then configure the sourcetype and index on the HEC. Creating tokens this way also allows you to quickly identify and trouble shoot issues with your inputs if they ever come up.

In terms of organizing data into indexes, separate tokens can be configured to send data to the same index if it makes sense to. So if you had several web application services running, each with their own HEC token, all the HEC inputs could all be configured to send their data to a "web_app" index. Configuring the data inputs in HEC allows for ease of configuration at the data source and flexibility in organizing data in Splunk. Read here for Splunk's best practices for creating custom indexes.

Thanks for reading and happy Splunking!

Tags: Splunk, event collector, Logging, gain visibility, docker, set up

Blog