Ceilometer & Splunk Integration via Python SDK:

Ceilometer is OpenStack’s telemetry. It’s source code is available at https://github.com/openstack/ceilometer. It collects all kinds of measurements and metering information such that no two agents need to be written to collect the same data. It’s primary consumer is the billing system for which it acts as a unique and central point of contact  to acquire all of the information collected across all OpenStack core components. The project strived to be efficient in terms of CPU and network costs. It supports both the push notifications from the existing services and the pull model by polling the infrastructure. Deployers can configure the type of data collected. Publishers are available as a Python implementation. A REST API is also available to expose the data collected by the metering system.

As an example of the events from Ceilometer, we review the PAAS event format. There are a number of PAAS services that have metering payloads. Instead of having each service define its own, the Ceilometer provides a minimum data set as described here.

Splunk is an IT tool that enable searching a needle in a haystack of information. Specifically, it forwards information collected from various sources (via its modular inputs) and indexes them in time series events which can then be searched and analyzed using a wide variety of search operators and charting tools. Splunk also comes with its own SDK and one that is available in Python.

Splunk SDK supports a variety of common operations with Splunk through its programmable APIs. In particular it supports a UDP modular input that can suit Ceilometer.

The host and the port binds the publisher to the subscriber. In this case, we can configure the host and UDP port for ceilometer to publish to Splunk.

#!/usr/bin/env python
#
# Copyright 2013 Ravi Rajamani.
import sys
from splunklib.modularinput import *
class CeilometerIntegrationScript(Script):
    def get_scheme(self):
        scheme = Scheme("Ceilometer Telemetry")
        scheme.description = "Streams events from Ceilometer."
        scheme.use_external_validation = True
        scheme.use_single_instance = True
        scheme.validation = ""
        ceilometer_argument = Argument("data_from_ceilometer")
        ceilometer_argument.data_type = Argument.data_type_string
        ceilometer_argument.description = "Telemetry data from Ceilometer to be produced by this input."
        ceilometer_argument.required_on_create = True
        return scheme
    def validate_input(self, validation_definition):
        data = str(validation_definition.parameters["data_from_ceilometer"])
        if not data:
            raise ValueError("Ceilometer data could not be read.")
    def stream_events(self, inputs, ew):
        """This function handles all the action: splunk calls this modular input
        without arguments, streams XML describing the inputs to stdin, and waits
        :param inputs: an InputDefinition object
        :param ew: an EventWriter object
        """
        for input_name, input_item in inputs.inputs.iteritems():
            # Create an Event object, and set its data fields
            event = Event()
            event.stanza = input_name
            event.data = "number=\"%s\"" % str(input_item["data_from_ceilometer"])
            # Tell the EventWriter to write this event
            ew.write_event(event)
if __name__ == "__main__":
    sys.exit(CeilometerIntegrationScript().run(sys.argv))