Wednesday, July 30, 2014

Today we are going to continue the discussion on the Splunk Forwarder port to SplunkLite.Net. Here we cover the event and Metadata. If we look at an input processor in Splunk input pipeline, we see a common pattern such as :
public class StdinInputProcessor :  InputProcessor{

    public StdinInputProcessor();
    public ~StdinInputProcessor();
   
    public virtual void init( XmlNode pluginInstanceConfig);
   
    public virtual EProcessorReturn execute(CowPipelineData pData);  
};

essentially the only two operations it is responsible for is initialization and execution.
The execute method merely copies the data from external to the event writer and sets metadata such as sourcekey etc.
Occassionally a producer consumer queue and a shutdown thread is utilized when there are more than one channel needs to be served and decoupling the producer and consumer helps. 

Event conversion happens after the data has been copied over. Few fields are populated to describe the event including source and destination metadata such as host, timestamp, sourcetype, index, etc.

CowPipelineData is a proxy object that wraps the PipelineData which manages the data that is passed around in the pipeline which maintains a map of predefined keys. PipelineData processes headers and does serialization as well. CowPipelineData provides threadsafe copy-on-write semantics. Each thread has an exclusive CowPipelineData although they may share the same PipelineData in a threadsafe manner.

No comments:

Post a Comment