Today I looked at a few course materials available on mecr.org and these are available for reading. One of the topics I looked at was evaluating complexity in terms of big O, Omega and Theta.  I want to briefly mention them here.

If we take a few methods f and g  where f is O(g(n)) we can show there are non-unique values of coefficient c and n that satisfy this i.e c > 1 n > 1 and there are different forms of polynomials that can satisfy the relationship for f and g.
Typically in calculating complexity, usually only the higher term matters. This becomes important in polynomials as it simplifies the discussion on complexity.
While O is applicable to a single algorithm. Omega is applicable to a class of algorithms. For example, we say that Omega for sorting algorithms is nlogn. This we see from different sorting algorithms having a complexity of O(nlogn).
Theta on the other hand is used for a range. For example if f(n) lies between c1.g(n) and c2.g(n) i.e
c1.g(n) < f(n) < c2.g(n) then we say f(n) is theta for g(n) for all n0, c1, c2 such that n > n0.
These two functions are generally of the same order.
big O, Omega, Theta are all related.
For example, f(n) is O(g(n)) and g(n) is Omega(f(n)). Then we can take two coefficients c and 1/c for which we can say f(n) <= c.g(n) and 1/c .f(n) <= g(n)
and consecutively we can say f(n) is Theta(g(n). In other words, f(n) is  O(g(n)) and also f(n) is Omega(g(n)).
There is also an interesting observation that
if f1 is O(g1(n))
and f2 is O(g2(n))
the f1 + f2 is O( max(g1, g2))
This is true even if we have more than two such functions. Its easier to visualize these as applicable to a  sequence of code blocks in a program where the overall program complexity is the max of any one of the blocks of code.
Also, we can show that with a choice of coefficients for the functions above, where say n3 is max of n1, n2 and c3 is max of c1, c2, then f1 + f2 = c3g1(n) + c3g2(n) = 2.c3.(max (g1,g2)) and that leads to
f1 + f2 is O(max(g1, g2))

In today's post I want to cover a topic on writing fast low level logs.  There are many logging providers and I've covered at different levels earlier. Some examples included log4net and log4cpp. Kernel mode tracing is often considered as an example of low level logging. Database systems also have their own system of logging.
There are a few salient features of most logging methods.
First is they enable formatting to the text that needs to be logged. These are variable length arguments and allow the different specifiers to be enabled
Second is they enable different levels of logging. By this we mean we control how much or how little of the logs to be written. For the components that provide the log strings, they could continue to log at different levels and the end user could decide how much or how little of the logs to be seen. This is a switch independent of all the components and each log provider is expected to honor these levels.
Components are expected to separate their logging with labels or strings that they identify by initializing or teardown.
Another feature is that these logging are based on file appenders and the logs are written at the end of the file.
Aside from the debug levels, there may be other metadata for the log.
But if we take the implementation of a logger, we neither use an event loop or a overlapped IO.
In principle, both could be used. However, neither are.
A logger has many clients that send messages. The messages are serialized onto the end of the file.
The offset of the file is incremented in a thread safe manner as opposed to locking the entire data.
or putting the entire log entry addition code within critical section. The process of writing the log is that of filling out buffers which are then flushed to disks. Typically this buffer is left to file system instead of maintaining a separate one. Some applications choose to implement a round robin circular linked list of buffers which are then flushed to disk.
No matter what the buffering is there can be many producers and one consumer and serialized queue. That is why this sounds similar to an overlapped IO.

In today's blog post we review CLI commands for Splunk. Again the command line option is complimentary to User Interface and works well for management operations and scriptability.
The command line option to enable Splunk to startup in debug mode is
splunk start splunkd --debug;
 if we were to clean all data , we could say
splunk clean all
or splunk clean [option] where eventdata, globaldata, userdata  or inputdata can be specified in option
In addition we can disable one or more components such as
splunk disable app, boot-start, deploy-client, deploy-server, dist-search, index, listen, local-index, perfmon, web-server, web-ssi.
 and followed with a way to toggle them back on with
splunk enable app, boot-start, deploy-client, deploy-server, dist-search, index, listen, local-index, perfmon web-server, web-ssi.
splunk display command will display the status for each of these.
 splunk list is different from splunk display. The list command lists all configurations and setting or the collections.The display option is only the state of that feature.
The CLI commands provide options for working with data.  These include:
splunk import userdata
and splunk export userdata, eventdata
If we made a change to the filter the data, such as with a conf file we can granularly enable or disable them such as with the following:
splunk reload ad, auth, deploy-server, index, monitor, registry-script, tcp, udp, perfmon, wmi
For most of these components when we enable or disable them, we can check the logs to set that they have indeed been enabled or disabled.
The CLI commands provide a way to work with the logs. we do this with
splunk find log
When the commands are applied to many different machines for a cluster, the CLI provides a way to do this.
For example, we can type splunk apply cluster-bundle to apply (make active) a cluster bundle to all the peers in the cluster. To check the status on all the peers, the splunk show cluster-bundle-status command can be used at the master.  For silent apply, we can say ./splunk apply cluster-bundle --skip-validation --answer-yes. The CLI list command also provides other cluster related options such as splunk list cluster-config, cluster-generation, cluster-peers and cluster-buckets.  The splunk rtsearch and splunk search commands are also available to search. Note that the splunk rt-search command is used for searching events before they are indexed and to preview reports as the events stream in. The command arguments are similar between the traditional search and the rt-search commands. App is used to specify an app context to run the search, batch is used to specify handle updates in preview mode, detach triggers an asynchronous search and displays the job id and ttl for the search,  header indicates whether to display a header in the table output mode, max_time indicates the length of time in seconds that a search job runs before it is finalized, maxout indicates the maximum number of events to return or send to stdout, output indicates how to display the job such as rawdata, table, csv and auto, timeout denotes the length of time in seconds that a search job is allowed to live after running. Defaults to 0 which means the job is canceled immediately after it is run. and wrap that indicates whether to wrap lines that exceed terminal width.
The splunk commands provide output that can be used to view the success or failure of your command. Typically an error message is provided when it fails.

 

The Event loop discussed in the previous post runs continuously and uses a Time-heap data structure which we discuss today. we maintain the time-outs on a heap, the pending heap. When we want to run them, we put them on the doubly linked list mentioned earlier.  The timeouts are in a heap because it is an efficient way to organize them by order of expiring timeouts. In this implementation I'm looking at, the left and the right children point to similar events with timeouts greater than the parent.  The left differs from the right only in the timeoutcount. if the timeout count is even, it is steered to the right.
whenever new elements are inserted they are bubbled up and down the tree as necessary. After every add or delete, the tree is fixed up.
Timeouts have a link pointer that indicates which list it belongs to. Note that when the timeouts are on the list we reuse the storage for the left and right pointers to be the forward and backward pointers in the doubly linked list
As a test to the Timeout heap, we could make sure that the events are added once and in the correct position on the pending queue and that they are removed from the pending when they are on the expired list to be run and that they are added back to the pending heap and in the correct position for re-triggering. Their position as left or right should remain unchanged if no new nodes were added or deleted.
By doing the test on a controlled heap we know that the events are processed in sequence. In the real world, the events should be guaranteed to be executed if their timeout expired but their sequence could be arbitrary. The heaps are supposed to be short and therefore none of them is expected to be infinite.

I found an implementation of an event loop and would like to compare it with an overlapped IO. Both code can work on Windows but we will leave the nuances of using it on any one platform. The semantics of an Event Loop is that some events are added and at the end of a timeout these  events are triggered. The timeout is determined if the events needs to be triggered the next time and if so the timeslice to wait, otherwise zero. Several helper threads can wait on  a set of poll able resources. The threads are kicked off by the loop, then waited on and then shut down.  The wait is for the timeout determined.  If this is successful, we will then recalibrate the before, timeout and after for the next round.
If the wait set was not successful, then it was probably because events got added or removed, so we check all the states and start all over.
There are a set of three lists maintained : a doubly linked list that is not full , a doubly linked list that is full, and a doubly linked list that is empty. The waiter thread moves between these queues.
In an overlapped IO this is very different. Where the events are serviced by different producers and consumers and can even associate it with multiple poll able resources.
There is no prediction to which order the events will get triggered in either case but one proceed in terms of beats and another proceeds in terms of the items in the IO.

In the post on using the SQL Server service broker as a modular input for Splunk, we introduced a technique but we now describe the complete solution. We mentioned that we could read messages from the Broker by opening a SqlConnection and executing a SQL statement. For every such message received we can create a Splunk modular input event wrapper and send off the data to Splunk.
The program implements the Splunk script object and implements the methods that are required to take the configuration from the user, apply the configuration and extract events. These events are extracted from the message broker as mentioned above.
The sampling rate is determined by the number of messages We fork a thread to process the messages for the designated queue. The config determines the queue names, the sampling intervals etc. The default sampling interval is zero.
We invoke the CLI command object from the SDK. Specifically, we say Command.Splunk("Search").
And then we add the parameter we want to search with.  we can check the cli.Opts to see if our search  command and parameter were added.  After defining the command object, we then create a job object to invoke it. We do it with the following:
var service = Service.Connect(cli.Opts)
var jobs = service.GetJobs();
var job = jobs.Create((string)cli.Opts["search"]);
We wait till the job is done. This we can check with the done flag on the job.
We retrieve the results of the jobs with the job.getResults command which returns a stream.  We can then open a ResultsReaderXml on this stream to retrieve the events.

Today we discuss more on modular input. In this case, we talk about the properties we can set on the modular input data structure. One of the properties is the index. This talks about which index the events should goto. There us a default index but its preferable to store out events by their own index. The second property is the stanza name a unique source type for these kind of events. In fact the stanza is user specified and the events are often retrieved based on these literals.

The idea behind modular inputs seems to be to envelop the events to be differentiated from others. These help even before  parsing and indexing have taken place.
When we use the script object with the input we are able to govern the flow of these events. We increase it slowly from a trickle to a deluge if we wanted by varying the polling interval or the sampling rate

In addition to differentiating and regulating the flow, the modular inut and scripts can filter On raw data before they are sent over. For example the inbound and outbound messages may not only be differentiated but one selected over another