In today's post we will cover another chapter in Exploring Splunk book. This chapter is on enriching data. We can use command like top and stats to explore the data. We can also add spark lines which are small line graphs to the data so that data patterns can be quickly and easily visualized.
With Splunk it is easy to exclude data that has already been seen. We do this with tagging. This helps detecting interesting events from noise.
When we have identified the fields and explored the data, the next step is to categorize and report the data
Different event types can be created to categorize the data. There are only two rules to keep in mind with event types. There are no pipes to be used with event type declaration and there cannot be nested searches aka subsearches to create event types. For example status = 2* to define success cases and status = 4* for client_errors.
More specific event types can be built on type of more general event types. For example web_error can include both client_errors and server_errors. The granularity of event types is left to user discretion since the results matter to the user.
Event types can also have tags. A more descriptive tag about the errors enhances the event type.
As an example, user_impact event tag can be used to report on the events separately.
Together event types and tags allow data categorization and reporting for voluminous machine data.Refining this model is usually an iterative effort.  We could start with a few useful fields and then expand the search. All the while, this adds more input to Splunk to organize and label the data.
We mentioned visualizing data with sparklines. We can also visualize data with charts and graphs. This is done from the create report tab of the search page.
For example, we can search with a query such as sourcetype=access* status=404 | stats count by category_id and then create a pie chart on the results. Hovering over the chart now gives details of the data.
Dashboards are yet another visualization tool. Here we present many different charts/graphs and other visualizations in a reporting panel. As with most reporting, a dashboard caters to an audience and effectively answers a few questions that the audience would be most interested in.  This can be gathered from user input and feedback iterations. As with charts and graphs, its best to start with a few high level fields before making it more sophisticated.
Alerts are another tool that can run periodically or on events when search results evaluate against a condition.There are three options to schedule an alert. First is to monitor whenever the condition happens. The second is to monitor on a scheduled basis as a less urgent information. Third is to monitor  using a realtime rolling window if certain number of things happen within a certain time-period.
Alerts can have associated actions that make them all the more useful. The actions can be specified via the wizard. Some actions could be say send an email, run a script, and show triggered alerts.