Wednesday, February 5, 2014

We discussed alerts, actions, charts, graphs, visualizations, and dashboards in the previous post. We now review recipes for monitoring and alerting. These are supposed to be brief solutions for common problems. Monitoring helps you see what is happening to your data. As an example, let us say we want to monitor how many concurrent users are there at any given time. This is a useful metric to see if a server is overloaded.  To do this, we search for the relevant events. Then we use the concurrency command to find the number of users that overlap. Then we use a time chart reporting command to display a chart of the number of concurrent users.
We specify this as search sourcetype=login_data | concurrency duration=ReqTime | timechart max(concurrency)
Let us say next that we want to monitor the inactive hosts.
we use the metadata command that gives information on host, source and source types
Here we specify
| metadata type=hosts | sort recentTime | convert ctime(recentTime) as Latest_Time
We can use tags to categorize data and use it with our searches.
In the above example, we could specify:
... | top 10 tag::host to specify top ten host types.
Since we talked about tag, we might as well see an example about event type
we could display a chart of how host types perform using only event types that end in _host with the following:
... | eval host_types=mvfilter(match(eventtype, "_host$"))
    | timechart avg(delay) by host_types
Another common question that we could help answer with monitoring is how did today  perform compared to previous month ?
For example we might want to view the hosts that were more popular today than previous month.
This we do with the following steps:
1. get the monthly usage for each host
2. get the daily usage for each host and append
3. use the stats to join the monthly and daily usages by host.
4. use sort and eval to format the results.
Let's try these commands without seeing the book.
| metadata type=hosts | sort duration | earliest = -30d@d | stat sum(duration) as monthly_usage by host | sort 10 - monthly_usage | streamstats count as MonthRank.
Cut and paste the above with changes for daily as
append[ | metadata type=hosts | sort duration | earliest = -1d@d | stat sum(duration) as daily_usage by host | sort 10 - daily_usage | streamstats count as DailyRank]
Next join the monthly and the daily rankings with stats command:
stats first(MonthRank) as MonthRank first(DayRank) as DayRank by host
Then we format the output :
eval diff=MonthRank-DayRank | sort DayRank | table DayRank, host, diff, MonthRank
Each of the steps can now be piped to the other and the overall search query can be pipe-concatenated to form a single composite query.




No comments:

Post a Comment