Friday, March 28, 2014

After discussing about MSMQ and how to monitor messages (not journaling) on the queues without requiring any additional switches or trigger services on a production instance, I stumbled on and found the tracing for MSMQ under logman query providers and validated that we don't seem to have trace where each message arrival is traced. Hence I found another argument point in favor of what I would like to build for my application.
In any case, here's some more context on how I stumbled on this finding that I want to write an MSMQ monitor for my application.
First, I'm increasingly studying high memory usage in select scenarios with my application and in this context I wanted to find out non-invasive options for production systems of customers. So I came across our windows tools namely logman/tracelog/xperf:
We can turn on private heap tracing on windows either with logman (an out-of-box tool : we just type 'logman query providers' http://msdn.microsoft.com/en-us/library/ms751538(v=vs.110).aspx) or xperf from Windows Performance Analysis Center (http://msdn.microsoft.com/en-us/performance/default.aspx)
tracelog -start heapTrace -f c:\temp\heapTrace.etl -heap -pids 1 <pid_of_splunkd>
tracelog -stop heapTrace
OR xperf -start HeapSession -heap -PidNewProcess "myapp.exe" -BufferSize 1024 -MinBuffers 128 -MaxBuffers 128 -stackwalk HeapAlloc+HeapRealloc
xperf -stop HeapSession -stop -d c:\temp\xperf_heap.etl
we could also explore stackwalking with xperf -help stackwalk and xperfview -> Get Stack counts by type
Xperf allows us to view heap allocation information in addition to kernel info such as CPU and with stacktraces at the time of allocation! More here : http://msdn.microsoft.com/en-us/library/windows/hardware/hh162920.aspx
And if tracing doesn't work we use Procdump to take a fulldump of our process on production systems without stopping the process http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
If we had the luxury we could use gflags, appverifier, pageheap and umdh

But anyways getting back to my discussion on my application. Here's what I wanted to try writing next after MSMQ monitor:
These will be thread based monitors for resource  usages and circular ring buffer of stacktrace collections
 A circular ring buffer of stack trace collections per scheduler or node is very helpful in diagnosing a variety of information. In terms of memory used, we use only pointer sized bytes for each frame ptr and ten of those for each backtrace and say ten to fifty of those buffers in the circular ring buffer should not add upto much.
The stack trace capture to circular ring buffer could be triggered by anything - exceptions, memory allocation and deallocation etc.
In fact the cpu usages are also easy to watch, we read cputicks and += the delta that we are on and off the stack.
 

No comments:

Post a Comment