In this post, I give some examples on DTrace:
DTrace is a tracing tool that we can use dynamically and safely on production systems to diagnose issues across layers. The common DTrace providers are :
dtrace - start, end and error probes
syscall - entry and return probes for all system calls
fbt - entry and return probes for all kernel calls
profile - timer driven probes
proc - process creation and lifecycle probes
pid - entry and return probes for all user-level processes
io - probes for all I/O related events.
sdt/usdt - developer defined probes
sched - for all scheduling related events
lockstat - for all locking behavior within the operating system
Syntax to specify commands  probe-description/predicate/{action}
Variables (eg self->varname = 123) and associative arrays (eg name[key] = expression) can be declared. They can be global, thread local or clause local. Associative arrays are looked up based on keys
Common builtin variables include :
args: the typed arguments to the current probe,
ourpsinfo: the process state for the current thread
execname : the name passed in
pid : the process id of the current process
probefunc | probemod | probename | probeprov: the function name, module name, name and providername,  of the current probe
timestamp vtimestamp  - timestamp and the amount of time the current thread has been running
Aggregate functions include count, sum, avg, min, max, lquantize, quantize, clear, trunc etc.
Actions include trace, printf, printa, stack, ustack, stop, copyinstr, strjoin and strlen.
DTrace oneliners:
Trace new processes:
dtrace -n 'proc:::exec_success { trace(ourspsinfo->pr_psargs); }'
Trace files opened
dtrace -n 'syscall::openat*:entry { printf("%s,%s", execname, copyinstr(arg1)); }'
Trace number of syscalls
dtrace -n 'syscall:::entry {@num[execname] = count(); trace(execname); }'
Trace lock times by process name
dtrace -n 'lockstat:::adaptive_block { @time[execname] = sum(arg1); }'
Trace file I/O by process name
dtrace -n 'io:::start { printf("%d %s %d", pid, execname, args[0]->b_bcount);}'
Trace the writes in bytes by process name
dtrace -n 'sysinfo:::writeoh { @bytes[execname] = sum(arg0); }'