Exception StackTrace associations for root cause analysis    

Problem statement: Given a method to collect root causes from many data points in errors in logs, can there be a determination of associations between root causes? 

Solution: There are two stages to solving this problem:   

Stage 1 – discover root cause and create a summary to capture it   

Stage 2 – use an association data mining algorithm on root causes.

Stage 1:   

The first stage involves a data pipeline that converts log entries to exception stacktraces and hashes them into buckets. Sample included.  When the exception stack traces are collected from a batch of log entries, we can transform them into a vector representation and using the notable stack frames as features. Then we can generate a hidden weighted matrix for the neural network   

We use that hidden layer to determine the salience using the gradient descent method.        

    

All values are within [0,1] co-occurrence probability range.       

    

The solution to the quadratic form representing the embeddings is found by arriving at the minima represented by Ax = b using the conjugate gradient method.     

We are given input matrix A, b, a starting value x, several iterations i-max, and an error tolerance epsilon < 1     

    

This method proceeds this way:      

    

set I to 0      

    

set residual to b - Ax      

    

set search-direction to residual.     

    

And delta-new to the dot-product of residual-transposed.residual.     

    

Initialize delta-0 to delta-new     

    

while I < I-max and delta > epsilon^2 delta-0 do:      

    

    q = dot-product(A, search-direction)     

    

    alpha = delta-new / (search-direction-transposed. q)      

    

    x = x + alpha.search-direction     

    

    If I is divisible by 50      

    

        r = b - Ax      

    

    else      

    

        r = r - alpha.q      

    

    delta-old = delta-new     

    

    delta-new = dot-product(residual-transposed,residual)     

    

     Beta = delta-new/delta-old     

    

     Search-direction = residual + Beta. Search-direction     

    

     I = I + 1      

    

Root cause capture – Exception stack traces that are captured from various sources and appear in the logs can be stack hashed. The root cause can be described by a specific stacktrace, its associated point of time, the duration over which it appears, and the time of fix introduced, if known.    

    

Stage 2:  

Association data mining determines whether two root causes occur together. The computation involves two computed columns namely Support and Probability. Support defines the percentage of cases in which a rule must exist before it is considered valid. We define that a rule must be found in at least 1 percent of cases.  

Probability defines how likely an association must be before it is considered valid. We will consider any association with a probability of at least 10 percent.  

Bayesian conditional probability and confidence can also be used. Associations have association rules formed with a pair of antecedent and consequent item-sets, so named, because we want to find the value of taking one item with another. Let I be a set of items, T be a set of transactions. Then an association A is defined as a subset of I that occurs together in T. Support (S1) is a fraction of T containing S1. Let S1 and S2 be subsets of I, then the association rule to associate S1 to S2 has support(S1->S2) defined as Support(S1 union S2) and a confidence (S1->S2) = Support(S1 union S2)/ Support(S1).  A third metric Lift is determined as Confidence(S1->S2)/Support(S2) and is preferred because a popular S1 gives high confidence for any S2 and lift corrects that by having a value greater than 1.0 when S2 is also significant.  

Certain databases allow the creation of association models that can be persisted and evaluated against each incoming request. Usually, a training/testing data split of 70/30% is used in this regard.  
Sample: https://jsfiddle.net/g2snw4da/