Monday, February 12, 2018

We were looking at some of the search queries that are collected from the community of those using  logs from an identity provider:

In particular, we were looking for a few lines above and below a match to include associated event attributes. This is easy with a streaming operation in the shell command with "grep –C<N> literal file".  In SQL this becomes slightly complicated involving a recursive common table expression. A nested query might work too provided the identifiers are continuous.
For example:
SELECT a.*
FROM Table1 as a,
(SELECT id FROM Table1 WHERE message LIKE '%hello%') as b
WHERE a.ids BETWEEN b.id-N AND b.id+N;
On the other hand by using max(b.id) < id  and  min(b.id) > id as the sentinels, we can now advance the sentinels row by row in a recursive query to always include a determined number of lines above and below the match
For example:
with sentinels(prevr, nextr, lvl) as (
  select nvl((select max(e.employee_id)
              from   hr.employees e
              where  e.employee_id < emp.employee_id),
              emp.employee_id) prevr,
         nvl((select min(e.employee_id)
              from   hr.employees e
              where  e.employee_id > emp.employee_id),
              emp.employee_id) nextr,
         1 lvl
  from   hr.employees emp
  where  last_name = @lastname
  union all
  select nvl((select max(e.employee_id)
              from   hr.employees e
              where  e.employee_id < prevr),
              prevr
         ) prevr,
         nvl((select min(e.employee_id)
              from   hr.employees e
              where  e.employee_id > nextr),
              nextr
         ) nextr,
         lvl+1 lvl
  from   sentinels
  where  lvl+1 <= @lvl
)
  select e.employee_id, e.last_name
  from   hr.employees e
  join   sentinels b
  on     e.employee_id between b.prevr and b.nextr
  and    b.lvl = @lvl
  order  by e.employee_id; 
 adapted from Oracle blog by Chris Saxon

The additional lines around a match provide additional attributes that may now be searched for direct information or indirectly tagged and counted as contributing towards the tally for the labels.

In the logs, we can leverage protocols other than http and oauth. For example, if we use SAML or other encrypted but shared parameters, we can use it for correlations. Similarly user agents generally give a lot of information about the origin and can be used to selectively filter the requests. In addition to protocols, applications and devices contributing to request parameters, cookies may also store information that can be searched when they make it to the logs. Most mobile devices also come with app stores from where packet capture applications for those devices can be downloaded and installed. Although the use of simulator and live debugging does away with the use of packet capture applications, they certainly form a source of information.
The logs for mobile devices can also be shared especially if they are kept limited and small with a finite number of entries.

No comments:

Post a Comment