We were looking at some of the search queries that are collected from the community of those using logs from an identity provider:
In particular, we were looking for a few lines above and below a match to include associated event attributes. This is easy with a streaming operation in the shell command with "grep –C<N> literal file". In SQL this becomes slightly complicated involving a recursive common table expression. A nested query might work too provided the identifiers are continuous.
For example:
SELECT a.*
FROM Table1 as a,
(SELECT id FROM Table1 WHERE message LIKE '%hello%') as b
WHERE a.ids BETWEEN b.id-N AND b.id+N;
On the other hand by using max(b.id) < id and min(b.id) > id as the sentinels, we can now advance the sentinels row by row in a recursive query to always include a determined number of lines above and below the match
For example:
with sentinels(prevr, nextr, lvl) as (
select nvl((select max(e.employee_id)
from hr.employees e
where e.employee_id < emp.employee_id),
emp.employee_id) prevr,
nvl((select min(e.employee_id)
from hr.employees e
where e.employee_id > emp.employee_id),
emp.employee_id) nextr,
1 lvl
from hr.employees emp
where last_name = @lastname
union all
select nvl((select max(e.employee_id)
from hr.employees e
where e.employee_id < prevr),
prevr
) prevr,
nvl((select min(e.employee_id)
from hr.employees e
where e.employee_id > nextr),
nextr
) nextr,
lvl+1 lvl
from sentinels
where lvl+1 <= @lvl
)
select e.employee_id, e.last_name
from hr.employees e
join sentinels b
on e.employee_id between b.prevr and b.nextr
and b.lvl = @lvl
order by e.employee_id;
adapted from Oracle blog by Chris Saxon
The additional lines around a match provide additional attributes that may now be searched for direct information or indirectly tagged and counted as contributing towards the tally for the labels.
In the logs, we can leverage protocols other than http and oauth. For example, if we use SAML or other encrypted but shared parameters, we can use it for correlations. Similarly user agents generally give a lot of information about the origin and can be used to selectively filter the requests. In addition to protocols, applications and devices contributing to request parameters, cookies may also store information that can be searched when they make it to the logs. Most mobile devices also come with app stores from where packet capture applications for those devices can be downloaded and installed. Although the use of simulator and live debugging does away with the use of packet capture applications, they certainly form a source of information.
The logs for mobile devices can also be shared especially if they are kept limited and small with a finite number of entries.
In particular, we were looking for a few lines above and below a match to include associated event attributes. This is easy with a streaming operation in the shell command with "grep –C<N> literal file". In SQL this becomes slightly complicated involving a recursive common table expression. A nested query might work too provided the identifiers are continuous.
For example:
SELECT a.*
FROM Table1 as a,
(SELECT id FROM Table1 WHERE message LIKE '%hello%') as b
WHERE a.ids BETWEEN b.id-N AND b.id+N;
On the other hand by using max(b.id) < id and min(b.id) > id as the sentinels, we can now advance the sentinels row by row in a recursive query to always include a determined number of lines above and below the match
For example:
with sentinels(prevr, nextr, lvl) as (
select nvl((select max(e.employee_id)
from hr.employees e
where e.employee_id < emp.employee_id),
emp.employee_id) prevr,
nvl((select min(e.employee_id)
from hr.employees e
where e.employee_id > emp.employee_id),
emp.employee_id) nextr,
1 lvl
from hr.employees emp
where last_name = @lastname
union all
select nvl((select max(e.employee_id)
from hr.employees e
where e.employee_id < prevr),
prevr
) prevr,
nvl((select min(e.employee_id)
from hr.employees e
where e.employee_id > nextr),
nextr
) nextr,
lvl+1 lvl
from sentinels
where lvl+1 <= @lvl
)
select e.employee_id, e.last_name
from hr.employees e
join sentinels b
on e.employee_id between b.prevr and b.nextr
and b.lvl = @lvl
order by e.employee_id;
adapted from Oracle blog by Chris Saxon
The additional lines around a match provide additional attributes that may now be searched for direct information or indirectly tagged and counted as contributing towards the tally for the labels.
In the logs, we can leverage protocols other than http and oauth. For example, if we use SAML or other encrypted but shared parameters, we can use it for correlations. Similarly user agents generally give a lot of information about the origin and can be used to selectively filter the requests. In addition to protocols, applications and devices contributing to request parameters, cookies may also store information that can be searched when they make it to the logs. Most mobile devices also come with app stores from where packet capture applications for those devices can be downloaded and installed. Although the use of simulator and live debugging does away with the use of packet capture applications, they certainly form a source of information.
The logs for mobile devices can also be shared especially if they are kept limited and small with a finite number of entries.
No comments:
Post a Comment