Purpose:
Many automations using a Workflow Management System framework must
reconcile with disparate sources of information. Some of them reside in the configuration
store, some in Workflow Management System and others in external systems. Azure
Cloud services and teams leveraging a Workflow Management System find themselves
having to write a query to get this information. While their purpose changes on
a case-by-case basis, the mechanism to do so has evolved to a point where it is
becoming a standard as seen from the above ARM service portfolio as well as the
requirements from foundational services. This document is a definitive guide
towards using the standard way to retrieve this information.
The benefit of standardization and this documentation is that it
has streamlined and smoothened the process that would inevitably be encountered
by teams attempting to write their own solutions. There are also significant
cost savings when following this guide and not resorting to the source depot
tools and offline mechanisms. It also improves visibility of the retrieval
process as well as the accuracy of the data with little or no maintenance.
Why Kusto?
Workflow Management System workflow authors are rightfully
concerned about the support for their workflows in different clouds such as
public, sovereign and air gapped clouds. The goal for these clouds has been to
build and prepare all the required packages and workflows to run on the low
side and then replicate to the high side. In this regard, Kusto comes with two
primary benefits:
1)
Separation of Read-only from read-write access
for all workflows and activities and universally accessible without specific
onboarding access. Data virtualization as if with public access has never been
this easy to do as it is with Kusto. Different systems can access each other’s
cluster and database using the same connection with little or no performance
overhead.
2)
Kusto has incredible support for language,
execution, and results from almost all angles so that writers can not only
write their query once but also safeguard the query from changes due to
maintenance, updates, data variations and software revisions. This eliminates a
lot of costs from conventional mechanisms seen using files, repositories,
toolsets, and others.
These reasons are sufficient to move the read-only traffic away
from Cosmos DB to Kusto.
Why Workflow Management System?
Azure Buildout workflows must be run in Workflow Management System
and while new deployment technologies may content to remain above ARM, Workflow
Management System must support foundational services deployed to dedicated
clusters as well as deployments in isolated clouds and everything in
between. Many workflows and activities
produce and consume information that must remain shared. Earlier, build-drop
locations and Blob storage proved great repositories but writing queries requires
a repository such as Kusto to enable these workflows and activities within Workflow
Management System.
Will configuration store also show up in Kusto?
If there is existing automation relying directly on configuration
store repository, it is not recommended that we make changes to those
automations immediately. The Kusto data does have the same information as the
depot including Cloud Information, geographical information, Region
information, AZ Information, DC Information, Edge Zone, and their corresponding
staging information. New automations and activities can leverage Kusto data
access. It is likely that the new workflows and activities will replace the old
ones rather than require upgrading and paying the technical debt on the old
ones.
How far behind is the data till it is refreshed?
Different systems have different refresh intervals and even the
refresh of the tables in the same database can vary. Usually this is a matter
of minutes but for large tables it can take up to an hour to refresh. Most of
these will be listed in the documentation from the corresponding systems under
Kusto access section but it is safe to assume that the data will lag the
transactional process of create, update and delete of the data in their corresponding
stores.
I just got a new task involving Kusto, how do I get
started?
Determine whether a producer or a consumer is required for Kusto
data access. The steps to follow in the guide differ according to a producer or
a consumer. Please refer to the post:
No comments:
Post a Comment