A note about Automation and infrastructure capabilities:

Infrastructure capabilities used to be expanded with pre-specified requirements, deterministic allocations based on back-of-the-envelope calculations, and step by step system development. The architecture used to involve service-oriented design and compositions within organizational units such as datacenters. The emerging trend, on the other hand, uses concurrent definitions of requirements and solutions, rapid integration into software-defined components with automation, and heuristics-based evolution. Demand can be managed with services that are redirected to different systems and load-balancing can ensure that all systems are utilized in deployment. With the move to zero-downtime maintenance, capabilities are serviced without any outages or notifications to the customer. Active and stand by servers are rotated so that the incoming traffic is handled appropriately.

Data backup, migration, and replication has brought significant improvements to the practice of system engineering. Content can easily be distributed between sites and near-real-time replication enables customers to find their data when they need it and not some later point.  This has had an impact on the lifecycles of the platform components where newer hardware or upgraded components are much easier to bring in than ever before. With the ease and convenience of updating or upgrading parts or whole systems at lower or intermediary levels, the total cost of ownership of the infrastructure is reduced while the features and capabilities boost productivity. Security of the infrastructure also improves as recent versions of hardware and software bring more robustness and compliances with vulnerability mitigations.

The use of commodity servers and expansion based on a cluster-node design also allows infrastructure to scale out with little or no disruption to existing services. As long as the traffic from higher levels is consolidated towards known entry points, the expansion of the lower level can proceed independently. With an increase in automation of the often-repeated chores to build, deploy and test the system, the overall time to the readiness of the system also reduces bring orders of magnitude scale-out capabilities at the same time that was budgeted.  

Every unit of deployment is also carefully studied to create templates and designs that can be reused across components. Virtual machines and their groups are easily t-shirt sized to meet the varying needs of different workloads and the steps taken to provision entire stacks on those leased resources also get executed automatically. This instantiates most resources customers demand from departments provisioning the infrastructure in a self-service model. Behind the scenes, the resources, their deployments, provisioning, usage calculations, and billing are automated to deliver reports that enable smarter use of those resources. With dry-runs and repeated walkthrough of execution steps, even the datacenters can be re-provisioned. This amount of flexibility resulting from templates and automation has proved critical for cloud-scale.

The container orchestration frameworks provide a good reference for automation and infrastructure design. 

Some more points for consideration from previous post

 Advantages of resource provisioning design:

1.      It enables deployment, scaling, load balancing, logging, and monitoring of all resources both built-in and client authored.

2.      There needs to be only one resource type in the control plane for a new adapter service that stands for the desired device configuration in terms of action and state.  A controller can reconcile the device to its device configuration. There needs to be only one controller for a resource type. The commands, actions and state of a device may be quite involved but the adapter service has a chance to consolidate a logical resource type.

3.      The device statistics can be converted to metrics and monitoring and offloaded for collection and analysis to its own stack. The metrics are pushed from other services. This has an opportunity to use an off the shelf solution. The data in written once but it can be read-only for analysis stacks. The Read-Write and Read-Only separation will help maintain the source of truth.

4.      Separation of control plane and data plane also provides opportunity to use other infrastructure at a later point of time that allows all the services to be cloud friendly and portable.

5.      One of the ways to enforce backward compatibility is to have specific versions for the APIs. In addition to having data syntax and semantics compatibility described via data contracts that can be verified offline via independent tools, it might be helpful to support multiple versions of the service from the start. Exposing resources via Representational State Transfer is a well-known design pattern

6.      The reconciliation of the resource can be achieved with the help of a dedicated controller. A controller for the device control plane resource object is a control loop that watches for the state of the resource and then makes, or request changes as needed. Each controller tries to move the current state to the desired state by reading the spec field associated with the resource object. The control plane resource object can be created, updated or deleted just like any other resource with representational state transfer API design. The controller can set additional states on the resource such as marking it as finished.

 Architecture trade-offs between Designer Workflows and Resource Provider Contracts

Introduction: Automations in the computing and storage industries often require infrastructure that supports the extensibility of logic that is specified by their clients. The infrastructure may support a limited number of predefined workflows that are applicable across clients, but they cannot rule out the customization that individual clients need. There are two ways to support the extensibility of any infrastructure platform. One technique involves the use of a dependency graph of workflows while the other technique involves the use of custom resources that are provisioned by resource providers external to the infrastructure. This article compares the two techniques for their pros and cons.

Description: When an Automation platform supports dependency elaboration via a flowchart, the user dictates the set of steps to be taken in the form of a controlled sequence including conditions on existing built-in workflows. By composing the steps in various modules, a client can write sophisticated logic to perform custom actions that can be re-used across their own modules. This extension allows them to make the infrastructure do tasks that were not available out of the box and the infrastructure records these specifications with the help of tasks and dependencies. The flowchart defined by the automation client gets persisted in the form of a dependency graph on tasks that can be either built-ins or client-defined.

The alternative to this approach is the representation of the result of these activities as resources to be provisioned. This technique generalizes automation as a system that provisions various resources and reconciles their states to the desired configuration defined by its clients. The representation of the resource allows the separation of control and data plane activities for the automation platform enabling many more capabilities for the platform without affecting any of the tasks involved on the data plane. Control and data plane refer to activities about provisioning versus usages of a resource. A resource once provisioned can participate in various data activities without requiring any changes to its provisioning or configuration which eliminates the need to interact with the automation or its client for the usages of the resource. Additionally, control plane activities are subject to governance, management, programmability, and security enhancements that are not easy to specify and manage using logic that gets encapsulated by the automation clients in their customizations without hooks on the automation side.

These two techniques have their own advantages. For example, the dependency-based orchestrations technique provides the following advantages:

1. Coordination and orchestration of activities across workflow boundaries are useful when workflows are componentized into multiple sub-workflows.

2. Orchestration of the workflows’ activities even when they have dependencies on external services which is useful some external service needs to be available for the activity to complete.

3. Enablement of "replay" when upstream artifacts change which prevents rewriting logic when those artifacts change

4. Generating regular and common names in the event catalog provides the ability to namespace, and map existing and new associations and their discoverability.

On the other hand, the resource provisioning architecture supports:

1. A nomenclature and discovery of resources that can be translated with export and import for portability.

2. It provides an opportunity to offload all maintenance to the reconciliation logic in-built into the corresponding operators that the platform maintains.

3. Scope and actions become more granular with export-import capabilities.

4. It improves visibility of all resources in the control plane that makes it easy to manage some or all of them with filters.

Eventually, both techniques require definitions and manifests to be declarative which is wonderful for their testing and validations or for their visualizations in viewers.

Conclusion: The universalization of logic via control and data plane separation enables the automation to be more platform-oriented and increase its portfolio of capabilities with little or no impact on the client's logic. This provides more opportunities for platform development than the dependency graph-based technique.

Kusto continued

 

Kusto query language and engine discussion continued from the previous article.

Let us now look at the Kusto query engine data ingestion path. Again, the admin node is the one to receive the command. This time the command is for data ingestion such as a log for a destination table. The admin node determines if the table has the right schema.

The admin node scans all nodes and finds an available data node to perform the processing and forwards the command to the target. The data node creates the data extent, copies the data, and sends back the extent information to the admin node. The admin node then adds the new shard reference to the table metadata and commits the new snapshot of the metadata. These extents now become sealed and immutable.

When the data is deleted, the actions are in reverse order. First, the reference tree is deleted before the actual data. The references are garbage collected and the actual delete is deferred until the threshold of the container is reached. Since the deletion of the data is deferred, it gives an opportunity to reclaim the reference tree. Queries involving deleted data are using a previous snapshot of the metadata which remains valid for the soon-to-be-deleted data.

Kusto query engine has a similar query parsing and optimization logic as the SQL Server. First, it parses the incoming script into an abstract syntax tree or AST for short.  Then it performs a semantic pass over this tree. In this process, it checks the names, resolves them, verifies that the user has the permissions to access the relevant entity, and then checks the data type and reference. After the semantic pass, the query engine will build an initial relational operator, a tree based on the AST Query engine will further attempt to optimize the query by applying one or multiple predefined rewriting rules. These rewriting rules involve pushdown predicates, replacing table access with extent union structure, splitting aggregation OPS into the leaves, and using top and operators that are replicated to each data extent Together with this parsing an optimization logic Kusto achieves a common abstract syntax tree that is suitable for the query to be executed on the cluster. Let us next look at executing this query on continuously increasing tabular data such as usage data

Kusto has certain limitations it was originally designed as an ad hoc query engine with immense support for fast querying and text data processing capabilities but, as a big data platform, it does not really replace traditional databases like SQL Server. When we attempt to use Kusto as a replacement to the SQL Server, we will run into some limitations these are mentioned below with their potential solutions. There are limits on query concurrency because the cluster runs usually on a collection of eight cores VM nodes. It can execute up to 8 * 10 queries concurrently. The actual number can also be determined by using a Kusto command of showing the cluster policy along with the query throttling limitation of 10 queries per core or node is necessary for a healthy operation of the Kusto query engine. There are limits on the node memory which is set to a number that cannot be larger than the node's physical memory. If it is, then the setting will have no effect. This also implies that a query can take up almost all the node's memory for its processing, but it cannot go beyond the limits of what is available as the node’s physical memory. There are also limits on memory a join or a summarized operation which protects queries from taking too much memory. Finally, there is a limit on the result set size. The number of datasets cannot exceed 500,000 rows and the data size itself cannot exceed 64 megabytes. If the script hits this limitation, it will result in a query error with a partial query failure message, and this can be overcome by summarizing the data to output so that only the interesting result is propagated. This can be done with techniques such as using a take operator to see a small sample of the result and using a project operator to output only the columns of interest. There are no limits on query complexity, but it is not advisable to have more than 5000 conditions in the where class. Lastly, all these limitations are settings that can be adjusted to suit the workload

 

 

Kusto continued

 Kusto comes very useful to query information on Azure resources:

Resources

| where type =~ 'Microsoft.Storage/storageAccounts'

| where tags['tag with a space']=='Custom value'

Or even as part of Azure CLI

az graph query -q "Resources | where type =~ 'Microsoft.Storage/storageAccounts' | where tags['tag with a space']=='Custom value'"

Another sample query for finding work items:

let TotalActionsToBeFiled = cluster('accessmanagement.kusto.windows.net').database('AccessGraph').DstsAccountListServer

| where CloudName=='Public' and DivisionName contains "MyCompanyBusinessUnit"

| distinct ServiceId, CloudName

| join kind=leftanti 

( cluster('accessmanagement.kusto.windows.net').database('AccessGraph').DstsAdoptionSRS 

| where CloudName =="Public"  

    ) on ServiceId

|

project env_ver = "2.1", 

env_name = "Microsoft.Tfs.WorkItemsHistory", 

env_time = tostring(now()), 

env_appId = "T:myAppGuid", 

env_appVer = "1", 

ver ="1.0",

name = "OneAccess.MyOrg",

serviceTreeId = ServiceId,

cloudType = CloudName,

environment = "Prod",

id = strcat(ServiceId, ":", CloudName),

assignedTo = "",

url = "https://microsoft.sharepoint.com/:w:/r/teams/oneAccess/_layouts/15/Doc.aspx?",

createdDate = tostring(now()),

dueDate = tostring(now(+90d)),

lastModifiedDate = tostring(now()),

completionCommitDate = "",

closedDate = "",

closedReason = "",

severity = "1";

//Total new items to be filed

let DoesNotExists = TotalActionsToBeFiled

| join kind=leftanti  (cluster('piesecurity.kusto.windows.net').database('OneAccess').OneAccess_MyOrg) on id ; 

// Total items which are already filed and but are still not resolved

let DoesExits = TotalActionsToBeFiled

| join kind=rightsemi (cluster('piesecurity.kusto.windows.net').database('OneAccess').OneAccess_MyOrg) on id ; 

// Combine these two tables

DoesNotExists

|union DoesExits

|order by dueDate asc

Kusto continued...

 

Kusto is amazingly fast, it is faster than Microsoft SQL Server and it can process billions of records in very little time. It is not a comparable relational database engine. Yet, Azure services are even written on top of Kusto instead of using Microsoft SQL Server. We will see how Kusto is fast but it's also important to note that realizing the characteristics of Kusto will help us do better troubleshooting and writing better queries using Kusto query language

Kusto is fast because it is giving up some features that SQL Server has and  at the same time, it is also leveraging features that SQL Server does not have. For example, SQL Server uses relational data and data constraints but kusto is hardly dealing with read write data. it reads data that is a almost never deleted nor edited. Kusto runs on columnar data so it filters out only those columns that need to be involved in the query with this feature. Kusto is extremely fast and it runs on a cluster with multiple nodes so it can scale arbitrarily

The admin node maintains the metadata store for the cluster. It reads the metadata from blob storage into immutable data structures. The gateway node performs request dispatches and makes and receives api calls. The data hierarchy comprises cluster, database, table, extent data, column and data blocks. The extents are stored on different nodes. A Kusto table has four components: metadata for the table that can be displayed with control commands, an extent directory, a number of extents and column indexes. A Kusto data extent has organization just like a mini table. The extent directory is ordered by ingest time which is why Kusto queries are recommended to have time filters

 

 

 A revisit to event-driven systems: 

Introduction: Software applications often make use of event-driven systems in addition to synchronous calls between components and services. This article revisits some of these discussions. 

Description: First, let us define the event-driven system. It is state based. When the state changes, the system is expected to take certain actions following a finite automaton. This kind of system knows only how to proceed from one state to another and the state transitions must be determined beforehand and validated to not run into endless loops or no activity. The states hold meaning for the application calling the system. 

There are several advantages to an event-driven system. Protocol handlers are a notable example of such a system. They do not have to remember the caller or maintain a session and can be entirely event-based. Message and requests arriving over the wire can be a representation of control plane state transition requests. This form of system is highly scalable. 

Events must be well-defined. Their scope and granularity determine the kind of actions taken. Event-driven systems can perform stateful processing. They can persist the states to allow the processing to pick up where it left off. The states also help with fault tolerance. This persistence of state protects against failures including data loss. The consistency of the states can also be independently validated with a checkpointing mechanism available from Flink. The checkpointing can persist from the local state to a remote store. Stream processing applications often take in the incoming events from an event log. Therefore, this event log stores and distributes event streams written to durable append-only log on tier 2 storage where they remain sequential by time. Flink can recover a stateful streaming application by restoring its state from a previous checkpoint. It will adjust the read position on the event log to match the state from the checkpoint. Stateful stream processing is therefore not only suited for fault tolerance but also reentrant processing and improved robustness with the ability to make corrections. Stateful stream processing has become the norm for event-driven applications, data pipeline applications, and data analytics applications.