Azure Monitoring continued...

 The access to the data is controlled by the storage access service key also called a blob token that is issued at the time of writing the blob so that the destination can use that token to import the data and handle the single-cast or multi-cast as appropriate to event handlers or appropriate Kusto Cluster. Data copying and rewrite is avoided by merely exchanging the payload information and blob tokens with the delivery service absorbing the process of fetching the permissions from GCS on behalf of the customer. 

The role of the distributed graph might be standing out in the form of a question at this point. It is a service that is used for searching logs and for transformations. It consists of a front-end service and a backend service with each individual component within the FE and BE cloud services as individual micro-services performing a specific task. The front-end service allows the customers to set up query conditions such as job scope and interval period.  

All the monitoring services are region-bound and can be repeated in other regions. Availability within the region such as for disaster recovery purposes requires the use of availability zones. The backend service merely schedules the workers for the appropriate handling of the logs to the customer’s storage account. 

Many miscellaneous activities are specific to the data and whether the data is logs or metrics such as scrubbing, logs to metrics transformations, normalization, and uploading which are handled by dedicated services and serve to enhance the pipeline described so far. The monitoring architecture is generic and always requiring queues, blobs, collections, schedulers, pub-subs, producer-consumers accounts, analysis and reporting stacks and their configurations.

Most of the resources for Azure monitoring are region scoped. This enables Azure Monitoring to be setup in each region.  Some shared data and resources across these regions may exist in a dedicated region which would power use cases of monitoring via the Azure portal.

 Let us take a closer look at how the monitoring data is gathered and analyzed internally within the cloud. The architecture behind Azure Monitoring is a diagnostics pipeline.  This pipeline consists of an ingestion gateway, a delivery service, a distributed graph, a normalizer service and scrubber, logs to metrics converter, and an uploader to a global database. The pipeline support ingestion, streaming, transformations, and querying. This is its hallmark. All these paths are supported end-to-end via the pipeline without any interference from each other. 

The idea behind the monitoring pipeline is one of queuing and pub-sub mechanisms. Logs and metrics flow from gateways to storage queues, where blobs are listened for, scrubbed, forwarded to event hubs, and uploaded to different destinations such as CosmosDB, Azure Data Lake Storage (ADLS), and delivery services. The rate of flow to the queues can be throttled and the schema hints can be propagated to the storage where the schema and notifications power the analytics. The metrics accumulation in an MDM facilitates the logic for throttling and rate adjustments while the schemas are mostly published and queried via Kusto. 

Configurations for different storage containers, queues, and hubs are defined between the collection and the delivery services. These are called Monikers and it is a pairing of Event hub and storage account. The ingestion service is responsible to connect the monitoring agent with its storage account. The use of this service reduces the number of monikers, the number of blob writes to storage, and the complexity of the distributed graph representation. The storage is billed in terms of transactions and what would earlier take hundreds of transactions and blob writes, would require only tens of transactions using the ingestion or ingress service. It can also aggregate the blobs before writing them to the storage account. 

The corresponding egress service is the delivery service and can be considered an equivalent of Apache Kafka. It comes with a set of producer and consumer definitions and this pub-sub service operates at the event level. There is an application programmability interface provided for consumers who would like to define the monikers instead of the control on the events. The setting up of monikers determines where and how the data is delivered and the use of monikers reduces the bill in an equivalent way to how the ingress did. The destinations are usually Kusto clusters and event hubs. The delivery service forms the core of the pipeline with agents and ingestion pouring data to storage defined by monikers. At the other end of the pipeline are the event hubs and Kusto clusters. 

Collection and Storage have pre-requisites. For example, when virtual machines are created, they automatically have a monitoring agent (MA) installed. This agent reaches out to a collection service with an intent to write and define a namespace. The handshake between the monitor and the agent gives the agent the configuration necessary to direct its data to a destination Moniker which can scale automatically for the storage account. 

Unlike the collection and the storage, which are automatically provisioned, the delivery and the paths are set up by the customer using the application programmability interfaces in the extensibility SDK associated with the delivery services. The delivery service then concerns itself merely with the resolving of monikers, the listening on the monikers, the filtering of the events, and its delivery to the Kusto clusters and event hubs. If the destination is unreachable or unavailable, the data is handed off to the snapshot delivery service which reads the delivery service namespaces for retries. The data is never put in memory when the delivery service forwards the data to a cache under a namespace key. The snapshot delivery service acts as the standby destination in place of the unreachable one. 

 

Introduction to Azure Monitoring:

Abstract: This article is an introduction to the programmability aspects of Azure Public cloud monitoring service that is known for its unprecedented scale and flexibility in reporting metrics from the resources hosted on the Azure Public Cloud.

Description: Azure Monitoring helps us maximize the availability of our applications and services hosted on Azure Public Cloud. It is a complete solution for collecting, analyzing, and acting on the telemetry from the Cloud Environment.  This monitoring program comprises of an application performance management system called the Application Insights, the host monitoring system called the VM Insights and Container Insights, the Log Analytics solution which allows drill down into the monitoring data, smart alerts, and automated actions which help support operations at scale, and visualizations with dashboard and workbooks. The data collected from this comprehensive solution become part of Azure Monitor Metrics.

Azure Monitoring is not only about metrics, but it is also about logs, and it allows us to gather insights, visualize, analyze, respond, and integrate. The monitoring data platform works for both metrics as well as logs. While events and traces become part of logs, metrics are numerical values that quantify application performance at a given point in time.  The metrics store and its visualization with metrics explorer and the log data and its filtering with Log Analytics are just applications dedicated to their respective data. The Azure Monitor uses the Kusto query language that is suitable for simple log queries that also include advanced functionalities such as aggregations, joins, and smart analytics. Kusto benefits from both SQL and Splunk querying practices.

One of the most interesting aspects of Azure Monitoring is that it collects metrics from Applications, Guest OS, Azure resource monitoring, Azure subscription monitoring, and azure tenant monitoring to include the depth and breadth of the systems involved. Alerts and Autoscale help determine the appropriate thresholds and actions that become part of the monitoring stack, so the data and the intelligence are together and easily navigated via the dashboard.  Azure Dashboards provide a variety of eye-candy charts that better illustrate the data to the viewers than the results of the query. Workbooks provide a flexible canvas for data analysis and the creation of rich visual reports in the Azure Portal.  The analysis is not restricted to just these two. Power BI remains the robust solution to provide analysis and interactive visualizations across a variety of data sources and it can automatically import log data from Azure monitor. Azure Event Hubs is a streaming platform and event ingestion service which permits real-time analytics as opposed to batching or storage-based analysis. APIs from the Azure monitor help with reading and writing data as well as configure and retrieve alerts.

 Networking Techniques Continued ...  

Introduction: This is a continuation of the article written about networking technologies in the cloud specifically the Azure public cloud.   

 

Description: Some techniques with networking drive down costs significantly. For example, Switched Embedded Teaming can be used to combine two 10 Gb ports to 20 Gb ports. This is a boost to the capacity and has little or no additional load on the CPU. Such techniques have always been part of the networking industry and its history. Modems that provided Point-to-Point connectivity enabled the bandwidth to be increased by cumulative additions of other modems. Multilink capabilities enabled all the modems to be active at once while the Bandwidth allocation protocol allowed these modems to be aggregated one by one. These techniques allowed IT administrators to extend the service life of the existing infrastructure and hardware so that new equipment could be purchased when the organization was ready instead of when incidents occurred.  The overall throughput could be improved so that high-priority applications could get the network access they need.  Service levels could be added to the existing default conditions with the help of Quality-of-Service improving protocols that prioritized traffic to and from critical applications. DiffServ and IntServ paradigms boosted the adoption of certain techniques and technologies in favor of others. IntServ and DiffServ are models of providing Quality-of-Service. IntServ stands for Integrated services and DiffServ stands for Differentiated Services. In the IntServ model, QoS is applied on a per-flow basis and addresses business model and charging concerns. Even in mobile phone networks, this is evident when certain billing options are desirable but not possible. In Differentiated services, the emphasis is on scalability, a flexible service model, and simpler signaling. Scalability here means we do not track resources for each of the considerable numbers of flows. Service model means we provide bands of service such as platinum, gold, and silver.  

 

Queuing is also a technique that helps improve service levels. Virtual Machine Multi-Queue on Windows Server improves throughput. A built-in software load balancer can distribute incoming requests where the policies are defined using the network controller. Putting this all together, we have a network controller that sits between a Gateway and a Hyper-V vSwitch that works with a load balancer to distribute traffic to other Hyper-V vSwitches which connect several other virtual machines. The Gateway works with a Hyper-V vSwitch that routes internal as well as external traffic to Enterprise sites and Microsoft Azure using IP Security protocol (IPSec) or Generic Routing Encapsulation (GRE) protocol. The Hyper-V vSwitches in the hybrid cloud is also able to send and receive Layer3 traffic via the gateway which rounds up overall connectivity for the entire Hybrid cloud with limitless expansion internally.  This mode of connecting Hybrid cloud with public cloud is here to stay for a while because customers for the public cloud have significant investments in their hybrid cloud and the public cloud cannot migrate the applications, services, devices, and organizations with their workloads without requiring code changes. Networking is probably the easiest technique to allow new ones to be written directly on the public cloud so that traffic may be eased out on to public cloud as investments in the hybrid cloud can be scaled back.   

 

 

Azure Software defined networking features:

Introduction: Azure is a public cloud with a portfolio of services. Azure Networking is one of the core services in the portfolio and offers Network-as-a-service functionality. This article discusses some of the main features of this service.

Description: Networking is all about links and communication. It involves layers of protocols, a mix of network topologies, hybrid equipment, naming and resolving mechanisms, access controls and policy specifications, and a variety of management and troubleshooting tools and services. Azure ExpressRoute provides optimal routing for best performance. The default traffic is over the Microsoft Global Network which is often referred to as the cold potato routing. Inter availability zones and Inter regions provide low latency and geographical networking. The traffic routes between Azure and the internet can be determine with routing preference. Azure Load balancer provides high performance with low latency.

There are manageability features that allow the management of on-premises, multi-cloud, 5G, and edge deployments. Connection is provided via Azure Virtual-WAN and the edge and 5G scenarios are enabled with Edge Zones, Edge Zones with Carrier, or private edge zones. These networks are secured by zero-trust based network security which involve segmentation, and the use of Azure WAF and Azure Bastion. There is intelligent threat detection with Azure DDoS protection. Private connectivity is available via Azure Private Link. Azure Network Virtual Appliance and Remote Access Service can provide end to end IP tunneling.

The Network as a service features easy to use scalable service and tools. The traffic is managed via Azure App Gateway and protected via Azure WAF. Azure FrontDoor helps define and monitor global routing. Firewall capabilities are turned on with Azure Firewall. VNet NAT is used to ensure reliable network address translation and can provide outbound connectivity.

Software defined networking is built into each Windows Server. When IT wants the ability to deploy applications quickly, SDN and network controller can be used, and policy can be managed with PowerShell. HyperV, and network controller can be used to create VxLAN overlays which does not require re-assignment of IP addresses. Hybrid SDN gateways can be used to assign and manage resources independently.

There is greater security and isolation of workloads with the use of network security groups and distributed firewall for micro-segmentation. North-South internet traffic and East-West intranet traffic can be established differently. User-defined routing can be configured with service chains can be established with 3^rd party appliances such as firewall, load balancer or content inspection. Cost is driven down by converging storage and network on Ethernet, and activating Remote Direct Memory Access (RDMA)

 

 

 Lessons learnt from build environment enhancements for building Sofware applications.

Introduction: Building software applications to produce the distributable is a routine for all applications. The logic in the build and automation systems is an investment for an organization and requires many man-hours for development. This article captures some of the lessons learned from different build environments.

Description: Build automations do not just invoke the compiler to build and link the code. It is an opportunity to introduce workflows that validate and vet the application, raise the software quality and introduce numerous gates for the application to be released.  Some of these are enumerated below:

1) The software application is dependent on several dependencies. These must be managed and updated so that the application remains defects and vulnerabilities free. Maintaining a source of truth for the application builds is an important aspect of the build system. Software the builds locally on developer workstations is fraught with environmental factors that are not often visible to the developers. Ensuring that the application always builds in a controlled environment even if the dependencies and modules are recreated each time, is one of the core responsibilities of the build.

2) Starting from a clean state to not pollute the environment with pre-existing binaries and builds is necessary for the build system. This is true for both the developer builds as well as the remote branch builds. These builds must make it easy for clean up so that the task of bringing in just the minimum for builds is done right each time.

3) Access permissions to the build must be tightly controlled. The introduction of tools and the versioning of the software cannot be modified without proper processes in place. It may be required for developer teams to introduce their own build logic.  A specific format, and file is required for this purpose. The introduction of the additional logic via the checkin of the file is like the introduction of code into the pipeline which requires no escalation of privilege.

4) The versioning of the builds must be proper. A lot of work has gone into the naming system, and this works for most versioning. It is called SemVer which is short for semantic versioning. Care must be taken to introduce the version stamps into each builds and binary so that the origin of a particular assembly can be unequivocally displayed.

5) the use of data protection, encryption and certification tools is as relevant to build artifacts as they are for securing customer data. There must be no tampering of artifacts or the cross contamination of dependencies without requiring these to be clean.

6) Several tools may be introduced into the build for static analysis and verification of the code as well as that of the binaries produced by the build. Recently these have included artifacts and images from container orchestration frameworks.

Lastly, the build automation must scale to requests from thousands of repositories and organizations. This requires a pool of resources to be always available.

 

Given clock hands positions for different points of time as pairs A[I][0] and A[I][1] where the order of the hands does not matter but their angle enclosed, count the number of pairs of points of time where the angles are the same

    public static int[] getClockHandsDelta(int[][] A) {

        int[] angles = new int[A.length];

        for (int i = 0; i < A.length; i++){

            angles[i] = Math.max(A[i][0], A[i][1]) - Math.min(A[i][0],A[i][1]);

        }

        return angles;

    }

    public static int NChooseK(int n, int k)

    {

        if (k < 0 || k > n || n == 0) return 0;

        if ( k == 0 || k == n) return 1;

        return Factorial(n) / (Factorial(n-k) * Factorial(k));

    }

 

    public static int Factorial(int n) {

        if (n <= 1) return 1;

        return n * Factorial(n-1);

    }

 

    public static int countPairsWithIdenticalAnglesDelta(int[] angles){

        Arrays.sort(angles);

        int count = 1;

        int result = 0;

        for (int i = 1; i < angles.length; i++) {

            if (angles[i] == angles[i-1]) {

                count += 1;

            } else {

                if (count > 0) {

                    result += NChooseK(count, 2);

                }

                count = 1;

            }

        }

        if (count > 0) {

            result += NChooseK(count, 2);

            count = 0;

        }

        return result;

    }

 

        int [][] A = new int[5][2];
         A[0][0] = 1;    A[0][1] = 2;
         A[1][0] = 2;    A[1][1] = 4;
         A[2][0] = 4;    A[2][1] = 3;
         A[3][0] = 2;    A[3][1] = 3;
         A[4][0] = 1;    A[4][1] = 3;
 1 2 1 1 2
1 1 1 2 2
4