Thursday, June 24, 2021

 

This is a continuation of the article originally written here.

This diagram depicts the organization of deployment-as-a-service stack.

Some of the features for the above platform include:

·        best practice in deployment for all tenant's
·        automatic migration for all cloud dependencies
·        Virtualization of deployment technologies and migrations from one set of artifacts to another
·        Asynchronous and background processing with continuous monitoring
·        Programmability options to work with various clients.
·        Analysis and Reporting dashboard
·        Ability to scope down deployments from cloud to on-premises.
·        Curated collection of recipes in automation
·        App-Store integration for allowing clients to opt into deployment stack via published and downloadable applications.
·        Support for browser-based request-response processing
·        Isolated and protected data for all tenants
·        Globally accessible and remote invocable deployment automations
·        Scalable number of client connections allowed for same repository of artifacts.
·        Extensions and customizations for all tenants.
·        Full transparency in the form of logs and events
·        Continuous availability of Deployment stack
·        Ability to provide service-levels for clients.
·        Providing multiple language support including internationalization and globalization
·        Opt-in modernization of existing deployment dependencies
·        One-stop-shop for all deployment activities in the public cloud

Some of the challenges that would be encountered in this regard would include the following:

1) The scope and the environment for a deployment might be different between technologies and mapping must be formed to enable declarations against one form of technology to be repurposed for another.

2) There are many external data stores where configurations may be kept.  File-system or source control is not the only source. Collecting and collocating all the configuration poses a significant task.

3) Artifacts might vary in syntax and semantics and a one-to-one correspondence might not exist between two technologies. This would require some canonicalization, reconciliation and prior translations to be setup.

4) Even if the same technology is used, the artifacts might have different scope and levels of change which may vary widely across deployments. Rolling one deployment to another environment or topology might require additional steps.

5) The recipes must be portable across infrastructure to allow them to vary independently. The more hardcoded literals are used in a script, the tougher it becomes to move or re-purpose the script for another angle.

6) There are several deployments where the environment variable and transient data are used a lot. These must be eliminated in favor of declarative syntax.

Conclusion:  This article tried to pose deployment options for a cloud software maker to be one that is repetitive and requires continued investments over time. In such a case, outsourcing the deployment logic to a deployment-as-a-service offering provides opportunities to save costs and develop best practices.

 

Wednesday, June 23, 2021

 Problem statement: Public cloud computing infrastructure hosts customer workloads as well as the ever-increasing portfolio of services offered by the cloud to its customers. These services from publishers both external and internal to the cloud require deployments over the public cloud. They write and maintain this logic and bear the cost of keeping up with the improvements and features available for this deployment logic. This article investigates the implementation platform for a global multi-tenant deployment-as-a-service offering from the public cloud. 

Solution:  

Multi-tenancy and software-as-a-service model is possible only with a cloud computing infrastructure.  The deployment logic for a service for a cloud differs significantly from that for a desktop. A cloud expects more conformance than a desktop or enterprise deployment justifying the need for a managed program. As Cloud service developers struggle to keep up with the speed of software development for cloud-savvy clients, they face deployment as a necessary evil that draws their effort from their mission. Even when organizations pay the cost up upfront in the first version released with a dedicated staff, they realize that the cloud is evolving at a pace that rivals their own release timeframes. Some may be able to keep up with the investments year after year but for most, this is better outsourced so that they spend less time on rewriting with newer deployment technologies or embracing the enhancements features to the cloud. 

Cloud service developers have an incentive to join this program. They need not be declarative about the resources they use. They just need to define the policies. This is a significant shift in the paradigm that has cost them tremendously in all their deployments so far. Cloud resources are not only scalable and limitless, but their efficient usage is also often neglected by service developers who often use the expedient solution or over-allocate the resources.  A managed program for deployment across these internal and external software makers not only passes on savings to the customers but also helps the cloud migrate quickly to better forms of development with little or no disruption to their customers. 

Deployment logic is quite complicated involving considerations for install, upgrade, rollback, and cleanup of control resources as well as the storage, migration, protection, and replication of data. Fortunately, the bulk of the deployment logic involves cloud resources and a managed program from the cloud is best positioned to onboard the service to the cloud. Concerns addressed and considerations made in the cloud can now be offered in the form of a billable service that will articulate the savings passed on to the customer.  


Tuesday, June 22, 2021

 Recipes and their relevance to automation. 

The world of automation is dependent on commands and sequences that achieve a certain outcome or move to the desired state for a resource. The infrastructure may be elastic, scalable, self-healing, and state-reconciling but it’s the collections of recipes that make one automation system more popular than another. Let us take a closer look at these recipes. Logic and recipes are somewhat different. Logic is defined as flows and represented by flow charts. These are suitable for programming languages. Recipes are more suited for scripting languages and cookie-cutter tasks. An automation engineer will demand consistency, easy identification, troubleshooting, and resolution. She will have little patience for debugging complex multi-branch workflows and focus instead on dividing and conquering a flat list of steps to find the buggy step. The scripts and programs have at least one thing in common. Both are curated over time so they become even more sought after as they get better at not only what they do but also what they can do. The joy of automation is the savings in time and cost when it is repeated without human intervention. Thousands of repetitions can be kicked off in no time and the process will not only assign individual identifiers but also correctly maintain the mapping between the results and their respective set of input variables. One such example is the commissioning of computing systems where the stack involves multiple layers. Automations have interesting by-products in the form of intermediary results, states, and configurations. They execute all the tasks local to a host, unlike services and functions that may even be serverless. Automation can easily be monitored for a predictable start, progress, and stop. A controller leverages this aspect for scheduling their executions which are also called jobs. Automations have certain characteristics just like many other massive software products. They are usually written once and run many times over and on different hosts. They are portable. They embrace shell-based invocations. They become a library of low-level commands and high-level scripts, or recipes, and their organization is determined by their reusability and the requirements of those that use them.  Some hierarchy might be introduced for nomenclature, but the recipes remain a single-level collection even if they have groupings represented by prefixes and suffixes. This collection is often called a cookbook or run book and includes shell-based invocations. The language of automation is PowerShell on windows and Bash on Linux. Remote processing tools such as curl and data transformation to JSON have popularized the scripts and made them even more powerful as they delegate the intensive processing to the right resource. Another aspect of automation that is quite popular is pipelining. The data output from a previous stage becomes the input to the next stage and the operators allow for transformation and analysis between stages of the pipelining process. Automations have a ubiquitous affinity to state and configuration persistence in the form of a relational store, a high-performance message broker such as RabbitMQ, ZeroMQ, or MSMQ, a query, and analysis stack as well as a reporting and visualization stack. Even if their representations vary, automation stands out for their recipes.


Monday, June 21, 2021

This is a continuation of the earlier article that introduced ServiceBus replication:
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
using Microsoft.ServiceBus;
using Azure.Messaging.ServiceBus;
using Microsoft.ServiceBus.Messaging;

namespace SBQueueSender
{
    /*
     * Program to complete structure and content replication of SB Entities from source namespace to destination namespace
     */
    class Program
    {
        private static string connectionString = "";
        private static string secondaryConnectionString = "";
        private static string sourceTopicOrQueueName = "";
        private static string sourceSubscriptionName = "";
        private static string destinationTopicName = "";
        private static string destinationSubscriptionName = "";
        private static string replicateStructure = "";
        private static string replicateContent = "";

        static async Task Main(string[] args)
        {
            IConfigurationBuilder builder = new ConfigurationBuilder().AddJsonFile("appsettings.json");
            IConfigurationRoot config = builder.Build();
            connectionString = config["primaryConnectionString"];
            secondaryConnectionString = config["secondaryConnectionString"];
            sourceTopicOrQueueName = config["sourceTopicName"];
            sourceSubscriptionName = config["sourceSubscriptionName"];
            destinationTopicName = config["destinationTopicName"];
            destinationSubscriptionName = config["destinationSubscriptionName"];
            replicateStructure = config["replicateStructure"];
            replicateContent = config["replicateContent"];

            if (string.IsNullOrWhiteSpace(connectionString) ||
                string.IsNullOrWhiteSpace(secondaryConnectionString) ||
                string.IsNullOrWhiteSpace(sourceTopicOrQueueName) ||
                string.IsNullOrWhiteSpace(sourceSubscriptionName) ||
                string.IsNullOrWhiteSpace(destinationTopicName) ||
                string.IsNullOrWhiteSpace(destinationSubscriptionName) ||
                string.IsNullOrWhiteSpace(replicateStructure) ||
                string.IsNullOrWhiteSpace(replicateContent))
            {
                Console.WriteLine("Please enter appropriate values in appsettings.json file. Exiting...");
                await Task.Delay(2000);
            }
            else
            {
                NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
                if (Boolean.TrueString.CompareTo(replicateStructure) == 0)
                {
                    await ReplicateStructure(namespaceManager, secondaryConnectionString);
                }

                if (Boolean.TrueString.CompareTo(replicateContent) == 0)
                {
                    await ReplicateContent(namespaceManager, secondaryConnectionString);
                }
            }
        }

        /// <summary>
        /// Replicates the structure of the SB namespace.
        /// </summary>
        /// <param name="namespaceManager"></param>
        /// <param name="destinationConnectionString"></param>
        /// <returns>A <see cref="Task"/> representing the result of the asynchronous operation.</returns>
        public static async Task ReplicateStructure(NamespaceManager namespaceManager, string destinationConnectionString)
        {
            var targetNamespaceManager = NamespaceManager.CreateFromConnectionString(destinationConnectionString);
            foreach (var queue in namespaceManager.GetQueues())
            {
                try
                {
                    Console.WriteLine(string.Format("Creating queue {0} ", queue.Path));
                    targetNamespaceManager.CreateQueue(queue);
                }
                catch (MessagingEntityAlreadyExistsException)
                {
                    Console.WriteLine(string.Format("Queue {0} already exists.", queue.Path));
                }
            }

            foreach (var topic in namespaceManager.GetTopics())
            {
                try
                {
                    Console.WriteLine(string.Format("Creating topic {0} ", topic.Path));
                    targetNamespaceManager.CreateTopic(topic);
                }
                catch (MessagingEntityAlreadyExistsException)
                {
                    Console.WriteLine(string.Format("Topic {0}  already exits...", topic.Path));
                }

                foreach (var subscription in namespaceManager.GetSubscriptions(topic.Path))
                {
                    try
                    {
                        Console.WriteLine(string.Format("Creating subscription {0} ", subscription.Name));
                        targetNamespaceManager.CreateSubscription(subscription);
                    }
                    catch (MessagingEntityAlreadyExistsException)
                    {
                        Console.WriteLine(string.Format("Subscription {0}  already exits...", subscription.Name));
                    }
                }
            }
        }

        /// <summary>
        /// Replicates the contents of the SB entities
        /// </summary>
        /// <param name="namespaceManager"></param>
        /// <param name="destinationConnectionString"></param>
        /// <returns>A <see cref="Task"/> representing the result of the asynchronous operation.</returns>
        public static async Task ReplicateContent(NamespaceManager namespaceManager, string destinationConnectionString)
        {
            await using var client = new ServiceBusClient(connectionString);
            foreach (var sbQueue in namespaceManager.GetQueues("messageCount Gt 0"))
            {
                sourceTopicOrQueueName = sbQueue.Path;
                ServiceBusReceiver receiver = client.CreateReceiver(sbQueue.Path);
                ServiceBusReceivedMessage receivedMessage = await receiver.ReceiveMessageAsync(new TimeSpan(0,0,30));
                await SendMessageToTopicAsync(destinationConnectionString, sbQueue.Path, receivedMessage);
            }

            foreach (var sbTopic in namespaceManager.GetTopics("messageCount Gt 0"))
            {
                sourceTopicOrQueueName = sbTopic.Path;
                ServiceBusReceiver receiver = client.CreateReceiver(sbTopic.Path);
                foreach (var sbSub in namespaceManager.GetSubscriptions(sbTopic.Path))
                {
                    sourceSubscriptionName = sbSub.Name;
                    Console.WriteLine("Listening on topic: {0} with subscription: {1}", sourceTopicOrQueueName, sourceSubscriptionName);
                    await ReceiveMessagesFromSubscriptionAsync(namespaceManager, sbTopic.Path, connectionString, sbTopic.Path, sbSub.Name);
                }
            }
        }

        private static async Task ReceiveMessagesFromSubscriptionAsync(NamespaceManager namespaceManager, string sbTopicPath, string connectionString, string sourceTopicName, string sourceSubscriptionName)
        {
            try
            {
                await using (ServiceBusClient client = new ServiceBusClient(connectionString))
                {
                    // create a processor that we can use to process the messages
                    ServiceBusProcessor processor = client.CreateProcessor(sourceTopicName, sourceSubscriptionName, new ServiceBusProcessorOptions()
                    {
                        MaxConcurrentCalls = 1,
                        AutoCompleteMessages = false,
                    });

                    // add handler to process messages
                    processor.ProcessMessageAsync += MessageHandler;

                    // add handler to process any errors
                    processor.ProcessErrorAsync += ErrorHandler;

                    // start processing 
                    await processor.StartProcessingAsync();

                    Console.WriteLine("press any key to stop ...");
                    Console.ReadKey();

                    // stop processing 
                    Console.WriteLine("\nStopping the receiver...");
                    await processor.StopProcessingAsync();
                    Console.WriteLine("Stopped receiving messages");
                }
            }
            catch (Exception e)
            {
                Console.WriteLine("Error: ", e);
            }
        }

        private static async Task MessageHandler(ProcessMessageEventArgs args)
        {
            string id = args.Message.MessageId;
            Console.WriteLine($"Received: {id} from subscription: {sourceSubscriptionName}");

            await SendMessageToTopicAsync(secondaryConnectionString, sourceTopicOrQueueName, args.Message);
            Console.WriteLine($"Sent: {id} from subscription: {sourceSubscriptionName}");

            // complete the message. messages is deleted from the queue.
            await args.CompleteMessageAsync(args.Message);
        }

        private static Task ErrorHandler(ProcessErrorEventArgs args)
        {
            Console.WriteLine(args.Exception.ToString());
            return Task.CompletedTask;
        }

        private static async Task SendMessageToTopicAsync(string connectionString, string destinationTopicName, ServiceBusReceivedMessage serviceBusReceivedMessage)
        {
            if (serviceBusReceivedMessage != null)
            {
                try
                {
                    // create a Service Bus client
                    await using (ServiceBusClient client = new ServiceBusClient(connectionString))
                    {
                        // create a sender for the topic
                        ServiceBusSender sender = client.CreateSender(destinationTopicName);
                        await sender.SendMessageAsync(new ServiceBusMessage(serviceBusReceivedMessage));
                        Console.WriteLine($"Sent a single message to the topic: {destinationTopicName}");
                    }
                }
                catch (Exception e)
                {
                    Console.WriteLine("Error: ", e);
                }
            }
            else
            {
                Console.WriteLine("received null message.");
            }
        }

        private static async Task SendTestMessageToTopicAsync(string body)
        {
            // create a Service Bus client 
            try
            {
                await using (ServiceBusClient client = new ServiceBusClient(connectionString))
                {
                    // create a sender for the topic
                    ServiceBusSender sender = client.CreateSender(sourceTopicOrQueueName);
                    await sender.SendMessageAsync(new ServiceBusMessage(body));
                    Console.WriteLine($"Sent a single message to the topic: {sourceTopicOrQueueName}");
                }
            }
            catch (Exception e)
            {
                Console.WriteLine("Error: ", e);
            }
        }
    }
}

Sunday, June 20, 2021

This is a continuation of the earlier article that introduced ServiceBus replication:
 Reference:
// Sample program for data migration
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
using Microsoft.ServiceBus;
using Azure.Messaging.ServiceBus;
 
namespace SBQueueSender
{
    class Program
    {
        private static string connectionString = "";
        private static string secondaryConnectionString = "";
        private static string sourceTopicOrQueueName = "";
        private static string sourceSubscriptionName = "";
        private static string destinationTopicOrQueueName = "";
        private static string destinationSubscriptionName = "";
 
        static async Task Main(string[] args)
        {
            IConfigurationBuilder builder = new ConfigurationBuilder().AddJsonFile("appsettings.json");
            IConfigurationRoot config = builder.Build();
            connectionString = config["primaryConnectionString"];
            secondaryConnectionString = config["secondaryConnectionString"];
            sourceTopicOrQueueName = config["sourceTopicOrQueueName"];
            sourceSubscriptionName = config["sourceSubscriptionName"];
            destinationTopicOrQueueName = config["destinationTopicOrQueueName"];
            destinationSubscriptionName = config["destinationSubscriptionName"];
 
            if (string.IsNullOrWhiteSpace(connectionString) ||
                string.IsNullOrWhiteSpace(secondaryConnectionString) ||
                string.IsNullOrWhiteSpace(sourceTopicOrQueueName) ||
                string.IsNullOrWhiteSpace(sourceSubscriptionName) ||
                string.IsNullOrWhiteSpace(destinationTopicOrQueueName) ||
                string.IsNullOrWhiteSpace(destinationSubscriptionName))
            {
                Console.WriteLine("Please enter appropriate values in appsettings.json file. Exiting...");
                await Task.Delay(2000);
            }
            else
            {
                NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
                await using var client = new ServiceBusClient(connectionString);
                foreach (var sbQueue in namespaceManager.GetQueues("messageCount Gt 0"))
                {
                    sourceTopicOrQueueName = sbQueue.Path;
                    ServiceBusReceiver receiver = client.CreateReceiver(sbQueue.Path);
                    ServiceBusReceivedMessage receivedMessage = await receiver.ReceiveMessageAsync();
                    await SendMessageToTopicAsync(secondaryConnectionString, sbQueue.Path, receivedMessage);
                }
 
                foreach (var sbTopic in namespaceManager.GetTopics("messageCount Gt 0"))
                {
                    sourceTopicOrQueueName = sbTopic.Path;
                    ServiceBusReceiver receiver = client.CreateReceiver(sbTopic.Path);
                    foreach (var sbSub in namespaceManager.GetSubscriptions(sbTopic.Path))
                    {
                        sourceSubscriptionName = sbSub.Name;
                        Console.WriteLine("Listening on topic: {0} with subscription: {1}", sourceTopicOrQueueName, sourceSubscriptionName);
                        await ReceiveMessagesFromSubscriptionAsync(connectionString, sbTopic.Path, sbSub.Name);
                    }
                }
            }
        }
 
        private static async Task ReceiveMessagesFromSubscriptionAsync(string connectionString, string sourceTopicName, string sourceSubscriptionName)
        {
            await using (ServiceBusClient client = new ServiceBusClient(connectionString))
            {
                // create a processor that we can use to process the messages
                ServiceBusProcessor processor = client.CreateProcessor(sourceTopicName, sourceSubscriptionName, new ServiceBusProcessorOptions());
 
                // add handler to process messages
                processor.ProcessMessageAsync += MessageHandler;
 
                // add handler to process any errors
                processor.ProcessErrorAsync += ErrorHandler;
 
                // start processing 
                await processor.StartProcessingAsync();
 
                while (processor.IsProcessing)
                {
                    System.Threading.Thread.Sleep(30000);
                }
 
                Console.WriteLine("Wait for a minute and then press any key for confirmation to end the processing");
                Console.ReadKey();
 
                // stop processing 
                Console.WriteLine("\nStopping the receiver...");
                await processor.StopProcessingAsync();
                Console.WriteLine("Stopped receiving messages");
            }
        }
 
        private static async Task MessageHandler(ProcessMessageEventArgs args)
        {
            string id = args.Message.MessageId;
            Console.WriteLine($"Received: {id} from subscription: {sourceSubscriptionName}");
 
            await SendMessageToTopicAsync(secondaryConnectionString, sourceTopicOrQueueName, args.Message);
            Console.WriteLine($"Sent: {id} from subscription: {sourceSubscriptionName}");
 
            // complete the message. messages is deleted from the queue. 
            await args.CompleteMessageAsync(args.Message);
        }
 
        private static Task ErrorHandler(ProcessErrorEventArgs args)
        {
            Console.WriteLine(args.Exception.ToString());
            return Task.CompletedTask;
        }
 
        private static async Task SendMessageToTopicAsync(string connectionString, string destinationTopicOrQueueName, ServiceBusReceivedMessage serviceBusReceivedMessage)
        {
            // create a Service Bus client 
            await using (ServiceBusClient client = new ServiceBusClient(connectionString))
            {
                // create a sender for the topic
                ServiceBusSender sender = client.CreateSender(destinationTopicOrQueueName);
                await sender.SendMessageAsync(new ServiceBusMessage(serviceBusReceivedMessage));
                Console.WriteLine($"Sent a single message to the topic: {destinationTopicOrQueueName}");
            }
        }
 
        private static async Task SendTestMessageToTopicAsync(string body)
        {
            // create a Service Bus client 
            await using (ServiceBusClient client = new ServiceBusClient(connectionString))
            {
                // create a sender for the topic
                ServiceBusSender sender = client.CreateSender(sourceTopicOrQueueName);
                await sender.SendMessageAsync(new ServiceBusMessage(body));
                Console.WriteLine($"Sent a single message to the topic: {sourceTopicOrQueueName}");
            }
        }
    }
}

Saturday, June 19, 2021

 Draining Service Bus 

Introduction: Service Bus is an Azure public cloud computing resource that acts as a message broker between publishers and subscribers. As a cloud resource, it can scale to arbitrary loads and become mission-critical. Its availability is improved by providing redundancy across regions so that when one goes down, it can failover to another. It comprises a variety of message holders such as Queues, Topics, Relays, and Event Hub listed within a namespace. They differ primarily in how they are used. For example, a queue enables one producer and consumer to send and receive ordered messages. A Topic allows a subscription so that many subscribers can receive messages. There can be several of each type of Service Bus entity and millions of messages in transit. This article describes the proper way to failover from one SB instance to another. 

Description: The availability of the Service Bus is further improved by its deployment to multiple low-latency availability zones in a geographical region. While the deployment across regions allows different instances to be provisioned, the deployments within a region across multiple availability zones are for the same instance. The service bus resource does not allow in-place enablement of zone redundancy to improve availability which requires some steps to be taken to preserve the structure and content of the original instance. There are no options to transfer both the structure and content at the same time and the only way to do that is to first replicate the structure on a target instance and then copy the data over after that.  This happens with the help of the logic that enumerates them and copies them one by one to the destination. The same logic applies to data migration. This control and data plane duplication can be custom-written into a single program, but it is better to leverage the built-in features of the resource that support the redundancy based on replicating the entities via internal automation. The steps for this migration can be listed as follows: Step 1: Create a new Premium Stock Keeping Unit (SKU) namespace. Step 2: Pair the source and destination namespaces with each other. Step 3: Sync or copy over the entities from the source to the destination namespace. Step 4: Commit the migration. Step 5: Drain entities from the source namespace using the post-migration name of the namespace Step 6: Delete the source namespace. Pairing a source namespace and destination namespace automatically copies over all the SB entities from the source namespace to the destination namespace. It is an in-build feature of the pairing mechanism of the Service Bus. The only catch in that replication is that pairing must be across regions and if we want to have another instance in the same region, we perform the structure and content migration to a new instance in a different region, break the pairing and then create a new pairing between that instance and a third instance back in the original region. Pairing does not replicate the messages that may still be held in the SB entities of the source namespace during the migration and just before it is completed. These messages must be drained. Each of the SB entities is enumerated and their messages read and copied to the corresponding entity in the destination namespace.  There are options in the programmability features of this Azure resource to automate the enumeration and transfer of messages. However, it is also possible to avoid writing this code. In such a case, a maintenance window must be in effect during the migration and the following steps will need to be taken. The sender applications are stopped. The receiver applications will process the messages currently in the source namespace and drain the queue. The queues and subscriptions in the source If the source namespace is empty, the migration steps listed earlier are performed. When the migration steps are complete, the sender applications may be restarted. The senders and receivers will now automatically connect with the destination namespace with the help of the alias that was set up for post-migration handling of the traffic from senders. This completes the structure and content replication of the Service Bus.


Friday, June 18, 2021

Azure monitoring continued...

 

Azure Monitoring also performs continuous monitoring which refers to processes and tools for monitoring each phase of the DevOps and IT operations lifecycles. It helps to continuously ensure the health, performance and reliability of the application and the infrastructure as it moves from deployment to production. It builds on Continuous Integration and Continuous deployment which are ubiquitously embraced by organizations for software development. Azure Monitoring is a unified monitoring solution that provides transparency to the application, runtime, host and cloud infrastructure layers. As a continuous monitoring tool, Azure Monitor allows gates and rollback of deployments based on monitoring data. Software releases to the services hosted in the cloud and have very short software development cycles and must pass through multiple stages and environments before it is made public.  Monitoring data allows any number of environments to be introduced without sacrificing the controls for software quality and gated release across environments. The data not only allows thresholds to be set but also alerts so that appropriate action may be taken. As the software makes its way to the final production environment, the alerts increase in levels and become more relevant and useful for eliminating risks from production environment.

It may be argued that tests and other forms of software quality control achieve that as the software goes through the CI/CD pipeline. While this is true, the software quality is enhanced by monitoring data because it is not intrusive or vulnerable to flakiness that many tests are prone to in different environments. The monitoring data, its visualization with dashboards need to be set only once even as the code and tests change over time. The investments in continuous monitoring and its implications boost the planning and predictability of software releases.

Monitoring has tremendous breadth and depth of scope, so a question about cost may arise. It can monitor individual Azure resources and it can monitor those resources for their availability, performance and operation.  Some monitoring data is collected by default and the monitoring data platform can be tapped to get more data from all participating monitoring agents. There is no cost associated with collecting, exporting and analyzing data by default. Costs might be associated with storage if a storage account is used or with ingestion if a workspace is used or with streaming if Azure Event Hubs are used. Monitoring involves a cost when running a long query, creating a metric or log query alert rule, sending a notification from any alert rule and accessing metrics through the API. At the resource level, the resource logs and platform metrics are automatically collected. At the subscription level, the activity log is automatically collected.