Chaos engineering

Administrators will find that this section is familiar to them.  There are technical considerations that emphasize design and service-level objectives, as well as the scale of the solution. But drills, testing and validations are essential to guarantee smooth operations. Chaos engineering is one such method to test the reliability of an infrastructure solution.  While reliability of individual resources might be a concern for the public provider, that of the deployments fall on the IaC authors and deployers. As a contrast from other infrastructure concerns such as security that has mitigation in theory and design involving Zero Trust and least privilege principles, Chaos Engineering is all about trying it out with drills and controlled environments.
By the time a deployment of infrastructure is planned, business considerations have included the following:
understanding what kind of solution is being created such as business-to-business, business-to-consumer, or enterprise software 2. Defining the tenants in terms of number and growth plans, 3. Defining the pricing model and ensuring it aligns with the tenants’ consumption of Azure resources. 4. Understanding whether we need to separate the tenants into different tiers and based on the customer’s requirements, deciding on the tenancy model. 
And a well-architected review would have addressed the key pillars of 1) Reliability, 2) Security, 3) Cost Optimization, 4) Operational Excellence, and 5) Performance efficiency. Performance is usually based on external factors and is very close to customer satisfaction. Continuous telemetry and reactiveness are essential to tuned up performance.

Security and reliability are operation concerns. When trying out the deployments for testing reliability, it is often required to inject faults and bring down components to check how the remaining part of the system behaves. The idea of injection of failure also finds parallels in beefing up security in the form of penetrative testing.   The difference is that security testing is geared towards exploitation while reliability testing is geared toward reducing mean time between failures.

The component-down testing quite drastic which involves powering down the zone. There are a lot of network connections to and from cloud resources, so it becomes hard to find an alternative to a component that is down. A multi-tiered approach is necessitated to enable robustness against component-down design. Mitigation often involve workarounds and diverting traffic to healthy redundancies.
Having multi-region deployments of components not only improves reliability but also draws business from the neighborhood of the new presence. A geographical presence for a public cloud is only somewhat different from that of a private cloud. A public cloud lists regions where the services it offers are hosted. A region may have three availability zones for redundancy and availability and each zone may host a variety of cloud computing resources – small or big. Each availability zone may have one or more stadium sized datacenters. When the infrastructure is established, the process of commissioning services in the new region can be referred to as buildouts. Including appropriate buildouts increases reliability of the system in face of failures.
Component-down testing for chaos engineering differs from business continuity and disaster recovery planning in that one discovers problems in reliability and the other determines acceptable mitigation. One cannot do without the other.

For a complete picture on reliability of infrastructure deployments, additional considerations become necessary. These include:

First, the automation must involve context switching between the platform and the task for deploying each service to the region. The platform co-ordinates these services and must maintain an order, dependency and status during the tasks.
Second, the task of each service itself is complicated and requires definitions in terms of region-specific parameters to an otherwise region agnostic service model.
Third, the service must manifest their dependencies declaratively so that they can be validated and processed correctly. These dependencies may be between services, on external resources and the availability or event from another activity.
Fourth, the service buildouts must be retry-able on errors and exceptions otherwise the platform will require a lot of manual intervention which increase the cost
Fifth, the communication between the automated activities and manual interventions must be captured with the help of the ticket tracking or incident management system
Sixth, the workflow and the state for each activity pertaining to the task must follow standard operating procedures that are defined independent of region and available to audit
Seventh, the technologies for the platform execution and that for the deployments of the services might be different requiring consolidation and coordination between the two. In such case, the fewer the context switches between the two the better.
Eighth, the platform itself must have support for templates, event publishing and subscription, metadata store, onboarding and bootstrapping processes that can be reused.
Ninth, the platform should support parameters for enabling a region to be differentiated from others or for customer satisfaction in terms of features or services available.
Tenth, the progress for the buildout of new regions must be actively tracked with the help of tiles for individual tasks and rows per services.

#codingexercise: CodingExercise-10-24-2024.docx

 This is a summary of the book titled “Palaces for the people” written by Eric Klinenberg and published by Crown in 2019. The author explains “How social infrastructure can help fight Inequality, Polarization, and the Decline of Civic Life”. The term social infrastructure refers to shared public spaces such as libraries, parks and coffee shops. His examples include a hurricane shelter in Houston that was made out from a church and vacant lots in Chicago that was converted into an urban farm. He is upbeat about innovative ways to improve, expand and maintain such structures and refers to them as an indicator of the health of a community and how it avoids crime. Libraries are a must in his inclusive list of such infrastructure. He lists threats such as commercial development that brings gentrification, absence of such infrastructure that leads to poorer health of already suffering communities, and shortcomings in disaster preparation. He aims this treatise exclusively for the United States.

 Social infrastructure is crucial for a community's health and well-being, as it includes buildings, public spaces, parks, libraries, and coffee shops. For example, public libraries offer a free, open place for people to socialize and volunteer organizations to meet. Libraries serve as bedrocks of civil society and offer responsibility and independence for young people. However, many libraries suffer from neglect due to reduced funding and services.

Social infrastructure can also play a role in preventing crime. Cities often focus on individual offenders, but experts suggest that focusing on the environments where crime flourishes, such as empty lots and abandoned buildings, may be more effective. Philadelphia researchers found a 39% drop in gun violence in areas around repaired structures and a 5% reduction in gun violence in vacant lots.

Research by landscape architect William Sullivan and environmental scientist Frances Kuo found that vegetation provides social benefits, such as lower crime rates in areas around buildings. Building prisons for the poor has been a main crime reduction policy, but the social costs have been as great as the economic expenses.

Commercial development can lead to increased property values and rents even as it displaces residents, and causes a decrease in crime rates. Yale professor Andrew Papachristos found a correlation between the number of new coffee shops and a reduction in local murder rates, regardless of the neighborhood's residents. However, street robberies rates declined in primarily white and Latino neighborhoods but tended to rise in gentrifying neighborhoods with primarily Black residents. This suggests that gentrification is not a viable anticrime strategy due to its social costs.

Social infrastructure is crucial for the health of people in poorer communities, as opioid addiction has reached epidemic levels in small and rural areas. Modern infrastructure, such as reliable power, clean water, fast transit, affordable food, and resilient structures, has improved public health more than any other modern intervention. In some areas, community activists and social entrepreneurs have turned vacant lots into urban agriculture, providing fresh, healthy food from farmer’s markets, fostering social ties, and reducing stress levels. A robust social infrastructure provides opportunities for the elderly to socialize and stay active, as seen in Singapore's high-rise complexes.

Social infrastructure is crucial in times of disaster, as societies worldwide invest trillions in hard infrastructure to cope with storms, floods, heat, drought, and migration. Religious groups and community organizations play a vital role in recovery from disasters, such as Hurricane Harvey and Hurricane Sandy. Policymakers are seeking creative ways to construct protective systems that double as social infrastructure, such as the Potomac Park Levee in Washington, DC, and Singapore's Marina Barrage and Reservoir project in Singapore. Grassroots organizations have devised innovative schemes for adapting to high waters, such as Bangladesh's "floating schools and libraries" program. Investors, including PayPal founder Peter Thiel, are backing floating-city concepts. The United States must rebuild its social infrastructure, as most significant infrastructure is a product of state funding or private philanthropy. Social infrastructure can determine the number of opportunities for meaningful social interactions and can make the difference between life and death during crises.

 

Count the number of array slices in an array of random integers such that the difference between the min and the max values in the slice is <= k

public static int getCountOfSlicesWithMinMaxDiffGEk(int[] A, int k){

        int N = A.length;

        int[] maxQ = new int[N+1];

        int[] posmaxQ = new int[N+1];

        int[] minQ = new int[N+1];

        int[] posminQ = new int[N+1];

        int firstMax = 0, lastMax = -1;

        int firstMin = 0, lastMin = -1;

        int j = 0, result = 0;

        for (int i = 0; i < N; i++) {

            while (j < N) {

                while(lastMax >= firstMax && maxQ[lastMax] <= A[j]) {

                    lastMax -= 1;

                }

                lastMax += 1;

                maxQ[lastMax] = A[j];

                posmaxQ[lastMax] = j;

 

                while (lastMin >= firstMin && minQ[lastMin] >= A[j]) {

                    lastMin -= 1;

                }

                lastMin += 1;

                minQ[lastMin] = A[j];

                posminQ[lastMin] = j;

                if (maxQ[firstMax] - minQ[firstMin] <= k) {

                    j += 1;

                } else {

                    break;

                }

            }

            System.out.println("result="+result + " i=" + i + " j="+ j);

            result += (j-i);

            if (result >= Integer.MAX_VALUE) {

                result = Integer.MAX_VALUE;

            }

            if (posminQ[firstMin] == i) {

                firstMin += 1;

            }

            if (posmaxQ[firstMax] == i) {

                firstMax += 1;

            }

        }

        return result;

    }

A: 3,5,7,6,3 K=2

result=0 i=0 j=2

result=2 i=1 j=4

result=5 i=2 j=4

result=7 i=3 j=4

result=8 i=4 j=5

9

 

 

 Managing copilots:

This section of the series on cloud infrastructure deployments focuses on the proliferation of copilots for different business purposes and internal processes. As with any flight, a copilot is of assistance only to the captain responsible for the flight to be successful. If the captain does not know where she is going, then the copilot immense assistance will still not be enough. It is of secondary importance that the data that a copilot uses might be prone to bias or shortcomings and might even lead to so-called hallucinations for the copilot. Copilots are after all large language models that work entirely on treating data as vectors and leveraging classification, regression and vector algebra to respond to queries. They don’t build a knowledge graph and do not have the big picture on what business purpose they will be applied to. If the purpose is not managed properly, infrastructure engineers might find themselves maintaining many copilots for different use cases and even reducing the benefits of where one would have sufficed.

Consolidation of large language models and their applications to different datasets is only the tip of the iceberg that these emerging technologies have provided as instruments for the data scientists. Machine Learning pipelines and applications are as diverse and silo’ed as the datasets that they operate on and they are not always present in data lakes or virtual warehouses. Consequently, a script or a prediction api written and hosted as an application does not make the best use of infrastructure for customer convenience in terms of interaction streamlining and improvements in touch points. This is not to say that different models cannot be used or that the resources don’t need to proliferate or that there are some cost savings with consolidation, it is about business justification of the number of copilots needed. When we work backwards from what the customer benefits or experiences, one of the salient features that works in favor of infrastructure management is that less is more. Hyperconvergence of infrastructure for various business purposes when those initiatives are bought into by stakeholders that have both business and technical representations makes the investments more deliberate and fulfilling.

And the cloud or the infrastructure management is not restrictive to experimentation, just that it is arguing against the uncontrolled experimentation and placing the customers in a lab. As long as experimentation and initiatives can be specific in terms of duration, budget and outcomes, infrastructure management can go the extra mile of cleaning up, decommissioning and even repurposing so that technical and business outcomes go hand in glove.

Processes are hard to establish when the technology is emerging and processes are also extremely difficult to enforce as new standards in the workplace. The failure of six sigma and the adoption of agile technologies are testament to that. However, best practices for copilot engineering are easier to align with cloud best practices and current software methodologies in most workplaces.

Reference: Previous articles on Infrastructure management: https://1drv.ms/w/s!Ashlm-Nw-wnWhPZmOhdL7Y5aiDLb6g?e=gsr9g4

 This is a summary of the book titled “Who built that?” written by Michelle Malkin and published by Simon and Schuster in 2015. The author collects biographies of highly prolific but lesser-known American inventors and uses it to argue against President Obama’s assertion in 2012 that business owners need help from government-funded program. Although there are political ideas and interpretations, this collection of mini-biographies is an interesting read about free market capitalism. American inventors and investors have changed the world. The profit motive and US Patent law serves them well as they become “tinkerpreneurs”. For example, Tony Maglica’s patented Maglite revolutionized flashlight design in 1978. Inventor Willis Carrier and marketer Irvine Lyle developed and sold air conditioning and refrigeration technologies. The Roebling family changed wire rope manufacturing and improved bridge building. Success stories for tinkerpreneurs often begins as an accumulation of individual efforts and fostered by a free market society. Toilet papers, inexpensive razors and other disposable products highlight this. Partnership between inventors and industrialists illustrate perseverance paired with capitalism. Modern advances in prosthetics shows how free enterprise supports invention. The only difference between earlier and now is a switch from “first to invent” to “first to file” mindset.

American innovation is threatened by changes to US patent laws in 2011 and "wealth shaming" by the political left. It is important to celebrate invention and remember the unsung heroes who helped make the United States a powerhouse of innovation. The work of US inventors in the 19th and 20th centuries testifies to the power of creativity, innovation, and the supportive environment backed by "American exceptionalism." Thanks to US patent laws and the free market economy, these creative, industrious inventors prospered from their work.

The Maglite, invented by Tony Maglica, revolutionized flashlight construction with its tough metal body, adjustable lighting, and superior design. Maglica attributes his success to America's strong patent laws, which provide the basis for defending against intellectual property theft.

The fathers of modern air conditioning, Willis Carrier, and Irvine Lyle began their journey to entrepreneurial success and technological innovation in 1902 by attempting to prevent multicolored printing jobs from bleeding in New York's summer heat. Their discoveries led to breakthroughs in the development of lifesaving pharmaceuticals, such as the polio vaccine, penicillin, and streptomycin.

The Roebling family, led by Johann (John) Augustus Roebling, was pioneers in innovation and entrepreneurship. Roebling immigrated to America in 1831 to escape Prussia's government control of engineering projects. He began his tinkerpreneurship by designing and patenting improvements for steam-powered machines. Roebling's first major "aha" moment came while working on a Pennsylvania canal, where he thought about replacing weak hemp ropes with stronger wire ropes. He was the first to create machines to make the rope uniform and sturdy, producing it with limited manpower. Roebling's first successful project using patented wire was a suspension aqueduct into Pittsburgh in 1845. He famously spanned Niagara Falls in 1855 and built the Covington-Cincinnati Bridge in 1867 with his son, Washington. The Roebling family stands as a testament to their nation's unprecedented ideas and ambition.

US Patent 4,286,311, filed by Tony Maglica in 1978, marked the beginning of a new era of heavy, rugged flashlights. Other small innovations, such as toilet paper, disposable razor blades, crown-type bottle caps, and fuller's earth, were created by inventors who recognized the need for practical solutions. These inventions created continual demand by keeping their products cheap and disposable.

Some of the most dramatic inventions emerged from collaboration, with inventors often forming partnerships with other inventors or visionary industrialists who provided financial backing and marketing support. For example, Nikola Tesla developed the alternating current (AC) used in light sockets in America without the help of inventor-industrialist George Westinghouse

Manufactured weather, such as Willis Carrier and Irvine Lyle, was largely influenced by industrialist Edward Libbey. They helped develop and patent glassmaking machines, transforming the industry from a handcraft controlled by labor unions to safe, cheap, fully automated production. Libbey and Owens fought against obstructionists and anticapitalists, transforming the world's relationship with glass and combating dangerous child labor practices.

The American creative spirit, which inspired past inventors, continues to thrive in fields like prosthetics. Modern American tinkerpreneurs build on past breakthroughs, such as A.A. Marks, Albert Winkley, and Edward Hanger. American "free enterprise" fosters invention, with companies like Bally Ribbon Company and BrainGate creating robotic limbs. However, the 2011 America Invents Act (AIA) is a special-interest boondoggle that enriches corporate lawyers, big business, and federal bureaucrats at the expense of independent inventors and innovators. The shift from "first to invent" to "first to file" patent laws favors multinational corporations and turns patent law against "small" inventors, the nation's most productive and creative members. Repealing the AIA is crucial, as opportunity and freedom are the key to promoting innovation in the future.

#Codingexercise: https://1drv.ms/w/s!Ashlm-Nw-wnWhNQ6C15Thp-sCFzgag?e=E4pnej

 This is a summary of the book titled “Healthcare disrupted” written by Jeff Elton and Anne O’Riordan and published by Wiley in 2016. The authors are business consultants who assert that the new business models are inevitable, and they will be about curing people and not pushing pills. Although written a few years ago and prior to the change in government policies, their writing continues to be thought provoking. When the industry is set to reach $18 trillion in spending in 2030, costs will be incurred more from “fee-for-service” model rather than anything else. The new health care model emphasizes health over disease and value over volume. Reformers who are focused on results-based healthcare say providers should earn based on outcomes. There are some challenges from patients’ behavior as well which makes certain diseases like diabetes difficult to treat. Other emerging business models include “Lean Innovators”, “Around the patient innovators”, and “Value Innovators”. Lean Innovators who are typically makers of generic drugs, also invent products. Patient space includes apps, sensors, and other technology into their offerings. Value innovators are proposing that treatment begins at patients’ home. It is more effective for costly conditions like heart failure and diabetes.

The health care revolution is characterized by patients becoming active consumers, leading to increased healthcare costs and a shift towards a value-based model. The "fee-for-service" model, which incentivizes medical professionals to provide procedures, drugs, and devices in the most expensive scenarios, is causing health costs to soar. This flawed approach has led to reforms such as quality reports about physicians and the US Medicare system processing half of its payments using performance measures by 2018. Accountable care organizations (ACOs) have also contributed to this shift. Health care companies operate on a value curve that progresses through four stages: "Simple product," "Enhanced product," "Integrated services," and "Living services." The final frontier of the value curve, the "final frontier," involves offering an array of services, with patient outcomes partially determining payment. This shift is challenging due to legal and regulatory constraints.

Pharmaceutical firms known as "Lean Innovators" are combining generic drugs with innovative products to avoid patent-expiration problems and focus on niche products. They operate in area A of the value curve, selling products but not delving far beyond that level. Lean Innovators are rooted in the generic drug industry, selling cheaper alternatives, and embracing supply chain efficiency. Examples include Teva Pharmaceutical Industries, Allergan PLC, and Valeant Pharmaceuticals International. They typically grow through acquisitions and have a lower cost of sales and R&D than big pharma companies. They can post EBITDAs that exceed their big pharma competitors, with companies like Allergan, Teva, and Valeant posting EBITDAs of 35.6% in 2014.

Around-the-Patient Innovators are companies that focus on addressing patients' lifestyle challenges rather than just selling a basic product. They invest in talent and research, such as Johnson & Johnson and Novartis, and aim to evolve with the changing healthcare market. These companies aim to provide a broader value proposition and partner with companies like Apple, Google, and Qualcomm to fill gaps in their offerings. They operate in areas B and C of the value curve and must nimbly innovate while maintaining their legacy businesses. For example, Novartis' Entresto, a heart failure treatment, requires patients to monitor their blood pressure, change their diets, and maintain activity levels to maintain effectiveness. By focusing on these aspects, Around-the-Patient Innovators can help improve patient outcomes and reduce the need for prescription drugs.

The healthcare industry is transitioning from a traditional model of specialty therapeutics, geographic regions, and settings to a new model emphasizing health over disease and value over volume. Value Innovators, life sciences companies, are pushing into areas C and D of the value curve. Boston Scientific, for example, is focusing on treating congestive heart failure, a costly condition that requires high patient compliance. Medtronic, a device maker, focuses on data and patient monitoring to manage costs and keep chronically ill patients out of expensive facilities. However, achieving real-world success is challenging due to the longer time horizon involved in treating the chronically ill. The future of healthcare will see care becoming untethered from traditional locations, with patients playing a bigger role in the decision-making process. The industry will redefine medicines and care, moving from an intervention-based model to an ongoing mode of managing patients' health.

#codingexercise: https://1drv.ms/w/s!Ashlm-Nw-wnWhNM_tgTe4304lDjcuw?e=rS1wVr

 This is a continuation of a series of articles on IaC shortcomings and resolutions. In this section, we discuss ways to transfer data from one Azure managed instance of Apache Cassandra server in a virtual network to another in a different network. The separation in terms of network for the Cassandra resource type only serves to elaborate on the steps needed to generalize the data transfer.

Data is organized in the Cassandra cluster as keyspaces and tables. The first approach is the direct approach using a command-line client like cqlsh to interact with the clusters. The steps are download the tables as csv files and upload them to the other server.

Example:

Step 1. At source server:

USE <keyspace>;

COPY <keyspace>.<table_name> TO 'path/to/file.csv' WITH HEADER = true;

Step 2. At destination server:

USE <keyspace>;

CREATE TABLE <table_name> (

    column1 datatype1,

    column2 datatype2,

    ...

    PRIMARY KEY (column1)

);

COPY <keyspace>.<table_name> (column1, column2, ...) FROM 'path/to/file.csv' WITH HEADER = true;

The other option is to read the data from one server and without a local artifact save the data to the destination. An example for this would appear as follows:

This option involves running a copy activity on a Databricks notebook using Apache Spark:

Example:

from pyspark.sql import SparkSession

# Initialize the Spark session

spark = SparkSession.builder \

    .appName("Copy Cassandra Data") \

    .config("spark.cassandra.connection.host", "<source-cassandra-host>") \

    .config("spark.cassandra.connection.port", "9042") \

    .config("spark.cassandra.auth.username", "<source-username>") \

    .config("spark.cassandra.auth.password", "<source-password>") \

    .getOrCreate()

# List of keyspaces and tables to copy

keyspaces = ["keyspace1", "keyspace2"]

tables = ["table1", "table2"]

for keyspace in keyspaces:

    for table in tables:

        # Read data from the source Cassandra cluster

        df = spark.read \

            .format("org.apache.spark.sql.cassandra") \

            .options(keyspace=keyspace, table=table) \

            .load()

        # Write data to the target Cassandra cluster

        df.write \

            .format("org.apache.spark.sql.cassandra") \

            .options(

                keyspace=keyspace,

                table=table,

                "spark.cassandra.connection.host"="<target-cassandra-host>",

                "spark.cassandra.connection.port"="9042",

                "spark.cassandra.auth.username"="<target-username>",

                "spark.cassandra.auth.password"="<target-password>"

            ) \

            .mode("append") \

            .save()

# Stop the Spark session

spark.stop()

Note, however, that we had started out with the source and destination in different networks. So, if the databricks server is also tethered to the same network as one of the servers, it will not be able to reach the other server. One way to get around that involves peering the network but that usually affects other resources and is not always a possibility.Another option involves adding private endpoints but the source and destination might have been connected to a delegated subnet ruling out that option. Consequently, we must include an additional step to a third location as an intermediary for data transfer that both networks can access such as a storage account over public IP networking.

This would require an example as follows:

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

import os

# Set up the Spark session

spark = SparkSession.builder \

    .appName("Export Cassandra to Azure Storage") \

    .config("spark.cassandra.connection.host", "<cassandra-host>") \

    .config("spark.cassandra.connection.port", "9042") \

    .config("spark.cassandra.auth.username", "<username>") \

    .config("spark.cassandra.auth.password", "<password>") \

    .getOrCreate()

# Define the Azure Storage account details

storage_account_name = "<storage-account-name>"

storage_account_key = "<storage-account-key>"

container_name = "<container-name>"

# Configure the storage account

spark.conf.set(f"fs.azure.account.key.{storage_account_name}.blob.core.windows.net", storage_account_key)

# Define keyspaces and tables to export

keyspaces = ["keyspace1", "keyspace2"]

tables = ["table1", "table2"]

# Export each table to CSV and upload to Azure Storage

for keyspace in keyspaces:

    for table in tables:

        # Read data from Cassandra

        df = spark.read \

            .format("org.apache.spark.sql.cassandra") \

            .options(keyspace=keyspace, table=table) \

            .load()

        # Define the output path

        output_path = f"wasbs://{container_name}@{storage_account_name}.blob.core.windows.net/{keyspace}/{table}.csv"

        # Write data to CSV

        df.write \

            .csv(output_path, header=True, mode="overwrite")

# Stop the Spark session

spark.stop()

Lastly, it does not matter whether an agent or an intermediary stash is used for the data transfer, but the size and the number of tables do matter for the reliability of the transfer especially if the connection or the execution can be interrupted. Choosing between the options requires us to make the copying logic robust.