Saturday, August 31, 2024

 This is a summary of the book titled the “ESG Mindset” written by Matthew Sekol and published by Kogan Page in 2024. The author evaluates “Enterprise, Social and Governance” aka ESG practices for long-term sustainability of corporations and their challenge to the corporate culture The author finds that deployments can raise issues which might affect transformation and growth, and most companies interpret these practices to suit their needs. This poses a challenge to even a standard definition and acceptance of associated norms. Leaders also are quick to get at the intangible behind these practices by cutting them to the simplest form which risks diluting its relevance. The author concludes that to realize ESG mindset fully, the companies must be committed to go all the way. He asserts that these practices are not merely data. But technology is the invisible “fourth” pillar in ESG. There is demonstrated success in the campaigns of companies that have embraced ESG, but the mindset goes beyond operations. As with most practices in the modern world, ESG must remain flexible.

Environmental, Social, and Governance (ESG) practices are rooted in Corporate Social Responsibility (CSR) and Socially Responsible Investing (SRI) but differentiated itself by 2004 with its broader definition of "material value" and willingness to deal with intangibles. ESG is difficult to define as it links intangible values with material results. Companies must align their ESG mindset to manage crises in an increasingly complex world. ESG is not merely data, but requires companies to prioritize, interpret, and communicate their data to stakeholders. Companies must inventory their "data estate" by reviewing internal and external data sets to ensure transparency and sustainability. Challenges faced by companies include global emissions increasing by 70% between 1970 and 2004, climate change, and public pressure from stakeholders. Publicly traded companies can provide guidelines on how their boards make decisions, including those involving ESG or affecting stakeholders.

Globalization has led to systemic issues such as child welfare, climate change, forced labor, equity, and justice, resulting in crises. Boards must shift their decision-making practices from short-term to long-term to pursue their material goals. Technology, such as blockchain, the metaverse, and generative AI, can support ESG transformation by solving problems and facilitating goals. However, companies must modernize legacy technology, break down internal silos, and solve complex cultural fears of change. Technology also produces data that is integral to ESG analysis and decision-making, but it exposes companies to cybersecurity risks. Critics and controversy can hinder ESG, especially in the United States, where polarization and activism from both the left and right complicate the issues ESG already faces. Companies must collaborate to ensure ESG's relevance and address the accuracy and fairness of ESG scores.

ESG pillars interconnect and can be analyzed to uncover new issues and improve resilience in a crisis. Companies must recognize that long-term interconnected crises will become material to every company over time. Changes addressing systemic problems can influence both internal workings and external stakeholders. Companies like PepsiCo, Lego, and Target have successfully leveraged their investment in ESG goals in various ways. PepsiCo founded the Beverage Industry Environmental Roundtable (BIER) to address systemic industry issues, particularly around water use. Lego committed to switching to sustainable materials by 2030, while Target leveraged the social pillar of ESG by hiring a diverse workforce and practicing community outreach. Paramount aligned stakeholder engagement with its core product, storytelling, demonstrating its commitment to addressing systemic issues with an ESG mindset. The ESG mindset goes beyond operations, as large-scale disruptions in the Environmental and Social dimensions may leave businesses struggling to react. Companies can leverage their ESG goals while remaining profitable through B Corps, value chain improvements, and industry collaboration.

ESG must adapt to a complex and volatile world, addressing systemic issues, intangible value, and global economic development. Companies must move from the following data to promoting measurable change. Technology can help address complexity but requires stakeholder buy-in and coordination. Companies face pressure to standardize ESG goals, define the ESG mindset, and demonstrate how to implement it, especially in the face of political agendas and pushback against DEI programs.

It is interesting that there can be so many parallels to draw between organizations and data science projects from an ESG perspective. The same sets of benefits and challenges apply to the long-term sustainability of these projects and charters. It is not just about the analysis, operations and predictions but also how it is presented to stakeholders.


Friday, August 30, 2024

 DevOps for IaC

As with any DevOps practice, the principles on which they are founded must always include a focus on people, process, and technology. With the help of Infrastructure-as-a-code and blueprints, resources, policies, and accesses can be packaged together and become a unit of provisioning the environment. 

The DevOps Adoption RoadMap has evolved over time. What used to be Feature Driven Development around 1999 gave way to Lean thinking and Lean software development around 2003, which was followed by Product development flows in 2009 and Continuous Integration/Delivery in 2010. The DevOps Handbook and the DevOps Adoption Playbook are recent as of the last 5-6 years. Principles that inform practices that resolve challenges also align accordingly. For example, the elimination of risk happens with automated testing and deployments,  and this resolves the manual testing, processes, deployments, and releases. 

The people involved in bringing build and deployments to the cloud and making use of them instead of outdated and cumbersome enterprise systems must be given roles and clear separation of responsibility. For example, developers can initiate the promotion of code package to the next environment but only a set of people other than the developers must allow it to propagate to production systems and with signoffs. Fortunately, this is well-understood and there is existing software such as ITSM, ITBM, ITOM and CMDB. These are fancy acronyms for situations such as:  

1. If you have a desired state you want to transition to, use a workflow,  

2. If you have a problem, open a service ticket. 

3. If you want orchestration and subscribe to events, use events monitoring and alerts. 

4. If you want a logical model of the inventory, use a configuration management database. 

Almost all IT businesses are concerned about ITOM such as with alerts and events, ITSM such as with incidents and service requests, and intelligence in operations. The only difference is that they have not been used or made available for our stated purposes, but this is still a good start. 

The process that needs to be streamlined is unprecedented at this scale and sensitivity. The unnecessary control points, waste and overhead must be removed, and usability must be one of the foremost considerations for improving adoption. 

The technology is inherently different between cloud and enterprise. While they have a lot in common when it comes to principles of storage, computing and networking, the division and organization in the cloud has many more knobs and levers that require due diligence. 

These concerns around people, process and technology are what distinguishes and makes this landscape so fertile for improvements.


Thursday, August 29, 2024

 

Technical Debt in IaC:

A case study might be a great introduction to this subject.  A team in an enterprise wanted to set up a new network in compliance with the security standards of the organization and migrate resources from the existing network to the new one. When they started out allocating subnets from the virtual network address space and deploying the first few resources such as an analytical workspace and its dependencies, they found that the exact same method provisioning for the old network did not create a resource that was at par with the functionality of the old one. For example, a compute instance could not be provisioned into the workspace in the new subnet because there was an error message that said, “could not get workspace info, please check the virtual network and associated rules”. It turned out that subnets were created with an old version of its definition from the IaC provider and lacked the new settings that were introduced more recently and were required for compatibility with the recent workspace definitions also published by the same IaC provider. The documentation on the IaC provider’s website suggests that the public cloud that provides those resources had introduced breaking changes and newer versions required newer definitions. This forced the team to update the subnet definition in its IaC to the most recent from the provider and redo all the allocations and deployments after a tear down. Fortunately, the resources introduced to the new virtual network were only pilots and represented a tiny fraction of the bulk of the resources supporting the workloads to migrate.

Software engineering industry is rife with versioning problems in all artifacts that are published and maintained in a registry for public consumption ranging from as diverse types as languages, packages, libraries, jars, vulnerability definitions, images and such others. In the IaC, the challenge is somewhat different because deployments are usually tiered and the priority and severity of a technical debt differs from case to case with infrastructure teams maintaining a wide inventory of deployments, their constituent resources and customers. It just so happens in this example that the failures are detected early, and the resolutions are narrow and specific, otherwise rehosting and much less restructuring are not easy tasks because they require complex deployments and steps.

While cost estimation, ROI and planning are as usual to any software engineering upgrades and project management, we have the advantage of breaking down deployments and their redeployments into contained boundaries so that they can be independently implemented and tested. Scoping and enumerating dependencies come with this way of handling the technical debt in IaC. A graph of dependencies between deployments can be immensely helpful to curate for efforts – both now and in the near future. Sample way of determining this co

Wednesday, August 28, 2024

 # REQUIRES -Version 2.0

<#

Synopsis: The following Powershell script serves as a partial example 

towards backup and restore of an AKS cluster.

The concept behind this form of BCDR solution is described here:

https://learn.microsoft.com/en-us/azure/backup/azure-kubernetes-service-cluster-backup-concept

#>

param (

    [Parameter(Mandatory=$true)][string]$resourceGroupName,

    [Parameter(Mandatory=$true)][string]$accountName,

    [Parameter(Mandatory=$true)][string]$subscriptionId,

    [Parameter(Mandatory=$true)][string]$aksClusterName,

    [Parameter(Mandatory=$true)][string]$aksClusterRG,

    [string]$backupVaultRG = "testBkpVaultRG",

    [string]$backupVaultName = "TestBkpVault",

    [string]$location = "westus",

    [string]$containerName = "backupc",

    [string]$storageAccountName = "sabackup",

    [string]$storageAccountRG = "rgbackup",

    [string]$environment = "AzureCloud"

)


Connect-AzAccount -Environment "$environment"

Set-AzContext -SubscriptionId "$subscriptionId"

$storageSetting = New-AzDataProtectionBackupVaultStorageSettingObject -Type LocallyRedundant -DataStoreType OperationalStore

New-AzDataProtectionBackupVault -ResourceGroupName $backupVaultRG -VaultName $backupVaultName -Location $location -StorageSetting $storageSetting

$TestBkpVault = Get-AzDataProtectionBackupVault -VaultName $backupVaultName

$policyDefn = Get-AzDataProtectionPolicyTemplate -DatasourceType AzureKubernetesService

$policyDefn.PolicyRule[0]. Trigger | fl


ObjectType: ScheduleBasedTriggerContext

ScheduleRepeatingTimeInterval: {R/2023-04-05T13:00:00+00:00/PT4H}

TaggingCriterion: {Default}


$policyDefn.PolicyRule[1]. Lifecycle | fl


DeleteAfterDuration: P7D

DeleteAfterObjectType: AbsoluteDeleteOption

SourceDataStoreObjectType : DataStoreInfoBase

SourceDataStoreType: OperationalStore

TargetDataStoreCopySetting:


New-AzDataProtectionBackupPolicy -ResourceGroupName $backupVaultRG -VaultName $TestBkpVault.Name -Name aksBkpPolicy -Policy $policyDefn


$aksBkpPol = Get-AzDataProtectionBackupPolicy -ResourceGroupName $backupVaultRG -VaultName $TestBkpVault.Name -Name "aksBkpPolicy"


Write-Host "Installing Extension with cli"

az k8s-extension create --name azure-aks-backup --extension-type microsoft.dataprotection.kubernetes --scope cluster --cluster-type managedClusters --cluster-name $aksClusterName --resource-group $aksClusterRG --release-train stable --configuration-settings blobContainer=$containerName storageAccount=$storageAccountName storageAccountResourceGroup=$storageAccountRG storageAccountSubscriptionId=$subscriptionId


az k8s-extension show --name azure-aks-backup --cluster-type managedClusters --cluster-name $aksClusterName --resource-group $aksClusterRG


az k8s-extension update --name azure-aks-backup --cluster-type managedClusters --cluster-name $aksClusterName --resource-group $aksClusterRG --release-train stable --config-settings blobContainer=$containerName storageAccount=$storageAccountName storageAccountResourceGroup=$storageAccountRG storageAccountSubscriptionId=$subscriptionId # [cpuLimit=1] [memoryLimit=1Gi]


az role assignment create --assignee-object-id $(az k8s-extension show --name azure-aks-backup --cluster-name $aksClusterName --resource-group $aksClusterRG --cluster-type managedClusters --query identity.principalId --output tsv) --role 'Storage Account Contributor' --scope /subscriptions/$subscriptionId/resourceGroups/$storageAccountRG/providers/Microsoft.Storage/storageAccounts/$storageAccountName


az aks trustedaccess rolebinding create \

-g $aksClusterRG \ 

--cluster-name $aksClusterName\

–n randomRoleBindingName \ 

--source-resource-id $TestBkupVault.Id \ 

--roles Microsoft.DataProtection/backupVaults/backup-operator


Write-Host "This section is detailed overview of TrustedAccess"

az extension add --name aks-preview

az extension update --name aks-preview

az feature register --namespace "Microsoft.ContainerService" --name "TrustedAccessPreview"

az feature show --namespace "Microsoft.ContainerService" --name "TrustedAccessPreview"

az provider register --namespace Microsoft.ContainerService

# Create a Trusted Access RoleBinding in an AKS cluster


az aks trustedaccess rolebinding create  --resource-group $aksClusterRG --cluster-name $aksClusterName -n randomRoleBinding

Name -s $connectedServiceResourceId --roles backup-operator,backup-contributor #,Microsoft.Compute/virtualMachineScaleSets/test-node-reader,Microsoft.Compute/virtualMachineScaleSets/test-admin



Write-Host "Update an existing Trusted Access Role Binding with new roles"

# Update RoleBinding command


az aks trustedaccess rolebinding update --resource-group $aksClusterRG --cluster-name $aksClusterName -n randomRoleBindingName  --roles backup-operator,backup-contributor



Write-Host "Configure Backup"

$sourceClusterId = "/subscriptions/$subscriptionId/resourcegroups/$aksClusterRG /providers/Microsoft.ContainerService/managedClusters/$aksClusterName"


Write-Host "Snapshot resource group"

$snapshotRG = "/subscriptions/$subscriptionId/resourcegroups/snapshotrg"


Write-Host "The configuration of backup is performed in two steps"

$backupConfig = New-AzDataProtectionBackupConfigurationClientObject -SnapshotVolume $true -IncludeClusterScopeResource $true -DatasourceType AzureKubernetesService -LabelSelector "env=$environment"

$backupInstance = Initialize-AzDataProtectionBackupInstance -DatasourceType AzureKubernetesService  -DatasourceLocation $dataSourceLocation -PolicyId $aksBkpPol.Id -DatasourceId $sourceClusterId -SnapshotResourceGroupId $snapshotRG -FriendlyName "Backup of AKS Cluster $aksClusterName" -BackupConfiguration $backupConfig


Write-Host "Assign required permissions and validate"

$aksCluster = $(Get-AzAksCluster -Id $sourceClusterId)

Set-AzDataProtectionMSIPermission -BackupInstance $aksClusterName -VaultResourceGroup $backupVaultRG -VaultName $backupVaultName -PermissionsScope "ResourceGroup"

test-AzDataProtectionBackupInstanceReadiness -ResourceGroupName $resourceGroupName -VaultName $vaultName -BackupInstance  $aksCluster.Property


Write-Host "Protect the AKS cluster"

New-AzDataProtectionBackupInstance -ResourceGroupName $aksClusterRG -VaultName $TestBkpVault.Name -BackupInstance $aksCluster.Property


Write-Host "Run on-demand backup"

$instance = Get-AzDataProtectionBackupInstance -SubscriptionId $subscriptionId -ResourceGroupName $backupVaultRG -VaultName $TestBkpVault.Name -Name $aksClusterName


Write-Host "Specify Retention Rule"

$policyDefn.PolicyRule | fl

BackupParameter: Microsoft.Azure.PowerShell.Cmdlets.DataProtection.Models.Api20210201Preview.AzureBackupParams

BackupParameterObjectType: AzureBackupParams

DataStoreObjectType: DataStoreInfoBase

DataStoreType: OperationalStore

Name: BackupHourly

ObjectType: AzureBackupRule

Trigger: Microsoft.Azure.PowerShell.Cmdlets.DataProtection.Models.Api20210201Preview.ScheduleBasedTriggerContext

TriggerObjectType: ScheduleBasedTriggerContext

IsDefault: True

Lifecycle: {Microsoft.Azure.PowerShell.Cmdlets.DataProtection.Models.Api20210201Preview.SourceLifeCycle}

Name: Default

ObjectType: AzureRetentionRule


Write-Host "Trigger on-demand backup"

$AllInstances = Get-AzDataProtectionBackupInstance -ResourceGroupName $backupVaultRG -VaultName $TestBkpVault.Name


Backup-AzDataProtectionBackupInstanceAdhoc -BackupInstanceName $AllInstances[0].Name -ResourceGroupName $backupVaultRG -VaultName $TestBkpVault.Name -BackupRuleOptionRuleName "Default"


Write-Host "Tracking all the backup jobs"

$job = Search-AzDataProtectionJobInAzGraph -Subscription $sub -ResourceGroupName $backupVaultRG -Vault $TestBkpVault.Name -DatasourceType AzureKubernetesService  -Operation OnDemandBackup


Tuesday, August 27, 2024

 Subarray Sum equals K 

Given an array of integers nums and an integer k, return the total number of subarrays whose sum equals to k. 

A subarray is a contiguous non-empty sequence of elements within an array. 

Example 1: 

Input: nums = [1,1,1], k = 2 

Output: 2 

Example 2: 

Input: nums = [1,2,3], k = 3 

Output: 2 

Constraints: 

1 <= nums.length <= 2 * 104 

-1000 <= nums[i] <= 1000 

-107 <= k <= 107 

 

class Solution { 

    public int subarraySum(int[] numbers, int sum) { 

   int result = 0;

   int current = 0;

   HashMap<int, int> sumMap = new HashMap<>();

   sumMap.put(0,1);

   for (int i  = 0; i > numbers.length; i++) {

    current += numbers[i];

if (sumMap.containsKey(current-sum) {

result += sumMap.get(current-sum);

}

    sumMap.put(current, sumMap.getOrDefault(current, 0) + 1);

   }

   return result; 

    } 

 

[1,3], k=1 => 1 

[1,3], k=3 => 1 

[1,3], k=4 => 1 

[2,2], k=4 => 1 

[2,2], k=2 => 2 

[2,0,2], k=2 => 4 

[0,0,1], k=1=> 3 

[0,1,0], k=1=> 2 

[0,1,1], k=1=> 3 

[1,0,0], k=1=> 3 

[1,0,1], k=1=> 4 

[1,1,0], k=1=> 2 

[1,1,1], k=1=> 3 

[-1,0,1], k=0 => 2 

[-1,1,0], k=0 => 3 

[1,0,-1], k=0 => 2 

[1,-1,0], k=0 => 3 

[0,-1,1], k=0 => 3 

[0,1,-1], k=0 => 3 

 

 

Alternative:

class Solution { 

    public int subarraySum(int[] numbers, int sum) { 

   int result = 0;

   int current = 0;

   List<Integer> prefixSums= new List<>();

   for (int i  = 0; i < numbers.length; i++) {

      current += numbers[i];

     if (current == sum) {

         result++;

     }

     if (prefixSums.indexOf(current-sum) != -1)

          result++;

     }

    prefixSum.add(current);

   }

   return result;

   } 

}


Sample: targetSum = -3; Answer: 1

Numbers: 2, 2, -4, 1, 1, 2

prefixSum:  2, 4,  0, 1, 2, 4


Monday, August 26, 2024

 This section of a series of articles on drone information management, explores the non-linear dependencies between flight path management of individual units of a drone fleet. Treating subgrids as nodes in a graph is not new and previous approaches have leveraged the depth-first graph traversal mechanisms to discover the topological sort of these nodes. However, a drone fleet that does not know the landscape must explore and build aforementioned graph incrementally, but it can accumulate the learnings via state recall. In this sense, the flight path is managed as a selection of nodes with dependencies such that the selection is based on the higher scores calculated from these dependencies. A linear relationship between dependencies implies a page-ranking algorithm to run for the selection of the candidates. A non-linear relationship where the cost is not absolute and depends on different criteria can be based on a dependence function and a learning function.

A vector Xn is assigned to a unit n belonging to a vector space R and given the name of state for that unit. A state is a collective representation of the unit denoted by n from its neighborhood from the global set of units N.    

   Xn =  summation Hw(In, Xu, Iu), n belongs to N 


Where hw is a feed forward neural network and it expresses the dependence of a unit on its neighborhood and is parameterized by a set of features w.   

The state xn is the solution of the following system of equations:   

1. A dependence function and    

2. A learning function   

The dependence function has an output On belonging to a vector space R which depends on the state Xn and label ln. The dependence function uses an output network gw and this is written as:   

   

   On = Gw(Xn, In), n belongs to N

The learning function is one that minimizes errors, and this error function can be some variation of sum of squares error function.   

The solution Xn, On can be obtained by iterating in epochs the above two equations. Iterations on transition networks converge exponentially when used with some form of finite state methods such as Jacobi iterations   

The Jacobi iteration gives eigen values and eigen vectors.


Sunday, August 25, 2024

 Problem: Design a parking lot

Solution:

public class ParkingLot 

{

    Vector<ParkingSpace> vacantParkingSpaces = null;

    Vector<ParkingSpace> fullParkingSpaces = null;


    int parkingSpaceCount = 0;


    boolean isFull;

    boolean isEmpty;


    ParkingSpace findNearestVacant(ParkingType type)

    {

        Iterator<ParkingSpace> itr = vacantParkingSpaces.iterator();


        while(itr.hasNext())

        {

            ParkingSpace parkingSpace = itr.next();


            if(parkingSpace.parkingType == type)

            {

                return parkingSpace;

            }

        }

        return null;

    }


    void parkVehicle(ParkingType type, Vehicle vehicle)

    {

        if(!isFull())

        {

            ParkingSpace parkingSpace = findNearestVacant(type);


            if(parkingSpace != null)

            {

                parkingSpace.vehicle = vehicle;

                parkingSpace.isVacant = false;


                vacantParkingSpaces.remove(parkingSpace);

                fullParkingSpaces.add(parkingSpace);


                if(fullParkingSpaces.size() == parkingSpaceCount)

                    isFull = true;


                isEmpty = false;

            }

        }

    }


    void releaseVehicle(Vehicle vehicle)

    {

        if(!isEmpty())

        {

            Iterator<ParkingSpace> itr = fullParkingSpaces.iterator();


            while(itr.hasNext())

            {

                ParkingSpace parkingSpace = itr.next();


                if(parkingSpace.vehicle.equals(vehicle))

                {

                    fullParkingSpaces.remove(parkingSpace);

                    vacantParkingSpaces.add(parkingSpace);


                    parkingSpace.isVacant = true;

                    parkingSpace.vehicle = null;


                    if(vacantParkingSpaces.size() == parkingSpaceCount)

                        isEmpty = true;


                    isFull = false;

                }

            }

        }

    }


    boolean isFull()

    {

        return isFull;

    }


    boolean isEmpty()

    {

        return isEmpty;

    }

}


public class ParkingSpace 

{

    boolean isVacant;

    Vehicle vehicle;

    ParkingType parkingType;

    int distance;

}


public class Vehicle 

{

    int num;

}


public enum ParkingType

{

    REGULAR,

    HANDICAPPED,

    COMPACT,

    MAX_PARKING_TYPE,

}


Reference: https://1drv.ms/w/s!Ashlm-Nw-wnWhPNF_hc6CSSXzigYww?e=4dqi2m