Database migration between Cassandra servers

This section describes the way to transfer data between the Azure Managed Instances of Apache Cassandra. This is done with the help of “nodetool” and “sstableloader” commands via the Azure CLI using the steps outlined below. For more information on the DBA commands for this instance, please visit https://learn.microsoft.com/en-us/azure/maaged-instance-apache-cassandra/dba-commands. Ensure that your automation identity has the necessary permission to run azcopy and az cli.

1. The first step involves creating a snapshot of the data. This is done with:

# Create a snapshot

az managed-cassandra cluster invoke-command \

  --resource-group <resource-group> \

  --cluster-name <cluster-name> \

  --host <host-ip> \

  --command-name nodetool \

  --arguments "snapshot <keyspace> -t <snapshot-name>"

# Download the snapshot files using azcopy

azcopy copy "https://<storage-account>.blob.core.windows.net/<container>/<path-to-snapshot>/*" "<local-directory>"

2. The second step involves uploading the snapshot to another instance. This is done with:

# Upload the snapshot files using azcopy

azcopy copy "<local-directory>/*" "https://<storage-account>.blob.core.windows.net/<container>/<path-to-upload>"

# Load the data using sstableloader

az managed-cassandra cluster invoke-command \

  --resource-group <resource-group> \

  --cluster-name <cluster-name> \

  --host <host-ip> \

  --command-name sstableloader \

  --arguments "-d <data-directory> <keyspace> <table>"

3. Finally, test that the data are similar.

Data Backup/Restore or Migration are usually considered last resort as most resources in Azure allow continuous replication across paired servers. But when they become inevitable, these steps make the task easier.

 Why are Enterprise Application registrations and Service principals’ creation excluded from IaC?

When we create an Azure App Registration, it results in the creation of both an Application Object and a Service Principal Object. The application object is the representation of the application in the home tenant and acts as a template or blueprint for creating service principals. It contains information about how the application can issue tokens, the resources it can access and the actions it can perform. One can configure API permissions, client secrets, branding, and App roles within this object definition.

A service principal aka Enterprise Application is a concrete instance of the application object within a specific tenant. It represents the application’s identity and comes with a unique object identifier that can be used for non-interactive automations. Each tenant where the application is used will have its own service principal. Most role-based access control assignments for the application are made using this object identifier.

It is possible to register the application using IaC but there are several drawbacks due to which they are not popular. For example, there are special permissions needed for the identity used with the pipeline to deploy such a registration in the Azure Entra aka Active Directory. Such permissions are usually difficult to distribute and must remain within the exclusive control of a centralized entity within the organization which is usually named the Identity and Access Management team. When the permissions are not sufficient, the idempotency of the resource creation and deletion is lost. At this point, the syntax and semantics of app registration appear to be broken and for this reason, it is difficult for anyone to create application registrations from code deploying to their subscriptions. All such registrations will end up in the same global active directory.

Another reason is that the provider registrations and the Active directory must remain up to date otherwise version discrepancies affect the IaC. Since the resource is global and not specific to the azure subscription-resource-group-and-resource tri-level hierarchy, the provider can change the syntax and semantics for the global entity that do not get captured with the IaC that once worked and does not later. For this reason, it is important that azure Active Directory resources not be treated the same as resources that instantiate services from Azure’s service portfolio

Previous article: https://1drv.ms/w/s!Ashlm-Nw-wnWhPVXO5Lemyhk5z34HA?e=8e2jVb

 

Data Backup from relational databases in the cloud.

Many Azure Infrastructure deployments that involve a MySQL or MSSQL server instance will tend to rely on the features available from the cloud resource for backup and recovery. While these resources support continuous replication with another instance and scheduled backups on the same instance, there is no feature available to backup individual databases. Application-engineering teams that rely on the data in a single database must rely on customized automation for this purpose. Infrastructure engineering teams that deploy these database servers for the benefit of these application engineering teams have a wide range of options to write such an automation. Most of them rely on leveraging the command line utilities from the database publishers with the two most popular on Azure being MySQL or MSSQL. A command line utility like mysqldump can provide automation to backup and restore individual databases and most automation relies on archiving those backup files on storage accounts or s3 storage that are designed to be durable. Depending on where the application engineering teams host these applications, the customizations could leverage the same hosts for creating backups and restores. For example, an infrastructure using Azure Kubernetes Service to host the applications If the application engineering teams prefer GitOps over Azure DevOps to transmit the secrets to the command-line utility, one option is to host the automation as GitHub Actions. This article demonstrates how to do that with two files: one for a GitHub Action workflow that deploys an AKS job to backup or restore a single database and another describing the AKS job. The automation can also parameterize the hosts, database source and destination, and storage accounts so that the process can be repeated for various consumers.

name: "MySql Backup Restore"

on:

  push:

    branches:

      - main

    paths:

      - 'mysqlconfig/U2-Prepay-Non-Prod/**'

defaults:

  run:

    shell: bash

 

#Enforing to use OIDC for authentication

permissions:

  actions: read

  checks: read

  contents: read

  deployments: read

  id-token: write

  issues: read

  discussions: read

  packages: read

  pages: read

  pull-requests: write

  repository-projects: read

  security-events: read

  statuses: read

 

jobs:

  pre-deploy:

    name: Prepay NonProd

    runs-on: [ uhg-runner ]

    environment: prod

    strategy:

      matrix:

        subscriptions: [${{ input.subscription }}]

    env:

      ARM_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}

      ARM_USE_OIDC: true

      ARM_SUBSCRIPTION_ID: ${{ input.subscription }}

      ARM_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}

      GH_PAT: ${{ secrets.GH_PAT }}

    steps:

      #Login to Azure

      - name: 'Az CLI login'

        uses: azure/login@v1

        with:

          client-id: ${{ secrets.AZURE_CLIENT_ID }}

          tenant-id: ${{ secrets.AZURE_TENANT_ID }}

          subscription-id: ${{ input.subscription }}

         

      - name: 'Action Checkout'

        uses: actions/checkout@v3

        with:

          fetch-depth: 0   

      - name: 'Git Config URL'

        run: git config --global url."https://${{ secrets.GH_PAT }}@github.com".insteadOf https://github.com

      - name: 'Setup working Directory'

        run: |

             WORK_DIR="./"

             echo "WORK_DIR=${WORK_DIR}" >> $GITHUB_ENV

             echo "Current working directory is $WORK_DIR"   

         

      - name: Get git changes

        id: changes

        run: |

          jsonfile=$(git diff --name-only --diff-filter=A ${{ github.event.before }} ${{ github.event.after }} | grep .json$ | head -1 | xargs)

          echo "$jsonfile"

 

          githubjson=$(cat ${jsonfile})

 

          echo $githubjson

 

          DB_HOST=`echo $(jq -r '.DB_HOST' <<< "$githubjson")`

          echo "DB_HOST=$DB_HOST" >> $GITHUB_ENV

 

          DB_USER=`echo $(jq -r '.DB_USER' <<< "$githubjson")`

          echo "DB_USER=$DB_USER" >> $GITHUB_ENV

 

          DB_NAME=`echo $(jq -r '.DB_NAME' <<< "$githubjson")`

          echo "DB_NAME=$DB_NAME" >> $GITHUB_ENV

 

          KV_NAME=`echo $(jq -r '.KV_NAME' <<< "$githubjson")`

          echo "KV_NAME=$KV_NAME" >> $GITHUB_ENV

 

          KV_SECRET_NAME=`echo $(jq -r '.KV_SECRET_NAME' <<< "$githubjson")`

          echo "KV_SECRET_NAME=$KV_SECRET_NAME" >> $GITHUB_ENV

 

          KV_BLOB_CONN_STR=`echo $(jq -r '.KV_BLOB_CONN_STR' <<< "$githubjson")`

          echo "KV_BLOB_CONN_STR=$KV_BLOB_CONN_STR" >> $GITHUB_ENV

 

          FILE_NAME=`echo $(jq -r '.FILE_NAME' <<< "$githubjson")`

          echo "FILE_NAME=$FILE_NAME" >> $GITHUB_ENV

 

          BACKUP_RESTORE=`echo $(jq -r '.BACKUP_RESTORE' <<< "$githubjson")`

          echo "BACKUP_RESTORE=$BACKUP_RESTORE" >> $GITHUB_ENV

 

          AKS_RG=`echo $(jq -r '.AKS_RG' <<< "$githubjson")`

          echo "AKS_RG=$AKS_RG" >> $GITHUB_ENV

 

          AKS_NAME=`echo $(jq -r '.AKS_NAME' <<< "$githubjson")`

          echo "AKS_NAME=$AKS_NAME" >> $GITHUB_ENV

 

          AKS_NAMESPACE=`echo $(jq -r '.AKS_NAMESPACE' <<< "$githubjson")`

          echo "AKS_NAMESPACE=$AKS_NAMESPACE" >> $GITHUB_ENV

         

          BLOB_CONTAINER_NAME=`echo $(jq -r '.BLOB_CONTAINER_NAME' <<< "$githubjson")`

          echo "BLOB_CONTAINER_NAME=$BLOB_CONTAINER_NAME" >> $GITHUB_ENV

 

          EMAIL_RECIPIENTS=`echo $(jq -r '.EMAIL_RECIPIENTS' <<< "$githubjson")`

          echo "EMAIL_RECIPIENTS=$EMAIL_RECIPIENTS" >> $GITHUB_ENV

 

          EMAIL_API=`echo $(jq -r '.EMAIL_API' <<< "$githubjson")`

          echo "EMAIL_API=$EMAIL_API" >> $GITHUB_ENV

 

      - name: Setup kubectl

        id: install-kubectl

        uses: Azure/setup-kubectl@v3

 

      - name: Setup kubelogin

        id: install-kubelogin

        uses: azure/use-kubelogin@v1

        with:

          kubelogin-version: 'v0.0.32'

 

      - name: Deploy Job to AKS

        id: aksdeploy

        run: |

          az account set --subscription=$ARM_SUBSCRIPTION_ID

         

          DB_PASSWORD=$(az keyvault secret show --name $KV_SECRET_NAME --vault-name $KV_NAME --query "value")

          DB_PASSWORD=$(echo "$DB_PASSWORD" | sed -e 's/[\/&]/\\&/g')

          echo "DB_PASSWORD=$DB_PASSWORD" >> $GITHUB_ENV

 

          BLOB_CONNECTION=$(az keyvault secret show --name $KV_BLOB_CONN_STR --vault-name $KV_NAME --query "value")

          echo "BLOB_CONNECTION=$BLOB_CONNECTION" >> $GITHUB_ENV

 

          az aks get-credentials --resource-group $AKS_RG --name $AKS_NAME --overwrite-existing --admin

         

          if kubectl get job mysql-backup-job  -n $AKS_NAMESPACE > /dev/null 2>&1; then

             echo 'deleting existing job......'

             kubectl delete jobs mysql-backup-job  -n $AKS_NAMESPACE

          else

             echo 'no job to delete, moving on....'

          fi

 

          kubelogin convert-kubeconfig -l azurecli

          sed -i -e "s#%%HOST%%#$DB_HOST#" ./aks-job.yaml;

          sed -i -e "s#%%USER%%#$DB_USER#" ./aks-job.yaml;

          sed -i -e "s#%%PASS%%#$DB_PASSWORD#" ./aks-job.yaml;

          sed -i -e "s#%%DB%%#$DB_NAME#" ./aks-job.yaml;

          sed -i -e "s#%%RFILE%%#$FILE_NAME#" ./aks-job.yaml;

          sed -i -e "s#%%BACRST%%#$BACKUP_RESTORE#" ./aks-job.yaml;

          sed -i -e "s#%%CONNSTR%%#$BLOB_CONNECTION#" ./aks-job.yaml;

          sed -i -e "s#%%CNTNAME%%#$BLOB_CONTAINER_NAME#" ./aks-job.yaml;

          sed -i -e "s#%%EMAIL%%#$EMAIL_RECIPIENTS#" ./aks-job.yaml;

          sed -i -e "s#%%EMAILAPI%%#$EMAIL_API#" ./aks-job.yaml;

          sed -i -e "s#%%AKSNS%%#$AKS_NAMESPACE#" ./aks-job.yaml;

          kubectl apply -f ./aks-job.yaml

 

aks-job.yaml:

apiVersion: batch/v1

kind: Job

metadata:

  name: mysql-backup-job

  namespace: %%AKSNS%%

spec:

  template:

    metadata:

      labels: {}

    spec:

      containers:

      - env:

        - name: DB_HOST

          value: %%HOST%%

        - name: DB_USER

          value: %%USER%%

        - name: DB_PASSWORD

          value: %%PASS%%

        - name: DB_NAME

          value: %%DB%%

        - name: BLOB_MNT

          value: .

        - name: RESTORE_FILE

          value: %%RFILE%%

        - name: BACKUP_RESTORE

          value: %%BACRST%%

        - name: BLOB_CONNECTION_STRING

          value: %%CONNSTR%%

        - name: BLOB_CONTAINER_NAME

          value: %%CNTNAME%%

        - name: EMAIL_RECIPIENTS

          value: %%EMAIL%%

        - name: EMAIL_API

          value: %%EMAILAPI%%

        image: mycontainerregistry.azurecr.io/mysql-backup-restore

        imagePullPolicy: Always

        name: mysql-backup-job

        resources: {}

      restartPolicy: Never

 

Previous articles: IaCResolutionsPart175.docx

 This is a summary of the book titled “Next! The power of reinvention in life and work” written by Joanne Lipman and published by Mariner Books in 2023. Transformation, whether from external circumstances or from internal motivation, does not have to be stressful. The author draws from remarkable transformations of both people and products from interviews and scientific research to uncover the process. Major transitions follow a four-part pattern. Gut instinct always feels right. Past failures can be learning. If something does not suit well, we forge a path ahead. Someone who believes in us can help us clarify our goals. If a company wants to make a transition, look for outside perspectives.

Major career transitions follow a four-part pattern: "Search," "Struggle," "Stop," and "Solution." The COVID pandemic prompted millions to rethink their careers, but most significant changes happen incrementally. Research and personal anecdotes reveal a common four-phase pattern: "Search," "Struggle," "Stop," and "Solution."

Deciding to make a career change is often the most challenging part, but it's important to listen to your gut instinct. Gut instincts can guide you in the right direction, stemming from your experience and knowledge. For example, James Patterson, a famous author, used his experience as an adman before becoming a best-selling novelist to appeal to customers and provide what they wanted. By following this pattern, individuals can successfully navigate their career or business transformations.

Past failures can help individuals grow and succeed by focusing on the learning process rather than the final results. Success often comes from going from failure to failure without losing enthusiasm. People who succeed after failure often search for reasons for their failures and work through the adjustment process to find a breakthrough idea. For example, figure skater Nathan Chen readjusted his mentality after losing a gold medal at the 2018 Olympics.

To learn from failure, reflect on the struggle and make small incremental changes. Allow for creative insights when searching for your next step, as decisive breakthroughs, or "aha moments," often emerge when you do something else entirely. Cognitive neuroscientists have found that during an "aha moment," the subconscious brain connects increments of unrelated or distantly related information to create a solution. To tap into the power of subconscious connections, distract yourself and create an incubation time for your subconscious brain to process your research and make connections.

Many people find themselves in need of a new career, often due to societal obstacles. Women, especially those with children, are more likely to embrace reinvention than men, creating companies that empower marginalized people. Women-owned firms have started to change the workplace into a more accepting, flexible environment.

To make a career- or life-changing transformation, it is important to prepare and lay down the groundwork before jumping into a new job. JP Morgan executive Will Brown took over 20 years to transition from being a Wall Street economist to a full-time farmer. Steve Jobs and Whitney Wolfe Herd have also made significant changes without consciously knowing it.

Trust that the struggle you are in will lead to better things and let it become a time of inspiration, research, and discovery. Find someone who believes in you to help clarify your goals. Ina Garten, for example, made her transition from working as a nuclear budget analyst to starting her culinary career by impulsively buying a 400-square-foot food shop in Westhampton Beach, New York.

Ina Garten, a successful chef, transitioned from a White House nuclear budget analyst to starting her culinary career with her husband's encouragement. This person helped her gain a professional reputation, attract celebrity clients, and expand her shops. When she decided to sell the shops, her husband's guidance helped her create best-selling cookbooks and start her TV career. It just takes one person who believes in you to help clarify your goals. Having someone who champions your strengths and provides a clear perspective can make all the difference in navigating tough career moves. Sharing your goals with trusted people can hold you accountable and support you throughout the process. If your company plans to make a transition, look for outside perspectives to guide the way. Innovative ideas rarely come from the C-suite, but outsiders may see potential where insiders can't. Companies in transition often benefit from outside perspectives, as they may see potential where insiders can't.

 Problem statement:

A message containing letters from A-Z can be encoded into numbers using the following mapping:

'A' -> "1"

'B' -> "2"

...

'Z' -> "26"

To decode an encoded message, all the digits must be grouped then mapped back into letters using the reverse of the mapping above (there may be multiple ways). For example, "11106" can be mapped into:

• "AAJF" with the grouping (1 1 10 6)

• "KJF" with the grouping (11 10 6)

Note that the grouping (1 11 06) is invalid because "06" cannot be mapped into 'F' since "6" is different from "06".

Given a string s containing only digits, return the number of ways to decode it.

The test cases are generated so that the answer fits in a 32-bit integer.

Example 1:

Input: s = "12"

Output: 2

Explanation: "12" could be decoded as "AB" (1 2) or "L" (12).

Example 2:

Input: s = "226"

Output: 3

Explanation: "226" could be decoded as "BZ" (2 26), "VF" (22 6), or "BBF" (2 2 6).

Example 3:

Input: s = "06"

Output: 0

Explanation: "06" cannot be mapped to "F" because of the leading zero ("6" is different from "06").

Constraints:

• 1 <= s.length <= 100

• s contains only digits and may contain leading zero(s).

class Solution {

    public int numDecodings(String s) {

        Integer count = 0;

        if (isValid(s.substring(0,1)))

            traverse(s.substring(1), count);

        if (s.length() >= 2 && isValid(s.substring(0,2)))

            traverse(s.substring(2), count);

        return count;

    }

    public boolean traverse(String s, Integer count) {

        if (String.isNullOrWhitespace(s)){

            count += 1;

            return true;

        }

        if (isValid(s.substring(0,1)))

            traverse(s.substring(1), count);

        if (s.length() >= 2 && isValid(s.substring(0,2)))

            traverse(s.substring(2), count);

        return count > 0;

    }

    public boolean isValid(String s) {

        if (s.length() == 1 && s.charAt(0) >= '0' && s.charAt(0) <= '9'){

            return true;

        }

        if (s.length() == 2 &&

           (s.charAt(0) > '0' && s.charAt(0) <= '2') &&

           ((s.charAt(0) == '1' && s.charAt(1) >= '0' && s.chartAt(1) <= '9') ||

            (s.charAt(0) == '2' && s.chartAt(1) >= '0' && s.chartAt(1) <= '6')) {

            return true;

        }

        return false;

    }

}

 

Just-in-time (JIT) access, also known as just-in-time privileged access management (JIT PAM), is a security approach that grants privileged access or permissions only for the finite moments needed. It eliminates always-on, persistent privileged access, known as "standing privileges." On the other hand,  Just Enough Access aka JEA model is essential for implementing the principle of least privilege. But "true least privilege" requires combining both models, so that organizations can minimize potential attackers' footholds and the paths to privilege that could escalate an attack. However, many enterprises struggle with having too many accounts with unnecessary privileges, standing access status quo, privilege blindness, and lack of context around privileged risk. By combining these approaches, organizations can significantly reduce the attack surface and minimize potential vulnerabilities. Some of the malpractices include deploying too many accounts with unnecessary privileges, permissions, and entitlements, a standing access status quo, privileged blindness, and lack of context around privileged risk.

In Amazon Web Services (AWS), limiting human access to cloud resources is crucial for security. AWS offers tools like AWS Identity and Access Management (IAM) and AWS IAM Identity Center for managing access. Granting just-in-time access to developers for a limited time based on approval is an effective way to limit active time frames for assignments to AWS resources. Okta's integration with IAM Identity Center allows customers to access AWS using their Okta identities. As an example, the roles could correspond to different job functions within your organization. For example, the “AWS EC2 Admin” role could correspond to a DevOps on-call site reliability engineer (SRE) lead, whereas the “AWS EC2 Read Only” role may apply to members of your development team. The step-by-step configuration for this involves setting up groups representing different privilege levels, enabling automatic provisioning of groups using SCIM protocol, assigning access for groups in Okta, creating permissions sets in IAM identity center, assign group access in your AWS organization, configuring Okta identity governance access requests and finally testing the configuration. Okta's integration with AWS minimizes persistent access assignments, granting access just in time for specific operational functions. This solution allows empty user groups to be assigned to highly-privileged AWS permissions, with Okta Access Requests controlling group membership duration.

In Azure, Conditional Access templates provide a convenient method to deploy new policies aligned with Microsoft recommendations. These templates are designed to provide maximum protection aligned with commonly used policies across various customer types and locations. The templates are organized into secure foundation, zero trust, remote work, protect administrator, and emerging threats. Certain accounts must be excluded from these templates such as emergency-access or break-glass accounts to prevent tenant-wide account lockout and some service accounts and service principals that are non-interactive and tied to any particular user.

 Principle of Just-in-Time (JIT) privileged access:

This is a security model used in Azure public cloud to grant temporary permissions to users for performing privileged tasks. This approach helps minimize the risk of unauthorized access by ensuring that elevated permissions are only available when needed and for a limited time. Users receive elevated permissions only for the duration necessary to complete specific tasks. Once the time expires, the permissions are revoked automatically. A dedicated service in Azure services portfolio by the name Azure AD Privileged Identity Management (PIM)  manages JIT access, allowing administrators to control and monitor privileged access to Azure resources and Azure AD. PIM can generate alerts for suspicious or unsafe activities, enhancing security monitoring. This is commonly used for administrative tasks, accessing sensitive data, or managing critical infrastructure.

Amazon Web Services aka AWS also supports something similar with its Privileged Access Management aka PAM solutions where third-party solutions can be integrated into the AWS to provide ephemeral JIT access, ensuring that users only have the necessary privileges for the duration of their tasks. AWS provides  regular fine-grained permissions for users, groups and roles with its Identity and Access Management policies which can even be used to restrict access to a certain time of the day. The single sign-on service can work with different identity providers to enforce JIT access. Finally, the AWS Security Token Service can issue temporary security credentials that provide limited time access to AWS resources.

To bolster the physical security, reducing the risk of malware or unauthorized access, streamlining and restricting activities that can be performed with the escalation of privilege, Microsoft hands out Secure Admin Workstations (SAWs) that are specialized and dedicated devices used exclusively for administrative tasks. They are particularly valuable in high-risk environments where security is paramount. Public clouds happen to be the most widely used cloud but there are other clouds that can be dedicated in scope specifically for governments, defense departments and those that require tighter access control and these are collectively called sovereign clouds. These clouds are especially benefited with SAW devices. Only authorized personnel can use SAWs, and they are often subject to strict security policies and monitoring. As an example, Microsoft uses approximately 35,000 SAW devices, with a small number dedicated to accessing these high-risk environments aka sovereign clouds.

These practices help ensure that Azure remains a secure platform for both administrators and users.