Data Backup
from relational databases in the cloud.
Many Azure
Infrastructure deployments that involve a MySQL or MSSQL server instance will
tend to rely on the features available from the cloud resource for backup and
recovery. While these resources support continuous replication with another
instance and scheduled backups on the same instance, there is no feature
available to backup individual databases. Application-engineering teams that
rely on the data in a single database must rely on customized automation for
this purpose. Infrastructure engineering teams that deploy these database
servers for the benefit of these application engineering teams have a wide
range of options to write such an automation. Most of them rely on leveraging
the command line utilities from the database publishers with the two most
popular on Azure being MySQL or MSSQL. A command line utility like mysqldump
can provide automation to backup and restore individual databases and most
automation relies on archiving those backup files on storage accounts or s3
storage that are designed to be durable. Depending on where the application
engineering teams host these applications, the customizations could leverage
the same hosts for creating backups and restores. For example, an
infrastructure using Azure Kubernetes Service to host the applications If the
application engineering teams prefer GitOps over Azure DevOps to transmit the
secrets to the command-line utility, one option is to host the automation as GitHub
Actions. This article demonstrates how to do that with two files: one for a
GitHub Action workflow that deploys an AKS job to backup or restore a single
database and another describing the AKS job. The automation can also
parameterize the hosts, database source and destination, and storage accounts
so that the process can be repeated for various consumers.
name:
"MySql Backup Restore"
on:
push:
branches:
- main
paths:
- 'mysqlconfig/U2-Prepay-Non-Prod/**'
defaults:
run:
shell: bash
#Enforing to
use OIDC for authentication
permissions:
actions: read
checks: read
contents: read
deployments: read
id-token: write
issues: read
discussions: read
packages: read
pages: read
pull-requests: write
repository-projects: read
security-events: read
statuses: read
jobs:
pre-deploy:
name: Prepay NonProd
runs-on: [ uhg-runner ]
environment: prod
strategy:
matrix:
subscriptions: [${{ input.subscription
}}]
env:
ARM_CLIENT_ID: ${{
secrets.AZURE_CLIENT_ID }}
ARM_USE_OIDC: true
ARM_SUBSCRIPTION_ID: ${{
input.subscription }}
ARM_TENANT_ID: ${{
secrets.AZURE_TENANT_ID }}
GH_PAT: ${{ secrets.GH_PAT }}
steps:
#Login to Azure
- name: 'Az CLI login'
uses: azure/login@v1
with:
client-id: ${{
secrets.AZURE_CLIENT_ID }}
tenant-id: ${{
secrets.AZURE_TENANT_ID }}
subscription-id: ${{
input.subscription }}
- name: 'Action Checkout'
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: 'Git Config URL'
run: git config --global
url."https://${{ secrets.GH_PAT }}@github.com".insteadOf
https://github.com
- name: 'Setup working Directory'
run: |
WORK_DIR="./"
echo
"WORK_DIR=${WORK_DIR}" >> $GITHUB_ENV
echo "Current working
directory is $WORK_DIR"
- name: Get git changes
id: changes
run: |
jsonfile=$(git diff --name-only
--diff-filter=A ${{ github.event.before }} ${{ github.event.after }} | grep
.json$ | head -1 | xargs)
echo "$jsonfile"
githubjson=$(cat ${jsonfile})
echo $githubjson
DB_HOST=`echo $(jq -r '.DB_HOST'
<<< "$githubjson")`
echo "DB_HOST=$DB_HOST"
>> $GITHUB_ENV
DB_USER=`echo $(jq -r '.DB_USER'
<<< "$githubjson")`
echo "DB_USER=$DB_USER"
>> $GITHUB_ENV
DB_NAME=`echo $(jq -r '.DB_NAME'
<<< "$githubjson")`
echo "DB_NAME=$DB_NAME"
>> $GITHUB_ENV
KV_NAME=`echo $(jq -r '.KV_NAME'
<<< "$githubjson")`
echo "KV_NAME=$KV_NAME"
>> $GITHUB_ENV
KV_SECRET_NAME=`echo $(jq -r
'.KV_SECRET_NAME' <<< "$githubjson")`
echo
"KV_SECRET_NAME=$KV_SECRET_NAME" >> $GITHUB_ENV
KV_BLOB_CONN_STR=`echo $(jq -r
'.KV_BLOB_CONN_STR' <<< "$githubjson")`
echo
"KV_BLOB_CONN_STR=$KV_BLOB_CONN_STR" >> $GITHUB_ENV
FILE_NAME=`echo $(jq -r '.FILE_NAME'
<<< "$githubjson")`
echo "FILE_NAME=$FILE_NAME"
>> $GITHUB_ENV
BACKUP_RESTORE=`echo $(jq -r
'.BACKUP_RESTORE' <<< "$githubjson")`
echo
"BACKUP_RESTORE=$BACKUP_RESTORE" >> $GITHUB_ENV
AKS_RG=`echo $(jq -r '.AKS_RG'
<<< "$githubjson")`
echo "AKS_RG=$AKS_RG"
>> $GITHUB_ENV
AKS_NAME=`echo $(jq -r '.AKS_NAME'
<<< "$githubjson")`
echo "AKS_NAME=$AKS_NAME"
>> $GITHUB_ENV
AKS_NAMESPACE=`echo $(jq -r
'.AKS_NAMESPACE' <<< "$githubjson")`
echo
"AKS_NAMESPACE=$AKS_NAMESPACE" >> $GITHUB_ENV
BLOB_CONTAINER_NAME=`echo $(jq -r
'.BLOB_CONTAINER_NAME' <<< "$githubjson")`
echo
"BLOB_CONTAINER_NAME=$BLOB_CONTAINER_NAME" >> $GITHUB_ENV
EMAIL_RECIPIENTS=`echo $(jq -r
'.EMAIL_RECIPIENTS' <<< "$githubjson")`
echo
"EMAIL_RECIPIENTS=$EMAIL_RECIPIENTS" >> $GITHUB_ENV
EMAIL_API=`echo $(jq -r '.EMAIL_API'
<<< "$githubjson")`
echo "EMAIL_API=$EMAIL_API"
>> $GITHUB_ENV
- name: Setup kubectl
id: install-kubectl
uses: Azure/setup-kubectl@v3
- name: Setup kubelogin
id: install-kubelogin
uses: azure/use-kubelogin@v1
with:
kubelogin-version: 'v0.0.32'
- name: Deploy Job to AKS
id: aksdeploy
run: |
az account set
--subscription=$ARM_SUBSCRIPTION_ID
DB_PASSWORD=$(az keyvault secret show
--name $KV_SECRET_NAME --vault-name $KV_NAME --query "value")
DB_PASSWORD=$(echo
"$DB_PASSWORD" | sed -e 's/[\/&]/\\&/g')
echo
"DB_PASSWORD=$DB_PASSWORD" >> $GITHUB_ENV
BLOB_CONNECTION=$(az keyvault secret
show --name $KV_BLOB_CONN_STR --vault-name $KV_NAME --query "value")
echo
"BLOB_CONNECTION=$BLOB_CONNECTION" >> $GITHUB_ENV
az aks get-credentials
--resource-group $AKS_RG --name $AKS_NAME --overwrite-existing --admin
if kubectl get job
mysql-backup-job -n $AKS_NAMESPACE >
/dev/null 2>&1; then
echo 'deleting existing job......'
kubectl delete jobs
mysql-backup-job -n $AKS_NAMESPACE
else
echo 'no job to delete, moving
on....'
fi
kubelogin convert-kubeconfig -l
azurecli
sed -i -e
"s#%%HOST%%#$DB_HOST#" ./aks-job.yaml;
sed -i -e
"s#%%USER%%#$DB_USER#" ./aks-job.yaml;
sed -i -e
"s#%%PASS%%#$DB_PASSWORD#" ./aks-job.yaml;
sed -i -e
"s#%%DB%%#$DB_NAME#" ./aks-job.yaml;
sed -i -e
"s#%%RFILE%%#$FILE_NAME#" ./aks-job.yaml;
sed -i -e
"s#%%BACRST%%#$BACKUP_RESTORE#" ./aks-job.yaml;
sed -i -e
"s#%%CONNSTR%%#$BLOB_CONNECTION#" ./aks-job.yaml;
sed -i -e
"s#%%CNTNAME%%#$BLOB_CONTAINER_NAME#" ./aks-job.yaml;
sed -i -e
"s#%%EMAIL%%#$EMAIL_RECIPIENTS#" ./aks-job.yaml;
sed -i -e
"s#%%EMAILAPI%%#$EMAIL_API#" ./aks-job.yaml;
sed -i -e
"s#%%AKSNS%%#$AKS_NAMESPACE#" ./aks-job.yaml;
kubectl apply -f ./aks-job.yaml
aks-job.yaml:
apiVersion:
batch/v1
kind: Job
metadata:
name: mysql-backup-job
namespace: %%AKSNS%%
spec:
template:
metadata:
labels: {}
spec:
containers:
- env:
- name: DB_HOST
value: %%HOST%%
- name: DB_USER
value: %%USER%%
- name: DB_PASSWORD
value: %%PASS%%
- name: DB_NAME
value: %%DB%%
- name: BLOB_MNT
value: .
- name: RESTORE_FILE
value: %%RFILE%%
- name: BACKUP_RESTORE
value: %%BACRST%%
- name: BLOB_CONNECTION_STRING
value: %%CONNSTR%%
- name: BLOB_CONTAINER_NAME
value: %%CNTNAME%%
- name: EMAIL_RECIPIENTS
value: %%EMAIL%%
- name: EMAIL_API
value: %%EMAILAPI%%
image:
mycontainerregistry.azurecr.io/mysql-backup-restore
imagePullPolicy: Always
name: mysql-backup-job
resources: {}
restartPolicy: Never
Previous
articles: IaCResolutionsPart175.docx
No comments:
Post a Comment