AKS - Airflow setup and use with SSO
Here’s a step-by-step guide to deploying and using Apache Airflow on Azure Kubernetes Service (AKS) [1]:
Step 1: Set Up Your AKS Cluster
If you don’t already have an AKS cluster, create one using IaC. This should already be done for you. Log into the Aks cluster with
az aks get-credentials --resource-group <resourcegroupname> --name <clustername>
kubelogin convert-kubeconfig -l azurecli
Step 2: Install Helm
Ensure Helm is installed on your local machine:
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
Step 3: Create a Namespace for Airflow
kubectl create namespace airflow
Step 4: Configure Workload Identity (Optional but Recommended)
This step allows Airflow to securely access Azure resources like Key Vault:
1. Create a service account:
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: airflow
namespace: airflow
EOF
1. Annotate the service account with your Azure identity:
kubectl annotate serviceaccount airflow \
azure.workload.identity/client-id=<CLIENT_ID> \
azure.workload.identity/tenant-id=<TENANT_ID> \
-n airflow
Step 5: Install External Secrets Operator (Optional for Key Vault Integration)
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets \
--namespace airflow \
--create-namespace \
--set installCRDs=true \
--wait
Step 6: Add the Apache Airflow Helm Chart
helm repo add apache-airflow https://airflow.apache.org
helm repo update
Step 7: Install Airflow
a kustomization is preferred:
or using helm
helm install airflow apache-airflow/airflow \
--namespace airflow \
--set executor=CeleryExecutor \
--set airflow.image.tag=2.8.1 \
--set createUser=true \
--set webserver.defaultUser.username=admin \
--set webserver.defaultUser.password=admin
You can get Fernet key with
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
and optionally save it as a secret
kubectl get secret --namespace airflow airflow-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode
I prefer creating a HelmRelease with the official airflow chart with the release file in the references and creating configMaps for values file also in the references and the webserver_config.py discussed a few steps below if you want to specify SSO during setup.
Step 8: Access the Airflow Web UI
Port-forward the web server service:
kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow
Then open your browser and go to: http://localhost:8080
For a complete setup, it could look like this: PIBI NonProd Airflow
Step 9: Setup the app registration for SSO
Add the following:
1. redirect URI as https://<your-airflow-domain>/oauth-authorized/azure or https://<your-airflow-domain>/oauth2/callback
2. Assign API permissions (openid, email, profile) for authentication.
3. Navigate to the token configuration page of the Azure AD application. For ID and access token, add an optional claim on the
1. email
2. preferred_username
3. given_name
4. family_name
5. UPN
4. Edit the groups claim to include sAMAccountName for both ID and Access tokens but leave out SAML.
5. Specify federated identity entries for use with your GitHub repository.
6. Optionally, create App roles on your Azure AD application such as airflow_nonprod_admin, airflow_nonprod_dev and airflow_nonprod_viewer
7. Make sure you have the right client id, client secret and tenant id for the next steps.
Step 10: Create a secret from the app registration in the previous step
kubectl create secret generic airflow-ad-secret \
--from-literal=client-id=<your-azure-client-id> \
--from-literal=client-secret=<your-azure-client-secret> \
--from-literal=tenant-id=<your-azure-tenant-id>
Step 11: Configure the airflow web server for SSO
Create the values file attached in the references or with the following modifications to bring your own values file:
SSO configuration
webserver:
defaultUser:
enabled: false
authBackend: "airflow.providers.microsoft.azure.auth.backend.azure_auth"
extraEnv:
- name: AIRFLOW__WEBSERVER__RBAC
value: "True"
- name: AIRFLOW__API__AUTH_BACKENDS
value: "airflow.api.auth.backend.deny_all"
- name: AIRFLOW__WEBSERVER__AUTH_BACKEND
value: "airflow.providers.microsoft.azure.auth.backend.azure_auth"
- name: AIRFLOW__MICROSOFT__CLIENT_ID
value: "<your-client-id>"
- name: AIRFLOW__MICROSOFT__CLIENT_SECRET
value: "<your-client-secret>"
- name: AIRFLOW__MICROSOFT__TENANT_ID
value: "<your-tenant-id>"
- name: AIRFLOW__MICROSOFT__REDIRECT_URI
value: "https://<your-airflow-domain>/oauth2/callback"
and create a ConfigMap for the values file named airflow-values.
Step 12: Upgrade the airflow deployment
Run the following command:
apply the yaml above
helm upgrade airflow apache-airflow/airflow \
--namespace <your-namespace> \
-f values.yaml
Step 13: Using webserver_config.py in Airflow to enable OAuth authentication
Just apply the updated /opt/airflow/webserver_config.py4 as shown below to the airflow container.
webserver_config.py
from airflow.www.fab_security.manager import AUTH_OAUTH
AUTH_TYPE = AUTH_OAUTH
OAUTH_PROVIDERS = [{
'name': 'Microsoft Azure AD',
'token_key': 'access_token',
'remote_app': {
'api_base_url': "https://login.microsoftonline.com/{TENANT_ID}",
'access_token_url': "https://login.microsoftonline.com/{TENANT_ID}/oauth2/v2.0/token",
'authorize_url': "https://login.microsoftonline.com/{TENANT_ID}/oauth2/v2.0/authorize",
'client_id': "{CLIENT_ID}",
'client_secret': "{CLIENT_SECRET}",
'jwks_uri': "https://login.microsoftonline.com/common/discovery/v2.0/keys"
}
}]
or create.a configMap named airflow-webserver-config with the webserver_config.py file attached in the references and pass it to your instance.
Restart the airflow webserver to apply changes.
Step 14: Configure the ingress for https and redirect URI
create the following YAML:
Specify callback
rules:
- host: <your-airflow-domain>
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: airflow-web
port:
number: 8080
Step 15: Test the SSO
• Navigate to https://<your-airflow-domain>.
• You should be redirected to Azure AD login.
• Upon successful login, you’ll be redirected back to Airflow.
webserver_config.py:
from __future__ import annotations
import os
from airflow.www.fab_security.manager import AUTH_OAUTH
from airflow.www.security import AirflowSecurityManager
from airflow.utils.log.logging_mixin import LoggingMixin
basedir = os.path.abspath(os.path.dirname(__file__))
# Flask-WTF flag for CSRF
WTF_CSRF_ENABLED = True
WTF_CSRF_TIME_LIMIT = None
AUTH_TYPE = AUTH_OAUTH
OAUTH_PROVIDERS = [{
‘name’:’Microsoft Azure AD’,
‘token_key’:’access_token’,
‘icon’:’fa-windows’,
‘remote_app’: {
‘api_base_url’: https://login.microsoftonline.com/{}.format(os.getenv(“AAD_TENANT_ID”)),
‘request_token_url’: None,
‘request_token_params’: {
‘scope’: ‘openid email profile’
},
‘access_token_url’: https://login.microsoftonline.com/{}/oauth2/v2.0/token.format(os.getenv(“AAD_TENANT_ID”)),
“access_token_params”: {
‘scope’: ‘openid email profile’
},
‘authorize_url’: https://login.microsoftonline.com/{}/oauth2/v2.0/authorize.format(os.getenv(“AAD_TENANT_ID”)),
“authorize_params”: {
‘scope’: ‘openid email profile’
},
‘client_id’: os.getenv(“AAD_CLIENT_ID”),
‘client_secret’: os.getenv(“AAD_CLIENT_SECRET”),
‘jwks_uri’: ‘https://login.microsoftonline.com/common/discovery/v2.0/keys’
}
}]
AUTH_USER_REGISTRATION_ROLE = “Public”
AUTH_USER_REGISTRATION = True
AUTH_ROLES_SYNC_AT_LOGIN = True
AUTH_ROLES_MAPPING = {
“airflow_prod_admin”: [“Admin”],
“airflow_prod_user”: [“Op”],
“airflow_prod_viewer”: [“Viewer”]
}
Class AzureCustomSecurity(AirflowSecurityManager, LoggingMixin):
Def get_oauth_user_info(self, provider, response=None):
Me = self._azure_jwt_token_parse(response[“id_token”])
Return {
“name”: me[“name”],
“email”: me[“email”],
“first_name”: me[“given_name”],
“last_name”: me[“family_name”],
“id”: me[“oid”],
“username”: me[“preferred_username”],
“role_keys”: me[“roles”]
}
# the first of these two appears to work with older Airflow versions, the latter newer.
FAB_SECURITY_MANAGER_CLASS = ‘webserver_config.AzureCustomSecurity’
SECURITY_MANAGER_CLASS = AzureCustomSecurity
Airflow-Repo:
Airflow-release:
Airflow-values.yaml:
