Saturday, May 10, 2025

 This is a summary of the book titled “The Leaders’ Playlist” written by Susan Drumm and published by River Grove Books in 2022. If there was a book that connected music to leadership, this might be the first. That’s because the author, as a leadership coach to top notch executives, connects leaders’ behaviors to childhood experiences and emotional triggers, specifically music which she then uses as a way to combat strong feelings at work and even overcome trauma. This forms her imaginative path to transformation.

She says music creates or strengthens neural connections and pathways and by changing your playlist, you can change your subconscious emotions and that leads to changes in your behavior and your life. Self-discovery through music can reveal your unique leadership style. You can even bring out the best in your yourself by changing your emotional frequency from anger and fear to love and empathy. To succeed, do the inner work. Invest reflection, effort and time to change your old and new playlists. She offers a seven-step process to do so.

The effect of music on body is science. For example, listening to music during exercise has proven to make you feel less fatigued, require less oxygen, and experience more endurance and energy. It even works on severe Alzheimer’s patients who don’t react to most stimuli, but music is an exception. She even gives a personal experience of including songs like “Jar of Hearts” a song about breakup, to soothe her stresses around betrayal and exclusion, and switching to songs like “24K Magic” that filled her with energy and happiness.

These are not a metaphor. Literally music alters subconscious drivers that helps you heal and change how you think leading to improved productivity. It is worthy of investment. Feelings like music emit frequencies that other people register. If a good laugh is infectious, good vibes from positive music is also infectious. This process of discovery demands deep reflection and curiosity as well as a willingness to explore memories and experiment. In a candid share of her memories, she talks about her resentment at her father’s infrequent violent outbursts on an otherwise overall good standing, that translated to her infrequent overreactions at work. The triggers for her fight or flight reaction were overcome with a heightened self-awareness and a new playlist. By recognizing and working on these, she even emerges a more authentic self as called out by others. By recognizing your wounds, you heal them. Her seven-step method to do so involves the following: 1. recognize that you are hitting a wall, so you can build a new playlist, 2. get to know yourself including those that you avoided introspecting earlier, 3. Identify five emotions that truly steer you, 4. connect your emotions with your memories and observe, 5. Build your “wounded” playlist to associate with those emotions that might include lyrics or rhythms that resonate with you. 6. Build your new playlist with the opposites of those emotions and tunes and 7. Find your “meaningful mission” by relating to a greater purpose that helps others. I appreciate the playing of upbeat tunes before conferences and meetings more after this reading.


Friday, May 9, 2025

 AKS - Airflow setup and use with SSO

Here’s a step-by-step guide to deploying and using Apache Airflow on Azure Kubernetes Service (AKS) [1]:

Step 1: Set Up Your AKS Cluster

If you don’t already have an AKS cluster, create one using IaC. This should already be done for you. Log into the Aks cluster with

az aks get-credentials --resource-group <resourcegroupname> --name <clustername>

kubelogin convert-kubeconfig -l azurecli

Step 2: Install Helm

Ensure Helm is installed on your local machine:

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Step 3: Create a Namespace for Airflow

kubectl create namespace airflow

Step 4: Configure Workload Identity (Optional but Recommended)

This step allows Airflow to securely access Azure resources like Key Vault:

1. Create a service account:

kubectl apply -f - <<EOF

apiVersion: v1

kind: ServiceAccount

metadata:

  name: airflow

  namespace: airflow

EOF

1. Annotate the service account with your Azure identity:

kubectl annotate serviceaccount airflow \

  azure.workload.identity/client-id=<CLIENT_ID> \

  azure.workload.identity/tenant-id=<TENANT_ID> \

  -n airflow

Step 5: Install External Secrets Operator (Optional for Key Vault Integration)

helm repo add external-secrets https://charts.external-secrets.io

helm repo update

helm install external-secrets external-secrets/external-secrets \

  --namespace airflow \

  --create-namespace \

  --set installCRDs=true \

  --wait

Step 6: Add the Apache Airflow Helm Chart

helm repo add apache-airflow https://airflow.apache.org

helm repo update

Step 7: Install Airflow

a kustomization is preferred:

or using helm

helm install airflow apache-airflow/airflow \

  --namespace airflow \

  --set executor=CeleryExecutor \

  --set airflow.image.tag=2.8.1 \

  --set createUser=true \

  --set webserver.defaultUser.username=admin \

  --set webserver.defaultUser.password=admin

You can get Fernet key with

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

and optionally save it as a secret

kubectl get secret --namespace airflow airflow-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode

I prefer creating a HelmRelease with the official airflow chart with the release file in the references and creating configMaps for values file also in the references and the webserver_config.py discussed a few steps below if you want to specify SSO during setup.

Step 8: Access the Airflow Web UI

Port-forward the web server service:

kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow

Then open your browser and go to: http://localhost:8080

For a complete setup, it could look like this: PIBI NonProd Airflow

Step 9: Setup the app registration for SSO

Add the following:

1. redirect URI as https://<your-airflow-domain>/oauth-authorized/azure or https://<your-airflow-domain>/oauth2/callback

2. Assign API permissions (openid, email, profile) for authentication.

3. Navigate to the token configuration page of the Azure AD application. For ID and access token, add an optional claim on the

1. email

2. preferred_username

3. given_name

4. family_name

5. UPN

4. Edit the groups claim to include sAMAccountName for both ID and Access tokens but leave out SAML.

5. Specify federated identity entries for use with your GitHub repository.

6. Optionally, create App roles on your Azure AD application such as airflow_nonprod_admin, airflow_nonprod_dev and airflow_nonprod_viewer

7. Make sure you have the right client id, client secret and tenant id for the next steps.

Step 10: Create a secret from the app registration in the previous step

kubectl create secret generic airflow-ad-secret \

  --from-literal=client-id=<your-azure-client-id> \

  --from-literal=client-secret=<your-azure-client-secret> \

  --from-literal=tenant-id=<your-azure-tenant-id>

Step 11: Configure the airflow web server for SSO

Create the values file attached in the references or with the following modifications to bring your own values file:

SSO configuration

webserver:

  defaultUser:

    enabled: false

  authBackend: "airflow.providers.microsoft.azure.auth.backend.azure_auth"

  extraEnv:

    - name: AIRFLOW__WEBSERVER__RBAC

      value: "True"

    - name: AIRFLOW__API__AUTH_BACKENDS

      value: "airflow.api.auth.backend.deny_all"

    - name: AIRFLOW__WEBSERVER__AUTH_BACKEND

      value: "airflow.providers.microsoft.azure.auth.backend.azure_auth"

    - name: AIRFLOW__MICROSOFT__CLIENT_ID

      value: "<your-client-id>"

    - name: AIRFLOW__MICROSOFT__CLIENT_SECRET

      value: "<your-client-secret>"

    - name: AIRFLOW__MICROSOFT__TENANT_ID

      value: "<your-tenant-id>"

    - name: AIRFLOW__MICROSOFT__REDIRECT_URI

      value: "https://<your-airflow-domain>/oauth2/callback"

and create a ConfigMap for the values file named airflow-values.

Step 12: Upgrade the airflow deployment

Run the following command:

apply the yaml above

helm upgrade airflow apache-airflow/airflow \

  --namespace <your-namespace> \

  -f values.yaml

Step 13: Using webserver_config.py in Airflow to enable OAuth authentication

Just apply the updated /opt/airflow/webserver_config.py4 as shown below to the airflow container.

webserver_config.py

from airflow.www.fab_security.manager import AUTH_OAUTH

AUTH_TYPE = AUTH_OAUTH

OAUTH_PROVIDERS = [{

    'name': 'Microsoft Azure AD',

    'token_key': 'access_token',

    'remote_app': {

        'api_base_url': "https://login.microsoftonline.com/{TENANT_ID}",

        'access_token_url': "https://login.microsoftonline.com/{TENANT_ID}/oauth2/v2.0/token",

        'authorize_url': "https://login.microsoftonline.com/{TENANT_ID}/oauth2/v2.0/authorize",

        'client_id': "{CLIENT_ID}",

        'client_secret': "{CLIENT_SECRET}",

        'jwks_uri': "https://login.microsoftonline.com/common/discovery/v2.0/keys"

    }

}]

or create.a configMap named airflow-webserver-config with the webserver_config.py file attached in the references and pass it to your instance.

Restart the airflow webserver to apply changes.

Step 14: Configure the ingress for https and redirect URI

create the following YAML:

Specify callback

rules:

  - host: <your-airflow-domain>

    http:

      paths:

        - path: /

          pathType: Prefix

          backend:

            service:

              name: airflow-web

              port:

                number: 8080

Step 15: Test the SSO

• Navigate to https://<your-airflow-domain>.

• You should be redirected to Azure AD login.

• Upon successful login, you’ll be redirected back to Airflow.

webserver_config.py:

from __future__ import annotations

import os

from airflow.www.fab_security.manager import AUTH_OAUTH

from airflow.www.security import AirflowSecurityManager

from airflow.utils.log.logging_mixin import LoggingMixin

basedir = os.path.abspath(os.path.dirname(__file__))

# Flask-WTF flag for CSRF

WTF_CSRF_ENABLED = True

WTF_CSRF_TIME_LIMIT = None

AUTH_TYPE = AUTH_OAUTH

OAUTH_PROVIDERS = [{

    ‘name’:’Microsoft Azure AD’,

    ‘token_key’:’access_token’,

    ‘icon’:’fa-windows’,

    ‘remote_app’: {

        ‘api_base_url’: https://login.microsoftonline.com/{}.format(os.getenv(“AAD_TENANT_ID”)),

        ‘request_token_url’: None,

        ‘request_token_params’: {

            ‘scope’: ‘openid email profile’

        },

        ‘access_token_url’: https://login.microsoftonline.com/{}/oauth2/v2.0/token.format(os.getenv(“AAD_TENANT_ID”)),

        “access_token_params”: {

            ‘scope’: ‘openid email profile’

        },

        ‘authorize_url’: https://login.microsoftonline.com/{}/oauth2/v2.0/authorize.format(os.getenv(“AAD_TENANT_ID”)),

        “authorize_params”: {

            ‘scope’: ‘openid email profile’

        },

        ‘client_id’: os.getenv(“AAD_CLIENT_ID”),

        ‘client_secret’: os.getenv(“AAD_CLIENT_SECRET”),

        ‘jwks_uri’: ‘https://login.microsoftonline.com/common/discovery/v2.0/keys’

    }

}]

AUTH_USER_REGISTRATION_ROLE = “Public”

AUTH_USER_REGISTRATION = True

AUTH_ROLES_SYNC_AT_LOGIN = True

AUTH_ROLES_MAPPING = {

    “airflow_prod_admin”: [“Admin”],

    “airflow_prod_user”: [“Op”],

    “airflow_prod_viewer”: [“Viewer”]

}

Class AzureCustomSecurity(AirflowSecurityManager, LoggingMixin):

    Def get_oauth_user_info(self, provider, response=None):

        Me = self._azure_jwt_token_parse(response[“id_token”])

        Return {

            “name”: me[“name”],

            “email”: me[“email”],

            “first_name”: me[“given_name”],

            “last_name”: me[“family_name”],

            “id”: me[“oid”],

            “username”: me[“preferred_username”],

            “role_keys”: me[“roles”]

        }

# the first of these two appears to work with older Airflow versions, the latter newer.

FAB_SECURITY_MANAGER_CLASS = ‘webserver_config.AzureCustomSecurity’

SECURITY_MANAGER_CLASS = AzureCustomSecurity

Airflow-Repo:

Airflow-release:

Airflow-values.yaml:


Thursday, May 8, 2025

 This is a summary of the book titled “The Psychology of Leadership” written by Sebastien Page and published by Harriman House in 2025. The author is a business executive who brings his experience and aspects of sports psychology, positive psychology and personality psychology to discover and frame tenets of universal leadership that promotes high-end performance. These tenets include being aware of unintended consequences, organizing your work around your goals, and not obsessing over winning. This also implies developing a personal sense of success, assessing those beliefs, giving a sense of purpose to everyone and developing patience.

By borrowing ideas from different perspectives and disciplines besides his own experience, he rounds up his framework with the specific mental skills that are critical to high performance. For example, sports psychology doesn’t define success as winning but learning to develop mental resilience to keep going when you lose. The PERMA model which stands for “positive emotions, engagement, relationships, meaning and accomplishment” offers the advantage of setting realistic goals and preparing for unforeseen consequences. This is especially helpful to long-term roadmaps.

Developing an innate sense of success also differentiate success and happiness. Success must lead to happiness. Real, enduring happiness centers more on “ERMA” in the PERMA. Good relationships with your colleagues, a sense that your work is meaningful and a feeling of genuine accomplishment can lead to personal happiness. A good leader knows that this leads to increased productivity and motivation in the team. To promote engagement and flow, optimize “Return on Time Spent”. Start by reviewing the job’s activities for everyone, the time and effort involved and how it aligns with goals. A sense of meaning benefits everyone. The R and the M in the PERMA stand for relationships and meaning. Relationships is how people trust one another, and Meaning is what binds them. An articulated purpose should come through in goal setting. Although A stands for accomplishments, success should not be the only priority. Success does boost well-being but in isolation, it is just a heady rush or short-lived emotion. Similarly, mastery isn’t a one-off experience, it is a sustainable character trait. When assessing your own beliefs on success, start with how you and your team relate to the world. Re-evaluating trivial mission statements, refining core beliefs and writing them down help. Engineering taught us about dynamic feedback loops to narrow down the discrepancies between expectations and reality. Working backwards from the goal helps us do that.

Finally, most leadership, charters, roadmap, goals, relationships, success and motivation must be accommodating to unknown and unforeseen circumstances as they develop. So, developing strategic patience may not be comfortable but it is needed.


Wednesday, May 7, 2025

 The DFCS drone video sensing platform is intended to be one that facilitates writing video sensing applications for a variety of purposes. The platform strives to do one use case for a video sensing application very well with its proprietary model and RAG enhanced autonomous routing, but the same knowledge base, world catalog and vector search and image processing pipeline can also be used for other applications.

The analytics engine of the DFCS platform supports natural language queries but this could be customized. For example, “Contextual Embeddings” was popularized by Anthropic and Microsoft communities for text data and this is applicable to the DFCS world catalog. Given a document, they split it into chunks of text and prepend a chunk specific explanatory context generated by an LLM. There is no loss of information at chunk boundaries. Let us call this a ContextRetrieval class and it has methods implemented with langchain for Azure to load a pdf and parse using AzureAIDocumentIntelligenceLoader, then process it with a method that splits the document efficiently ensuring no loss of information at chunk boundaries, follow it with generating a context for each chunk using a ChatPromptTemplate by identifying the main topic in the chunk along with its relation to the broader context and including key figures, dates or percentages.

While the contextualized chunks can be used with an RAG for semantic similarity-based search aka dense retrievers, it can also be used with lexical search like Best Match 25 aka BM25 to find exact matches and specific terminology. So, the class has an additional method to supplement retrieval. When the document arrives in the pipeline, the dense and sparse retrievers quickly narrow down a search space to get relevant chunks. The ContextRetrieval class has methods to create bm25 index. Then for each of these relevant chunks, a large language model is used to generate an answer using a prompt. With the generated answers, the ContextRetrieval class performs a second stage processing called re-ranking by analyzing the deep semantic relationship between the query and the chunk, considering factors like factual alignment, answer coverage, and contextual relevance. This was demonstrated to have a better match than using chunks without contexts or the whole document as the baseline.

Since the world catalog is based on location, contextual embeddings leverage the indexes on location and the associated keypoint features so that the responses for the user query are more aligned.


Tuesday, May 6, 2025

 This is a summary of the book titled “The Perfect Swarm - The Science of Complexity in Everyday Life” written by Len Fischer and published by Basic Books in 2009. The author is a physicist who is writing about group behavior in a style that reflects humor and wit while drawing on incredible gamut of evidence from experiments, historical events, contemporary cases and real-life experiences. His coverage of group behavior might be exciting for professionals in all walks of life. He cites unbelievable examples such as self-organizing locust swarms and fish schools, progression from chaos to order without any guided intelligence, positive feedback and chain reactions throwing into a chaos while “negative feedback” stabilizing the group, swarm without leader or roadmap hitting the goal, and such others. He tells us a group uses one of three tactics to reach a consensus regardless of right or wrong: deciding by majority, debating till agreement, or using swarm intelligence. Almost always, a group of experts outperforms individual members in making decisions. When you face many options, look for patterns in information, but test them to make sure they are reliable and decide on a solution that surpasses your expectations.

The science of complexity studies the rules and processes of self-organization, allowing complex structures and relationships to emerge out of chaos without a central director or single intelligence. Swarm behavior becomes swarm intelligence when a group can solve a problem collectively in a way that individuals within the group cannot. Systems exhibit two types of dynamic patterns: cycles as in a family quarrel and adaptive systems as in cheering becoming a unison. Swarms have no leaders, but members can pass information to one another through observation and rules.

Crowds have emergent complex structures that arise from physical and social forces between individuals. Certain phenomena, such as positive feedback and chain reactions, can throw a swarm back into chaos. Positive feedback, such as between a microphone and an amplifier, can lead to a loud sound and cause a system to crash, while chain reactions, like locusts feeling happy to congregate can lead to swarms of up to 100 billion locusts and a human running in a neighborhood in Columbus, Ohio believing a tidal wave was coming, was joined by everyone in the neighborhood one by one.

Negative feedback balances destabilizing forces, as described by Adam Smith's "invisible hand" theory where prices stabilize after instability or disturbance.

In city streets, humans and bees follow 3 similar principles of swarming, avoiding collisions, aligning with their closest bees, and attracting each other. This behavior is similar to how humans lead crowds, even when they don't know the leader or target. Ant colonies use pheromones to find the shortest route to and from scarce resources, drawing more ants to follow the shortest trail. This phenomenon can be used to succeed in the market by imitating others.

People move like ants, establishing distinct pedestrian lanes and maintaining traffic flow at a certain population density. Traffic clogs can be managed by urban designers taking traffic flow into account, such as widening crowded areas or placing pillars strategically in buildings. In a panic, people follow the swarm about 60% of the time and search for alternative means of escape about 40% of the time. Planning emergency strategies in advance is crucial to avoid panics and ensure safety.

To make decisions in a group, individuals can either vote or generate an average opinion to guide the group. The method depends on the question being asked. For estimating questions, asking everyone to come up with a number on their own and then averaging the responses can be more accurate. For multiple-choice questions, voting and going with the majority can be more effective. Experts are most useful when dealing with problems at the intersection of knowledge and initiative. Groups often struggle to reach consensus due to the three choices: follow the will of the majority, debate the issue until they reach consensus, or use swarm intelligence. Groupthink is a dangerous phenomenon where members overvalue the group's ethics and insight. Swarm intelligence emerges when individuals spontaneously and voluntarily interact to solve problems, resembling stakeholders rather than shareholders. Swarms are more likely than other groups to share their power or even give it away.

Networks are sets of items and connections that emerge among people, often combining elements of both deliberate planning and randomness. They are not evenly distributed, with connections clustering according to a "power law," with a few nodes having more connections than most others. Understanding the hubs and shortcuts that link parts of a network is crucial in various fields, such as public health and marketing. To decipher the importance of information, one can use an approach from gold miners, pick up obvious gems, sift through the data until nuggets emerge, and look for patterns in the unsorted mass of data. Patterns can emerge naturally and spontaneously in all areas of life and using heuristics can help make quick decisions. These include recognition, fluency, weighing, taking the best, and “satisficing” or exceeding expectations.

Reference: previous summaries: https://1drv.ms/w/c/d609fb70e39b65c8/EV8_nILFMeBGqIyIpFQtSGAB8bX0HfvlKUXZ6IfYvgsxTA?e=BMT3LD


Monday, May 5, 2025

 Database calls are fast and the curation of objects lends itself to query operators but does not take advantage of the progressive and rolling sequence of objects detected frame-by-frame or its bookkeeping along with multiple UAV sequence tracking which is something messaging paradigm and event processors have solved successfully in several data and telemetry pipelines and from a traditional grounding that databases are queues. With the shift in paradigm from rows to events and similar SQL operators across both such as with Flink, the DFCS drone video sensing platform does not demand adherence to a database or messaging paradigm if the interface supports the following requirements: 1. standard query operator on objects detected with world co-ordinates attributes. 2. participation in retrieval augmentation along with a vector store and search and 3. support analytics stacks with programmability that can support custom drone sensing applications built independent of the UAV swarm sensing-analysis-routing architecture dedicated to the swarms’ flights. In addition, some criteria are suggested for messaging pipelines which include:

1. Noise filtering: This involves sifting through data to spotlight the essentials.

2. Long-Term data retention: this involves safeguarding valuable data for future use

3. Event-trimming: This customizes data for optimal analytics so that the raw data is not dictating eccentricities in the charts and graphs.

4. Data condensation: this translates voluminous MELT data into focused metrics and prevents archiving or cleaning up as messages are removed from the queue.

5. Operational Efficiency Boosting: This amplifies operating speed and reliability.

This article implements a sample for this alternative with emphasis on multiple stream processors


Sunday, May 4, 2025

 Agentic object detection 

import requests

def get_bounding_boxes(api_key, image_path, search_text):

    headers = {"Authorization": f"Bearer {api_key}"}

    with open(image_path, "rb") as image_file:

        files = {"file": image_file}

        data = {"prompt": search_text}

        response = requests.post(

            "https://api.landing.ai/v1/agentic-object-detection",

            headers=headers,

            files=files,

            data=data

        )

    if response.status_code == 200:

        detections = response.json().get("detections", [])

        return [detection["box"] for detection in detections] # Returns [x_min, y_min, x_max, y_max] boxes

    else:

        raise Exception(f"API Error: {response.text}")

# Usage example

api_key = "your_api_key_here"

boxes = get_bounding_boxes(api_key, "sample.jpg", "red shoes")

print(f"Found {len(boxes)} matching objects:")

for box in boxes:

    print(f"- Bounding box: {box}")