Part 2 – Reducing operational costs of chatbot model deployment. 

This is the second part of the chatbot application discussion here. 

The following strategies are required to reduce operational costs for the deployed chat model otherwise even idle ones can incur about a thousand dollars per month. 

1. The app service plan for the app service that hosts the chat user interface must be reviewed for CPU, memory and storage. 

2. It should be set to scale dynamically. 

3. Caching mechanisms must be implemented to reduce the load to the app service. Azure Redis cache can help in this regard. 

4. If the user interface has significant assets in terms of JavaScripts and images, Content Delivery Networks could be leveraged. 

5. CDNs reduce latency and offload the traffic from the app service to distributed mirrors. 

6. It might be hard to envision the model as a database but vector storage is  used and there is an index as well. It is not just about embeddings matrix.  Choosing the appropriate database tier and sku and optimizing the queries can help with the cost. 

7. Monitoring and alerts can help to proactively identify performance bottlenecks, resource spikes and anomalies. 

8. Azure Monitor and application insights can track metrics, diagnose issues, and optimize resource usage. 

9. If the chat model experiences idle periods, then the associated resources can be stopped and scaled down during those times. 

10. You don’t need the OpenAI service APIs. You only need the model APIs. Note the following: 

1. Azure OpenAI Model API: this is the API to the GPT models used for text similarity, chat and traditional completion tasks. 

2. Azure OpenAI service API: this encompasses not just the models but also the security, encryption, deployment and management functionalities to deploy models, manage endpoints and control access. 

3. Azure OpenAI Search API allows the chatbot model to retrieve from various data sources. 

11. Storing the vectors and the embeddings and querying the search APIs does not leverage the service API. The model APIs are a must so include that in the deployment but trim the data sources to just your data. 

12. Sample deployment: 

 

 

 

 

Problem #1:  

A stream of arbitrary integers appears in no particular order and without duplicates 

 the rank of each integer is determined by the number of smaller integers before and after it up to the current position 

 Write a method to get the rank of the current integer in an efficient way. 

 eg: 7, 1, 3, 9, 5, 8 

 

Solution:  

A max heap is used to keep track of the elements we have seen and to count those that are smaller 

 

using System; 

using System.Collections.Generic; 

using System.Linq; 

 

public class Test 

{ 

private static SortedList<int, int> itemRankPair = new SortedList<int, int>(); 

public static void Main() 

{ 

     var items = new List<int>(){7, 1, 3, 9, 5, 8}; 

      

     for (int i = 0; i < items.Count; i++) 

     { 

              var item = items[i]; 

              Console.WriteLine("Item={0}", item); 

              if (itemRankPair.ContainsKey(item) == false) 

              { 

                  itemRankPair.Add(item, GetRank(item)); 

              } 

              Console.WriteLine(); 

              for (int j = 0; j < i; j++) 

              { 

                  int k = items[j]; 

                  if (k >= item) 

                  { 

                            itemRankPair[k] += 1; 

                             

                            Console.WriteLine("item={0}, Rank={1}", k, itemRankPair[k]); 

                  } 

              } 

              Console.WriteLine(); 

     } 

     foreach (var k in itemRankPair.Keys.ToList()){ 

Console.WriteLine("item={0}, Rank={1}", k, itemRankPair[k]); 

     } 

} 

 

private static int GetRank(int n) 

{ 

int rank = 0; 

foreach( var key in itemRankPair.Keys.ToList()) 

{ 

if (key < n) 

{ 

    rank++;            

} 

} 

return rank; 

} 

 

} 

 

This is a summary of the book “Streetlights and shadows: searching for the keys to adaptive decision making” written by Gary Klein and published by MIT press in 2009. This book is in a series of books by the author on decision making in complex situations. He has spent decades interviewing people such as firefighters, soldiers and pilots and brings us to some common themes and the errors in judgement. People tend to simplify complex situations until their beliefs become dangerous. This book teaches ways to avoid that and make us more adaptive. Communication and feedback are helpful but often misunderstood. He suggests blending intuition with logical analysis and draws away from guidelines and procedures to tacit knowledge including intuition, pattern recognition and mental models.

The human eye uses different systems for daylight and darkness, with "cone cells" focusing on detail and "rod cells" being more sensitive to faint light. Decision making should be approached differently in well-known and clearly defined situations and in ambiguous ones. Most common beliefs about decision making work in "bright and clear conditions," but they can be limiting or dangerous.
Incorrect beliefs include teaching people specific procedures, which can distort thinking and hinder learning and expertise. Cognitive biases, such as framing and representativeness heuristics, can also distort decision-making. Using logic instead of intuition is essential for sound decision-making, as people may overthink and back away from strong initial decisions.
Blocking out several choices and comparing them is also a common belief, but it takes too long to be useful in most situations. Experts read situational cues and react when faced with different options, making it crucial to consider both intuition and logic when making decisions.

In ambiguous situations, reducing uncertainty by acquiring more information is true, but some problems are mysteries that require analysis and action. It is important to wait until all evidence is available before jumping to conclusions. Giving feedback on the consequences of actions helps employees learn better, but it is easier in well-ordered situations and more difficult in complex, shifting circumstances. Drawing inferences from data is the "assembly line" metaphor, where data and relevance are not self-evident. Create narratives early in a situation and revise them as new data arrives. To start a project, clearly describe your goal, but this rule can stall progress in complex and shifting situations. Instead, set qualitative goals and redefine them as needed, using "managing by discovery" to gather data and adapt to the situation. This approach helps in navigating complex situations and making informed decisions.

Identifying and eliminating the biggest risks is crucial for effective risk-mitigation plans. However, inexperienced organizations may struggle with accurate evaluations. Resilience engineering and an adaptive mindset are essential for quick and effective responses. Assigning roles and writing ground rules can help define common ground, but communication is difficult due to individual experiences. Recognize the fragility of common ground.

To make better decisions in complex situations, start by shifting your mindset from a "mental storehouse" to a "circle" of adapting, sense-making, and decision-making. This involves practicing "unlearning" and recognizing the importance of intuition and analysis. As you move through life, you develop, change, and shed your skin, allowing for continual growth and change.
Make a circle of expertise, with adapting, sense-making, and decision-making points along the outer edge. This approach allows you to make sense of a situation and adapt as you act on it.
Switch from explicit knowledge to tacit knowledge, which includes recognizing patterns, mental models, perceptions, skills, and judgment. Convert tacit knowledge to explicit knowledge by internalizing it with the deep well of tacit knowledge.
Become an expert in your field by drawing lessons from experience and developing sophisticated mental models about your capabilities. Expertise doesn't prevent mistakes but helps you recognize them more quickly.

Previous book summary: BookSummary61.docx

Summarizing Software: SummarizerCodeSnippets.docx. 

Get the Longest Increasing Subsequence:

For example, the LIS of [10, 9, 2, 5, 3, 7, 101, 18] is [2, 5, 7, 101] and its size() is 4.

public static int getLIS(List<Integer> A) {

if (A == null || A.size() == 0) return 0;

var best = new int[A.size() + 1];

for (int I = 0; I < A.size(); i++)

      best[i] = 1;

for (int I = 1; I < A.size(); i++)

 {

     for (int j = 0; j < I; j++) {

            if (A[i] > A[j]) {

                   best[i]  = Math.Max(best[i], best[j] + 1);

            }

     }

}

List<Integer> b = Arrays.asList(ArrayUtils.toObject(best));

return b.stream()

.max(Comparator.comparing(Integer::valueOf))

.get();

}

 

 

 

 

Sample code for reading and writing data on Azure Machine Learning Workspace notebooks can be found online but working examples can be elusive because it is not called out as clearly that azureml.core package has been superceded by azure.ai.ml package. The following example demonstrated just how to do that.

The core objects used in this sample are Datastore and Dataset to describe the connection information and the data associated with a location.

 

from azure.ai.ml.entities import AzureDataLakeGen2Datastore

from azure.ai.ml.entities._datastore.credentials import ServicePrincipalCredentials

from azure.ai.ml import MLClient

 

ml_client = MLClient.from_config()

 

store = AzureDataLakeGen2Datastore(

    name="adls_gen2_example",

    description="Datastore pointing to an Azure Data Lake Storage Gen2.",

    account_name="mytestdatalakegen2",

    filesystem="my-gen2-container",

    credentials=ServicePrincipalCredentials(

            tenant_id="00000000-0000-0000-0000-000000000000",

            client_id="00000000-0000-0000-0000-000000000000",

            client_secret="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

        )

)

 

ml_client.create_or_update(store)

from azure.ai.ml import Dataset

dataset_path = "<fully-qualified-url-from-datastore-path>" # this could also be substituted with

# abfss://container@storageaccount.dfs.core.windows.net/path/to/file"

dataset = Dataset.from_delimited_files(dataset_path)

dataset.take(5).to_pandas_dataframe()

 

The same Dataset class can be used to write to a csv using:

import pandas as pd

from azure.ai.ml import Dataset

data = {

    'Name': ['Alice', 'Bob', 'Charlie'],

    'Age': [25, 30, 22],

    'Salary': [60000, 70000, 55000]

}

df = pd.DataFrame(data)

dataset = Dataset.from_pandas_dataframe(df)

csv_path = "<fully-qualified-url-from-datastore-path>" # this could also be substituted with

# abfss://container@storageaccount.dfs.core.windows.net/path/to/file"

dataset.to_delimited_files(csv_path)

There are N points (numbered from 0 to N−1) on a plane. Each point is colored either red ('R') or green ('G'). The K-th point is located at coordinates (X[K], Y[K]) and its color is colors[K]. No point lies on coordinates (0, 0). 

We want to draw a circle centered on coordinates (0, 0), such that the number of red points and green points inside the circle is equal. What is the maximum number of points that can lie inside such a circle? Note that it is always possible to draw a circle with no points inside. 

Write a function that, given two arrays of integers X, Y and a string colors, returns an integer specifying the maximum number of points inside a circle containing an equal number of red points and green points. 

Examples: 

1. Given X = [4, 0, 2, −2], Y = [4, 1, 2, −3] and colors = "RGRR", your function should return 2. The circle contains points (0, 1) and (2, 2), but not points (−2, −3) and (4, 4). 

class Solution { 

    public int solution(int[] X, int[] Y, String colors) { 

        // find the maximum 

        double max = Double.MIN_VALUE; 

        int count = 0; 

        for (int i = 0; i < X.length; i++) 

        { 

            double dist = X[i] * X[i] + Y[i] * Y[i]; 

            if (dist > max) 

            { 

                max = dist; 

            } 

        } 

  

        for (double i = Math.sqrt(max) + 1; i > 0; i -= 0.1) 

        { 

            int r = 0; 

            int g = 0; 

            for (int j = 0; j < colors.length(); j++) 

            { 

                if (Math.sqrt(X[j] * X[j] + Y[j] * Y[j]) > i)  

                { 

                    continue; 

                } 

  

                if (colors.substring(j, j+1).equals("R")) { 

                    r++; 

                } 

                else { 

                    g++; 

                } 

            } 

            if ( r == g && r > 0) { 

                int min = r * 2; 

                if (min > count) 

                { 

                    count = min; 

                } 

            } 

        } 

  

        return count;  

    } 

} 

  

Compilation successful. 

  

Example test:   ([4, 0, 2, -2], [4, 1, 2, -3], 'RGRR') 

OK 

  

Example test:   ([1, 1, -1, -1], [1, -1, 1, -1], 'RGRG') 

OK 

  

Example test:   ([1, 0, 0], [0, 1, -1], 'GGR') 

OK 

  

Example test:   ([5, -5, 5], [1, -1, -3], 'GRG') 

OK 

  

Example test:   ([3000, -3000, 4100, -4100, -3000], [5000, -5000, 4100, -4100, 5000], 'RRGRG') 

OK