Azure Databricks Cluster usage report.

1.       From UI:

a.       Click compute in the sidebar.

b.       Select the compute resource to view metrics for

c.       Click the metrics tab.

d.       Refer to the CPU utilization cluster metrics. This is based on total CPU seconds’ cost. The metric is averaged out based on which time interval is displayed in the chart. The default time interval is the last hour.

e.       There is also a GPU metric chart. This is based on the percentage of GPU utilization also averaged out for the chosen time interval.

2.       From PowerShell:

a.       Install-Module -Name DatabricksPS

b.       RESPONSE=$(curl http://localhost:50342/oauth2/token --data "resource=https://management.azure.com/" -H Metadata:true -s)

c.       ACCESS_TOKEN=$(echo $RESPONSE | python -c 'import sys, json; print (json.load(sys.stdin)["access_token"])')

d.       accesstoken=$ACCESS_TOKEN

e.       $apiUrl="https://management.azure.com/subscriptions/656e67c6-f810-4ea6-8b89-636dd0b6774c/resourceGroups/rg-temp/providers/Microsoft.Databricks/workspaces/wks-rg-temp-1?api-version=2023-02-01"

f.        $headers=@{"Authorization"= "${accesstoken}"}

g.       Invoke-RestMethod -Method "GET" -Uri $apiUrl -headers $headers

Sample Response:

```                                   

properties : @{managedResourceGroupId=/subscriptions/656e67c6-f810-4ea6-8b89-636dd0b6774c/resourceGroups/dbw-rg-temp-1; parameters=;

             provisioningState=Succeeded; authorizations=System.Object[]; createdBy=; updatedBy=; workspaceId=8965842579407484;

             workspaceUrl=adb-8965842579407484.4.azuredatabricks.net; createdDateTime=7/9/2023 8:14:11 PM}

id         : /subscriptions/656e67c6-f810-4ea6-8b89-636dd0b6774c/resourceGroups/rg-temp/providers/Microsoft.Databricks/workspaces/wks-rg-temp-1

name       : wks-rg-temp-1

type       : Microsoft.Databricks/workspaces

sku        : @{name=premium}

location   : centralus

tags       :

```

h.       $accesstoken=”dapie12c.....b3a2-2”

i.         $headers=@{"Authorization"= "Bearer ${accesstoken}"}

j.         $apiUrl="https://adb-8965...484.4.azuredatabricks.net/api/2.0/clusters/get" # curl --request GET "https://${apiUrl}/api/2.0/clusters/get"    --header "Authorization: Bearer ${accesstoken}"     --data '{ "cluster_id": "1234-567890-a12bcde3" }'

 

Sample Response:

PS /home/ravi> Invoke-RestMethod -Method "GET" -Uri $apiUrl -headers $headers -Body '{ "cluster_id": "0709-203735-epaybeni" }'

 

cluster_id                   : 0709-203735-epaybeni

creator_user_name            : ravibeta@hotmail.com

driver                       : @{private_ip=10.139.64.4; public_dns=40.113.230.98; node_id=6a94d151f85c48f691ecfa2b501ddb8c;

                               instance_id=9f3e9e6cbf344608942f28e5db5c22a2; start_timestamp=1688935207640; node_attributes=; host_private_ip=10.139.0.4}

spark_context_id             : 2865750645175411723

driver_healthy               : True

jdbc_port                    : 10000

cluster_name                 : Ravi Rajamani's Personal Compute Cluster

spark_version                : 13.2.x-cpu-ml-scala2.12

spark_conf                   : @{spark.databricks.delta.preview.enabled=true; spark.databricks.cluster.profile=singleNode; spark.master=local[*, 4]}

azure_attributes             : @{first_on_demand=1; availability=ON_DEMAND_AZURE; spot_bid_max_price=-1}

node_type_id                 : Standard_DS3_v2

driver_node_type_id          : Standard_DS3_v2

custom_tags                  : @{ResourceClass=SingleNode}

autotermination_minutes      : 4320

enable_elastic_disk          : True

disk_spec                    :

cluster_source               : UI

single_user_name             : ravibeta@hotmail.com

policy_id                    : 00164B5BAEB11244

enable_local_disk_encryption : False

instance_source              : @{node_type_id=Standard_DS3_v2}

driver_instance_source       : @{node_type_id=Standard_DS3_v2}

data_security_mode           : LEGACY_SINGLE_USER_STANDARD

runtime_engine               : STANDARD

effective_spark_version      : 13.2.x-cpu-ml-scala2.12

state                        : RUNNING

state_message                :

start_time                   : 1688935055795

last_state_loss_time         : 0

last_activity_time           : 1688935483841

last_restarted_time          : 1688935294611

num_workers                  : 0

cluster_memory_mb            : 14336

cluster_cores                : 4

default_tags                 : @{Vendor=Databricks; Creator=ravibeta@hotmail.com; ClusterName=Ravi Rajamani's Personal Compute Cluster;

                               ClusterId=0709-203735-epaybeni}

init_scripts_safe_mode       : False

 

k.       Set-DatabricksEnvironment -AccessToken $accesstoken -ApiRootUrl $apiUrl

l.         Get-DatabricksCluster | Stop-DatabricksCluster

The above uses PAT tokens. It can also use Azure AD as follows:

PS /home/ravi> $credUser = Get-Credential                                                                              

 

PowerShell credential request

Enter your credentials.

User: ravibeta@hotmail.com

Password for user ravibeta@hotmail.com: ****************

 

PS /home/ravi> $tenantId = "1f4c33e1-e960-43bf-a135-6db8b82b6885"; $subscriptionId = "656e67c6-f810-4ea6-8b89-636dd0b6774c";

PS /home/ravi> $resourceGroupName = "rg-temp"                                                                              

PS /home/ravi> $workspaceName="ravibeta@hotmail.com"

PS /home/ravi> $azureResourceId="/subscriptions/656e67c6-f810-4ea6-8b89-636dd0b6774c/resourceGroups/rg-temp/providers/Microsoft.Databricks/workspaces/wks-rg-temp-1"

PS /home/ravi> $workspaceName="wks-rg-temp-1"                                                                                                              PS /home/ravi> $clientId="50996fd9-da74-4f41-b262-490d074bc807"

PS /home/ravi> $apiUrl="https://adb-8965842579407484.4.azuredatabricks.net/"                                                                               PS /home/ravi> Set-DatabricksEnvironment -ClientID $clientId -Credential $credUser -AzureResourceID $azureResourceId -TenantID $tenantId -ApiRootUrl $apiUrl

Connect-AzAccount -UseDeviceAuthentication will not be sufficient for above cmdlet.

The same credential can be used with more than one environment.

Reference: https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/usage-detail-tags

 

Application of MicrosoftML rxFastTree algorithm to Insurance payment validations and predictions:

Logistic regression is a well-known statistical technique that is used to model binary outcomes. It can be applied to detect root causes of payment errors. It uses statistical measures, is highly flexible, takes any kind of input and supports different analytical tasks. This regression folds the effects of extreme values and evaluates several factors that affect a pair of outcomes.

Logistic regression differs from the other Regression techniques in the use of statistical measures. Regression is very useful to calculate a linear relationship between a dependent and independent variable, and then use that relationship for prediction. Errors demonstrate elongated scatter plots in specific categories. Even when the errors come with different error details in the same category, they can be plotted with correlation. This technique is suitable for specific error categories from an account.

One advantage of logistic regression is that the algorithm is highly flexible, taking any kind of input, and supports several different analytical tasks:

-          Use demographics to make predictions about outcomes, such as probability of defaulting payments.

-          Explore and weight the factors that contribute to a result. For example, find the factors that influence customers to make a repeat late payment.

-          Classify claims, payments, or other objects that have many attributes.

Microsoft ML rxFastTree algorithm is also an example. 

 

 The gradient boost algorithm for rxFastTree is possible with several loss functions including the squared loss function.

The algorithm for the least squares regression can be written as :

 

1. Set the initial approximation 

 

2. For a set of successive increments or boosts each based on the preceding iterations, do

 

3. Calculate the new residuals

 

4. Find the line of search by aggregating and minimizing the residuals

 

5. Perform the boost along the line of search

 

6. Repeat 3,4,5 for each of 2.

 

Some methods of organization for large scale Infrastructure-as-a-Code deployments. 

The purpose of IaC is to provide a dynamic, reliable, and repeatable infrastructure suitable for cases where manual approaches and management practices cannot keep up. When automation increases to the point of becoming a cloud-based service responsible for the deployment of cloud resources and stamps that provision other services that are diverse, consumer facing and public cloud general availability services, some learnings can be called out that apply universally across a large spectrum of industry clouds. 

A service that deploys other services must accept IaC deployment logic with templates, intrinsics, and deterministic execution that works much like any other workflow management system. This helps to determine the order in which to run them and with retries. The tasks are self-described. The automation consists of a scheduler to trigger scheduled workflows and to submit tasks to the executor to run, an executor to run the tasks, a web server for a management interface, a folder for the directed acyclic graph representing the deployment logic artifacts, and a metadata database to store state. The workflows don’t restrict what can be specified as a task which can be an Operator or a predefined task using say Python, a Sensor which is entirely about waiting for an external event to happen, and a Custom task that can be specified via a Python function decorated with a @task. 

The organization of such artifacts posed two necessities. First, to leverage the builtin templates and deployment capabilities of the target IaC provider as well as their packaging in the format suitable to the automation that demands certain declarations, phases, and sequences to be called out. Second the co—ordination of context management switches between automation service and IaC provider. This involved a preamble and an epilogue to a context switch for bookkeeping and state reconciliation. 

This taught us that large IaC authors are best served by uniform, consistent and global naming conventions, registries that can be published by the system for cross subscription and cross region lookups, parametrizing diligently at every scope including hierarchies, leveraging dependency declarations, and reducing the need for scriptability in favor of system and user defined organizational units of templates. Leveraging supportability via read-only stores and frequently publishing continuous and up-to-date information on the rollout helps alleviate the operations from the design and development of IaC. 

IaC writers frequently find themselves in positions where the separation between pipeline automation and IaC declarations are not clean, self-contained or require extensive customizations. One of the approaches that worked on this front is to have multiple passes on the development. With one pass providing initial deployment capability and another pass consolidating and providing best practice via refactoring and reusability. Enabling the development pass to be DevOps based, feature centric and agile helps converge to a working solution with learnings that can be carried from iteration to iteration. The refactoring pass is more generational in nature. It provides cross-cutting perspectives and non-functional guarantees. 

A library of routines, operators, data types, global parameters and registries are almost inevitable with large scale IaC deployments but unlike the support for programming language-based packages, these are often organically curated in most cases and often self-maintained. Leveraging tracking and versioning support of source control, its possible to provide compatibility as capabilities are made native to the IaC provider or automation service. 

Reference: IaC shortcomings and resolutions. 

 

 

IaC Resolutions Part 8:

Previous articles in this regard have been discussing resolutions for shortcomings in the use of Infrastructure-as-a-code (IaC) in various scenarios. This section discusses the resolution for the case when changes to resources involves breaking a deadlock in state awareness between, say a pair.

Let us make a specific association between say a firewall and a network resource such as a gateway. The firewall must be associated with the gateway to prevent traffic flow through that appliance. When they remain associated, they remember the identifier and the state for each other. Initially, the firewall may remain in detection mode where it is merely passive. It becomes active in the prevention mode. When the modes are attempted to be toggled, the association prevents it. Neither end of the association can tell what state to be in without exchanging the information and when they are deployed or updated in place, neither knows about or informs the other.

There are two ways to overcome this limitation.

First, there is a direction established between the resources where the update to one forcibly updates the state of the other. This is supported by the gateway when it allows the information to be written through by the update in the state of one resource.

Second, the changes are made by the IaC provider first to one resource and then to the other so that the update to the other picks up the state of the first during its change. In this mode, the firewall can be activated after the gateway knows that there is such a firewall.

If the IaC tries to form an association while updating the state of one, the other might end up with an inconsistent state. One of the two resolutions above works to mitigate this.

This is easy when there is a one-to-one relationship between resources. Sometimes there are one-to-many relationships. For example, a gateway might have more than a dozen app services as its backend members and each member might be allowing public access. If the gateway must consolidate access to all the app services, then there are changes required on the gateway to route traffic to each app service as intended by the client and a restriction on the app services to allow only private access from the gateway.

Consider the sequence in which these changes must be made given that the final operational state of the gateway is only acceptable when all barring none remain reachable for a client through the gateway.

If the app services toggle the access from public to gateway sooner than the gateway becomes operational, there is some downtime to them, and the duration is not necessarily bounded if one app service fails to listen to the gateway. The correct sequence would involve first making the change in the gateway to set up proper routing and then restricting the app services to accept only the gateway. Finally, the gateway validates all the app service flows from a client before enabling them.

Each app service might have nuances about whether the gateway can reach it one way or another. Usually, if they are part of the same vnet, then this is not a concern, otherwise peering might be required. Even if the peering is made available, routing by address or resolution by name or both might be required unless they are universally known on the world wide web. If the public access is disabled, then the private links must be established, and this might require both the gateway and the app service to do so. Lastly, with each change, an app service must maintain its inbound and outbound properly for bidirectional communication, so some vetting is required on the app service side independent of the gateway.

Putting this altogether via IaC requires that the changes be made in stages and each stage validated independently.

 

Problem Statement: A 0-indexed integer array nums is given.

Swaps of adjacent elements are able to be performed on nums.

A valid array meets the following conditions:

·       The largest element (any of the largest elements if there are multiple) is at the rightmost position in the array.

·       The smallest element (any of the smallest elements if there are multiple) is at the leftmost position in the array.

Return the minimum swaps required to make nums a valid array.

 

Example 1:

Input: nums = [3,4,5,5,3,1]

Output: 6

Explanation: Perform the following swaps:

- Swap 1: Swap the 3^rd and 4^th elements, nums is then [3,4,5,3,5,1].

- Swap 2: Swap the 4^th and 5^th elements, nums is then [3,4,5,3,1,5].

- Swap 3: Swap the 3^rd and 4^th elements, nums is then [3,4,5,1,3,5].

- Swap 4: Swap the 2^nd and 3^rd elements, nums is then [3,4,1,5,3,5].

- Swap 5: Swap the 1^st and 2^nd elements, nums is then [3,1,4,5,3,5].

- Swap 6: Swap the 0^th and 1^st elements, nums is then [1,3,4,5,3,5].

It can be shown that 6 swaps is the minimum swaps required to make a valid array.

Example 2:

Input: nums = [9]

Output: 0

Explanation: The array is already valid, so we return 0.

 

Constraints:

·         1 <= nums.length <= 10⁵

·         1 <= nums[i] <= 10⁵

Solution:

class Solution {

    public int minimumSwaps(int[] nums) {

        int min = Arrays.stream(nums).min().getAsInt();

        int max = Arrays.stream(nums).max().getAsInt();

        int count = 0;

        while (nums[0] != min && nums[nums.length-1] != max && count < 2 * nums.length) {           

            var numsList = Arrays.stream(nums).boxed().collect(Collectors.toList());

            var end = numsList.lastIndexOf(max);

            for (int i = end; i < nums.length-1; i++) {

                swap(nums, i, i+1);

                count++;

            }

 

            numsList = Arrays.stream(nums).boxed().collect(Collectors.toList());

            var start = numsList.indexOf(min);

            for (int j = start; j >= 1; j--) {

                swap(nums, j, j-1);

                count++;

            }

        }

 

        return count;

    }

 

    public void swap (int[] nums, int i, int j) {

        int temp = nums[j];

        nums[j] = nums[i];

        nums[i] = temp;

    }

}

 

Input

nums =

[3,4,5,5,3,1]

Output

6

Expected

6

 

Input

nums =

[9]

Output

0

Expected

0

 

 

 

Fixing path-based routing in Application Gateways:

The Azure Application Gateway is a resource that can consolidate web traffic to diverse set of resources such as App Services and Function Apps. When there are multiple resources, it is possible to route the traffic independently to each resource. Typically, a custom probe is created to test these independent traffic flow to different resources. When a custom probe is created, the most frequent and sometimes frustrating error response to get is 404 not found http status code even when the direct request to the response returns a successful 200 http status code. This article explains how to configure the listener, the routing rules, the backend pool members, and the backend http setting to get a successful http response from each of the resources just like they would have given when reached directly.

One of the hard to visualize concepts is that the traffic that flows to a backend pool member through the gateway is not pass-through traffic even when TLS is configured as end-to-end. The portion of the flow between the client and the gateway and the gateway and the backend pool member are initiated differently. The clients initiate the first part, and the gateway initiates the second part even when the request body and certain parameters remain the same to keep the data propagated through the gateway to the backend pool member. Throughout this article, we will assume that this break in flow to a resource is seamless and invisible to the user and that it occurs over https to cover the more general case of targeting a resource via the gateway as if it were targeted directly. Add a certificate, self-signed or signed, to enable the client to connect over https.

The first step is configuring the listener properly. A listener listens to a specific combination of IP address and port. There can be only as many probes as there are listeners. So, we can start with one listener that listens to https traffic at the front-end public ip address of the gateway. Since each app service and function app will have a default public endpoint, it is important that the listener be configured as multi-site rather than basic. Since all app services and function have endpoints with “azurewebsites.net” suffix, the at least one hostname that the listener must listen to is “*.azurewebsites.net”.

The next step is to configure the back-end pool members. Path-based routing can route the traffic consolidated by the listener to various back-end pool members. Note that we use the word instead of redirect because while http/s redirection is possible, here we are merely assigning the traffic to go to different destinations based on the relative paths mentioned in a route that we will configure next. The prerequisite to differentiating different targets for that purpose, is to separate out multiple backend pools, preferably one for each target if they are hosted on different endpoints.

The step after that is to configure the backend setting. This setting will specify the port and it is best to pick the hostname from the backend pool. It will require a cert for the data transfer between the gateway and the backend target, so it can take a public key certificate that can be downloaded from the trusted publisher of azurewebsites.net certificates.

The step after that is to configure the route. This step will ask for the listener as well as the target. When we specify the path-based routing, every relative path will have its target and http setting. Since the http setting is specific to the port and all backend pool members in this case leverage the same https port, they can use the same backend setting while the target differ for each path based route. There will be a default case to specify as well.

With these configuration, when the probe is created to reach the listener at the root path, each flow to a  target will be independently routed. This is demonstrated as shown below.

Reference: https://gw-temp-2.booksonsoftware.com

curl -i -k -H "Host: fn-app-temp-1.azurewebsites.net" "https://gw-temp-2.booksonsoftware.com/api/HttpExample/?name=Ravi"  > root.htmlcurl -i -k -H "Host: web-app-temp-1.azurewebsites.net/web/" "https://gw-temp

-2.booksonsoftware.com/"  > root.html

C:\Users\ravib.DESKTOP-1K6OB4E>curl -i -k -H "Host: fn-app-temp-1.azurewebsites.net" "https://gw-temp-2.booksonsoftware.com/api/HttpExample/?name=Ravi"

HTTP/1.1 200 OK

Date: Tue, 04 Jul 2023 02:18:13 GMT

Content-Type: text/plain; charset=utf-8

Transfer-Encoding: chunked

Connection: keep-alive

Request-Context: appId=cid-v1:78e72797-a318-46f1-8401-29719dbd5478

 

Hello, Ravi. This HTTP triggered function executed successfully.

            

Bidding is inherently a form of game with tangible rewards that has become as important as shopping and eCommerce. This software makes it easy for agencies to build auction campaigns. This form of human interaction is one of the most interactive and often more fulfilling than video games and rewards points by virtue of the products sold. In fact, the revenue from the proceeds of an auction and the benefits of an ecosystem around a software offering are as significant as eCommerce with the exception that shopping is inherently personal, and bidding is inherently social. Bidding is widely recognized as a significant contributor to increasing morale for employees of an organization as well as for individual shoppers to find alternatives that are not usually covered by seller platforms. Also, products can be sourced to appeal to demographics targeted by the software. Bids have been another factor in improving customer endearment and loyalty. This article explores how a platform can bring the best practices of cloud solutions to the bidding industry to make it more local and organic with the facility to making it more interesting. The technical solution involves a cloud-based solution comprising of publishers and subscribers, a ledger and set of routines to promote programmability, scripting, and user interface.

Most software makers are focused on trying to lead the market and improve the value offerings of their products to businesses and individuals. Bid point accumulation and redeeming services in favor of purchases are delegated to companies that develop and integrate these services for organizations and their employees. Unlike reward points for loyalty, auctions are more engaging and more satisfying than redeeming or shopping with reward points. The bid points can also be offered as products for purchases or bidding. Commercial applications for bidding are not a new concept as demonstrated by features of ECommerce companies and dedicated applications like DealDash but taking the concept to small and medium businesses or the common person directly, is realized by virtue of repeatable infrastructure or multitenant solutions and that is the objective of this proposal. They are not required to use a specific product and their usage of that product is not collected to determine their bid point grants. The market is full of companies that excel in certain segments of the order and fulfilment of reward points or provide digital redeeming services, but none focus on developing a software development kit that can integrate with businesses and organizations to engage their employees, customers, and end-users. The development of a technical solution and its branding provide immense opportunities for revenue generation.