Azure Databricks Cluster usage report.
1.
From UI:
a.
Click compute in the sidebar.
b.
Select the compute resource to view metrics for
c.
Click the metrics tab.
d.
Refer to the CPU utilization cluster metrics.
This is based on total CPU seconds’ cost. The metric is averaged out based on
which time interval is displayed in the chart. The default time interval is the
last hour.
e.
There is also a GPU metric chart. This is based
on the percentage of GPU utilization also averaged out for the chosen time
interval.
2.
From PowerShell:
a.
Install-Module -Name DatabricksPS
b.
RESPONSE=$(curl
http://localhost:50342/oauth2/token --data
"resource=https://management.azure.com/" -H Metadata:true -s)
c.
ACCESS_TOKEN=$(echo $RESPONSE | python -c
'import sys, json; print (json.load(sys.stdin)["access_token"])')
d.
accesstoken=$ACCESS_TOKEN
e.
$apiUrl="https://management.azure.com/subscriptions/656e67c6-f810-4ea6-8b89-636dd0b6774c/resourceGroups/rg-temp/providers/Microsoft.Databricks/workspaces/wks-rg-temp-1?api-version=2023-02-01"
f.
$headers=@{"Authorization"=
"${accesstoken}"}
g.
Invoke-RestMethod -Method "GET" -Uri
$apiUrl -headers $headers
Sample Response:
```
properties :
@{managedResourceGroupId=/subscriptions/656e67c6-f810-4ea6-8b89-636dd0b6774c/resourceGroups/dbw-rg-temp-1;
parameters=;
provisioningState=Succeeded; authorizations=System.Object[]; createdBy=;
updatedBy=; workspaceId=8965842579407484;
workspaceUrl=adb-8965842579407484.4.azuredatabricks.net;
createdDateTime=7/9/2023 8:14:11 PM}
id :
/subscriptions/656e67c6-f810-4ea6-8b89-636dd0b6774c/resourceGroups/rg-temp/providers/Microsoft.Databricks/workspaces/wks-rg-temp-1
name : wks-rg-temp-1
type :
Microsoft.Databricks/workspaces
sku : @{name=premium}
location : centralus
tags :
```
h.
$accesstoken=”dapie12c.....b3a2-2”
i.
$headers=@{"Authorization"=
"Bearer ${accesstoken}"}
j.
$apiUrl="https://adb-8965...484.4.azuredatabricks.net/api/2.0/clusters/get"
# curl --request GET "https://${apiUrl}/api/2.0/clusters/get" --header "Authorization: Bearer ${accesstoken}" --data '{ "cluster_id":
"1234-567890-a12bcde3" }'
Sample Response:
PS /home/ravi> Invoke-RestMethod -Method "GET" -Uri $apiUrl
-headers $headers -Body '{ "cluster_id":
"0709-203735-epaybeni" }'
cluster_id :
0709-203735-epaybeni
creator_user_name :
ravibeta@hotmail.com
driver : @{private_ip=10.139.64.4;
public_dns=40.113.230.98; node_id=6a94d151f85c48f691ecfa2b501ddb8c;
instance_id=9f3e9e6cbf344608942f28e5db5c22a2;
start_timestamp=1688935207640; node_attributes=; host_private_ip=10.139.0.4}
spark_context_id :
2865750645175411723
driver_healthy : True
jdbc_port :
10000
cluster_name : Ravi
Rajamani's Personal Compute Cluster
spark_version :
13.2.x-cpu-ml-scala2.12
spark_conf :
@{spark.databricks.delta.preview.enabled=true;
spark.databricks.cluster.profile=singleNode; spark.master=local[*, 4]}
azure_attributes :
@{first_on_demand=1; availability=ON_DEMAND_AZURE; spot_bid_max_price=-1}
node_type_id :
Standard_DS3_v2
driver_node_type_id :
Standard_DS3_v2
custom_tags :
@{ResourceClass=SingleNode}
autotermination_minutes : 4320
enable_elastic_disk : True
disk_spec :
cluster_source : UI
single_user_name :
ravibeta@hotmail.com
policy_id :
00164B5BAEB11244
enable_local_disk_encryption : False
instance_source :
@{node_type_id=Standard_DS3_v2}
driver_instance_source :
@{node_type_id=Standard_DS3_v2}
data_security_mode :
LEGACY_SINGLE_USER_STANDARD
runtime_engine :
STANDARD
effective_spark_version :
13.2.x-cpu-ml-scala2.12
state :
RUNNING
state_message :
start_time :
1688935055795
last_state_loss_time : 0
last_activity_time :
1688935483841
last_restarted_time :
1688935294611
num_workers : 0
cluster_memory_mb :
14336
cluster_cores : 4
default_tags :
@{Vendor=Databricks; Creator=ravibeta@hotmail.com; ClusterName=Ravi Rajamani's
Personal Compute Cluster;
ClusterId=0709-203735-epaybeni}
init_scripts_safe_mode :
False
k.
Set-DatabricksEnvironment -AccessToken $accesstoken
-ApiRootUrl $apiUrl
l.
Get-DatabricksCluster | Stop-DatabricksCluster
The above uses PAT tokens. It can also use Azure AD as follows:
PS /home/ravi> $credUser = Get-Credential
PowerShell credential request
Enter your credentials.
User: ravibeta@hotmail.com
Password for user ravibeta@hotmail.com: ****************
PS /home/ravi> $tenantId =
"1f4c33e1-e960-43bf-a135-6db8b82b6885"; $subscriptionId =
"656e67c6-f810-4ea6-8b89-636dd0b6774c";
PS /home/ravi> $resourceGroupName = "rg-temp"
PS /home/ravi> $workspaceName="ravibeta@hotmail.com"
PS /home/ravi>
$azureResourceId="/subscriptions/656e67c6-f810-4ea6-8b89-636dd0b6774c/resourceGroups/rg-temp/providers/Microsoft.Databricks/workspaces/wks-rg-temp-1"
PS /home/ravi> $workspaceName="wks-rg-temp-1"
PS /home/ravi>
$clientId="50996fd9-da74-4f41-b262-490d074bc807"
PS /home/ravi>
$apiUrl="https://adb-8965842579407484.4.azuredatabricks.net/"
PS /home/ravi> Set-DatabricksEnvironment -ClientID $clientId
-Credential $credUser -AzureResourceID $azureResourceId -TenantID $tenantId
-ApiRootUrl $apiUrl
Connect-AzAccount -UseDeviceAuthentication will not be sufficient for above
cmdlet.
The
same credential can be used with more than one environment.
No comments:
Post a Comment