The following is a list of some errors and resolutions encountered with deploying Overwatch dashboards:

1.       Overwatch dashboard fails with errors mentioning missing tables.

The databases that Overwatch needs are consumer database usually named overwatch and the ETL database, usually named overwatch_etl. These databases are deployed with the Overwatch notebook runners and there are two versions the 70 and 71. The latter version requires the storage account to be created and a csv to be uploaded to the deployment folder within the overwatch container or bucket in a public cloud storage account. The csv requires a mount location referred to as the storage prefix where all the files associated with the creation and use of database are kept. There are two files there, one each for overwatch consumer database and overwatch_etl database which persist the database outside the catalog of the databricks instance.

When the notebook runs, the tables are created within the catalog and the associated file on the storage account. Over sixty jobs are run to create these tables and eventually all the tables appear in the catalog. Due to the high number of jobs, failures are common and the tables are not all populated. Rerunning the notebook a few times, helps to close the gap towards a complete database.

 

2.       Overwatch has mismatching files and/or database and must need to be redeployed but the starting point is not clean

Due to the versions of notebook used and the intermittent failures from executing any one, it is quite likely that a redeploy from a clean slate is required. Deleting just the persistence files from the storage account will not help because the catalog and the databricks instance might keep a mention of stale configuration. Although a cleanup script is available along with the Overwatch deployment notebooks, it is best to execute the following command for a speedy resolution:

DROP DATABASE overwatch_etl CASCADE;

DROP DATABASE overwatch CASCADE;

-- CLEAR CACHE;

This will delete the associated files from the storage account as well. It is also advisable that if the Overwatch is being upgraded even for a stale deployment, it could be followed up by recreating the storage account container and mounting it on the databricks cluster.

 

3.       When the storage prefix refers to the remote location via the abfss@container.storage.dfs.core.windows.net naming scheme, frequently the unauthorized error displays.

Although mounts are deprecated and abfss is relatively newer than the mounts, creating a mount initially helps prevent repeated resolution for every name lookup. This can be done with the following script:

configs = {"fs.azure.account.auth.type": "OAuth",

          "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",

          "fs.azure.account.oauth2.client.id": "<application-id>,

          "fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope="<scope-key>",key="<key-name>"),

          "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<tenant-id>/oauth2/token"}

#dbutils.fs.unmount("/mnt/overwatch-wd")

dbutils.fs.mount(

  source = "abfss://container@storageaccountname.dfs.core.windows.net/",

  mount_point = "/mnt/overwatch-wd",

  extra_configs = configs)

 

dbutils.fs.ls("/mnt/overwatch-wd")