Azure Data Factory and its programmability.

 As with any resource in the Azure public cloud that can be provisioned with an ARM template, Azure Data Factory provides several way to automate its operations such as with the Command Line Interface, Its RESTful APIs and SDK in various languages. Since calling over the public internet with REST APIs allows the caller and the target to be in different networks, this article suggests some API calls for the frequently encountered usages. Specifically, it calls out two use cases: 1) to trigger a pipeline job  and 2) to link external storage accounts so that new pipelines can be created.

Use case 1: Triggering a pipeline job:

This is done with the help of a

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelines/{pipelineName}/createRun?api-version=2018-06-01

Request

With the parameters as

factoryName – for the name of the ADF

pipelineName – for the name of the pipeline within the ADF

resourceGroupName – for the name of the resource group

subscription id – for the id of the subscription

api-version – this could be set to 2018-06-01

The Authorization header will have the Bearer token corresponding to identity with which to execute the request.

The request body can take additional parameters as:
{

  "OutputBlobNameList": [

    "exampleoutput.csv"

  ]

}

The result will usually have a runId with a GUID as value.

Use case 2: Creating a linked service to an external Amazon S3 compatible storage such as ones hosted on-premise.

This is done with the help of a

PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/linkedservices/{linkedServiceName}?api-version=2018-06-01

request

again, with the same parameters as in use case 1.

The request header will carry an authorization with Bearer token of the identity of the caller.

The request body will carry the details for the corresponding service.

{

  "properties": {

    "type": "AzureStorage",

    "typeProperties": {

      "connectionString": {

        "type": "SecureString",

        "value": "DefaultEndpointsProtocol=https;AccountName=examplestorageaccount;AccountKey=<storage key>"

      }

    }

  }

}

The example here shows AzureStorage but for Amazon S3 compatible storage, an account name, an access key and secret will suffice as payload to the API request above.

The response will be similar to what’s shown below except that it will be for Amazon S3 Compatible storage.

{

  "id": "/subscriptions/12345678-1234-1234-1234-12345678abc/resourceGroups/exampleResourceGroup/providers/Microsoft.DataFactory/factories/exampleFactoryName/linkedservices/exampleLinkedService",

  "name": "exampleLinkedService",

  "type": "Microsoft.DataFactory/factories/linkedservices",

  "properties": {

    "type": "AzureStorage",

    "typeProperties": {

      "connectionString": {

        "type": "SecureString",

        "value": "**********"

      },

      "encryptedCredential": "ew0KICAiVmVyc2lvbiI6ICIyMDE3LTExLTMwIiwNCiAgIlByb3RlY3Rpb25Nb2RlIjogIktleSIsDQogICJTZWNyZXRDb250ZW50VHlwZSI6ICJQbGFpbnRleHQiLA0KICAiQ3JlZGVudGlhbElkIjogIkRGLURPR0ZPT0QtWUFOWkhBTkctV1VfM2FiMTk0NjYtNWUxNi00NzU1LWJlNzktMjI2ZTVmZWU3YzY0Ig0KfQ=="

    }

  },

  "etag": "0a0062d4-0000-0000-0000-5b245bcf0000"

}

Reference:

1.       https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-s3-compatible-storage?tabs=data-factory

2.       https://learn.microsoft.com/en-us/rest/api/datafactory/linked-services/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP#linkedservices_create

3.       Prior articles on IaC: IaCResolutionsPart41.docx