Introduction:

This article is a continuation of the series of articles starting with the description of SignalR service. In this article, we continue with our study of Azure Stream Analytics from the last article. We were comparing Apache Flink and Kafka with The Azure Stream Analytics and were observing comparisons between Flink and Azure stream analytics to determine watermarks and sequence events along their timelines. We now investigate automations for performing stream analysis.

Jobs and clusters form the two main components of the stream analytics. When the job is created, the deployment can be validated. The job itself is represented by an ARM template which is a JSON notation and it defines the infrastructure and configuration for the project. The template uses declarative syntax so there is no need to write commands to create the deployment. The template takes parameters such as the location, stream analytics job name and number of streaming units which is then applied to the resource and the job is created.

Deployment using the template can be kicked off directly from the CLI, PowerShell, portal and SDK. All of these provide programmability options.  Resources can be cleaned up by deleting the resource group.

The input to the job can also be configured with the help of a Powershell cmdlet. The New-AzStreamAnalyticsInput cmdlet takes the job name, job input name and resource group name, and the job input definition as parameters. Even the blob storage can be passed in as an input. AccessPolicyKey and Shared AccessKey are derived from the connection strings to the data source. The output to the job is similarly configured with the help of a JobOutputDefinition and it takes the storage account access key as a parameter. Blobs will be stored in a container from that account. Finally the transformation query can be specified via the New-AzStreamAnalyticsTransformation cmdlet which takes the job name, job transformation name, resource group name, and the job transformation definition as parameters. This declaration contains a query property that defines the transformation query.

The Start-AzStreamAnalyticsJob cmdlet takes the job name, resource group name, output start mode and startTime as parameters and kick starts the job.