Azure Data factory and self-hosted Integration Runtime: 

This is a continuation of the articles on Azure Data Platform as they appear here. Azure Data Factory is a managed cloud service from the Azure public cloud that supports data migration and integration between networks. This article focuses on setting up a site-to-site VPN for connecting on-premises to the Azure cloud.

Azure, self-hosted and Azure-SSIS integration runtimes are the flavors of compute infrastructure that the Azure Data Factory uses to provide data integration capabilities across different network environments.  These include executing a data flow in a managed Azure compute environment, copying data across data stores in a public or private networks, dispatching and monitoring transformation activities and natively executing SQL Server integration services packages in a managed Azure compute environment. Out of these, the self-hosted runtime can be used for data movement and activity dispatch across on-premises and Azure networks. Self-hosted integration runtime cannot be used for managed compute, autoscale and dataflow but it can be used for on-premises data access, private link/private endpoint and custom component/driver. It requires the on-premises network to be connected to Azure via ExpressRoute or VPN. The private endpoints are managed by the Azure Data Factory Service. 

The setting up of site-to-site connection involves the use of Azure Virtual WAN.

An IPSec/IKE VPN connection is required to connect to Azure resources over virtual WAN. This involves a VPN device located on-premises that has an externally facing public IP address assigned to it. The steps involved to set this up are: 1. Create a virtual WAN, 2. Configure virtual hub Basic settings, 3. Configure site-to-site VPN gateway settings. 4. Create a site, 5. Connect a site to a virtual hub, 6. Connect a VPN site to a virtual hub, 7. Connect a VNet to a virtual hub, 8. Download a configuration file, and 9. View or edit the VPN gateway.

The pre-requisites on the Azure side of the connection are 1. An Azure subscription, 2. A virtual network without any existing virtual network gateways and IP address range to use for the virtual hub private address space.

The Virtual WAN is actually a set of resources collectively insantiated to represent a virtual overlay of the Azure network. It requires subscription, resource group, location, name and type as Basic or Standard. Basic is used to create only the site-to-site connection while Standard has advanced features.

A virtual hub is required to contain a dedicated gateway for site-to-site functionality. It requires subscription, location, name, private address space in CiDR notation, capacity in terms of routing infrastructure units, routing preference and a router Autonomous System Number.

The site-to-site connection is configured with the router ASN, Gateway scale units and routing preference as Microsoft network or Internet.

Next, a site is configured in the Virtual WAN to correspond to the physical location from where the connections will be initiated. It requires the location, name, device vendor as Citrix, Cisco, Barracuda, etc. and a private address space. Links can be added to represent the physical links at the location.

When the site is created, it can be viewed from the virtual WAN page. The VPN site is then connected to the virtual hub. The connection of sites requires settings such as a Pre-shared key, protocol such as IKEv2 or IKEv1, IPSec as default or custom, a flag to indicate if the default route will propagate so that virtual networks connecting to the hub will have this gateway reachability added to their routing table, a flag to indicate if the policy based traffic selector must be left disabled, a flag to indicate if the traffic selector must be configured and a connection mode selected from default, initiator only or responder only choices.

When the connection is made, its status will show as updating. After the updating completes, the site shows the connection and connectivity status. A virtual network can then be connected and the VPN device configuration information can be downloaded.