Azure application gateway and overcoming app services with private links. 

The previous post described a problem. This post describes the solution to that problem in terms of self-explanatory azure command-line statements. 

rg="rg-demo-1" 

planName="asp-demo-1" 

appName="w-a-6" 

vnetName="vnet-demo-1" 

gwyName="gwy-demo-1" 

az group create -n $rg -l "CentralUS" 

az appservice plan create -g $rg -n $planName --sku P1V2 --is-linux 

az webapp list-runtimes --os-type linux 

az webapp create -g $rg -p $planName -n $appName --runtime "DOTNETCORE|6.0" 

az network vnet create -n $vnetName -g $rg --address-prefix 10.0.0.0/16 --subnet-name Default --subnet-prefix 10.0.0.0/24 

az network vnet subnet create -g $rg  --vnet-name $vnetName -n Apps --address-prefixes 10.0.1.0/24 

az network application-gateway create -g $rg -n $gwName --capacity 1 --sku Standard_v2 --vnet-name $vnetName --subnet Default --public-ip-address pip-demo-1 

echo $gwName 

az network application-gateway create -g $rg -n $gwyName --capacity 1 --sku Standard_v2 --vnet-name $vnetName --subnet Default --public-ip-address pip-demo-1 --waf-policy pol-waf-4 --priority 1001 

az network application-gateway create -g $rg -n $gwyName --capacity 1 --sku WAF_v2 --vnet-name $vnetName --subnet Default --public-ip-address pip-demo-1 --waf-policy pol-waf-4 --priority 1001 

az network application-gateway create -g $rg -n $gwyName --capacity 1 --sku WAF_v2 --vnet-name $vnetName --subnet Default --public-ip-address pip-demo-1 --waf-policy "/subscriptions/c9fe3899-495c-475e-9349-dfb252897b9e/resourceGroups/rg-temp/providers/Microsoft.Network/ApplicationGatewayWebApplicationFirewallPolicies/pol-waf-4" --priority 1001 

az network application-gateway address-pool update -g $rg -n appGatewayBackendPool --gateway-name $gwyName --servers "${appName}.azurewebsites.net" 

 az network application-gateway http-settings update -g $rg -n appGatewayBackendHttpSettings --gateway-name $gwyName --host-name-from-backend-pool true 

az network vnet subnet update -g $rg --vnet-name $vnetName -n Apps --disable-private-endpoint-network-policies true 

webAppId=$(az webapp show -g $rg -n $appName --query "id" --out tsv) 

echo $webAppId 

az network private-endpoint create -g $rg -n "${appName}-endpoint" --vnet-name $vnetName --subnet Apps --private-connection-resource-id $webAppId --connection-name "${appName}-connection" --group-id sites 

az network private-dns zone create -g $rg -n privatelink.azurewebsites.net 

az network private-dns link vnet create -g $rg -n "${appName}-dnslink" --registration-enabled false --virtual-network $vnetName --zone-name privatelink.azurewebsites.net 

az network private-endpoint dns-zone-group create -g $rg -n $appName --endpoint-name "${appName}-endpoint" --private-dns-zone privatelink.azurewebsites.net --zone-name privatelink.azurewebsites.net 

az network application-gateway address-pool update -g $rg -n appGatewayBackendPool --gateway-name $gwyName --servers "${appName}.azurewebsites.net" 

az network application-gateway update --name $gwyName --resource-group $rg 

az network application-gateway update --name $gwyName --resource-group $rg 

az webapp config access-restriction add -g $rg -n $appName --rule-name 'WebAppAccess' --priority 200 --action Allow --vnet-name $vnetName --subnet Default 

Caveat: Check for the successful 200 http status code when requesting the gateway url after the address pool update and subsequently when the private dns zone is created. The address pool update is important to get the gateway to refresh again so it recognizes the changes made. 

Sample deployment: gwy3proof.zip 

 

 

Azure application gateway and overcoming app services with private links.

 One of the most frequently encountered and troublesome situations with the integration of application gateway and app services is that the app services might have a private link enabled.  When a client reaches the app service directly with its public fully qualified domain name, there will be a successful response regardless of what the app’s private link connects to but when the client reaches the same app through the application gateway, it will get a 502 or 403. The 502 error occurs when the app service is added as a backend pool member using its ip address rather than the public fully qualified domain name. The 403 error occurs because Microsoft does not route the traffic over the public ip network between the gateway and the app service but chooses to determine the routing. If the app were reachable, a proper response would be 200 for success or 404 for no web page available.

An app service might want a private link for many purposes but most of them would have been equally solvable via access rules. For example, if an app service wanted to communicate with another app service privately, it could establish a private link between the two comprising of private endpoints designated with a private ip address for each of the two app services. But this can also be equivalently set up with an IP address access rule that puts the remote app service as the allowed traffic origin in the local configuration of the app service. The rule would be specified in both app services’ access restrictions. When such a rule is added, the gateway must also be allowed with its own access rule in both app services access restrictions assuming clients could target each app service independently through the gateway. By maintaining connectivity to be over IP versus links, the app services will continue to give successful response through the application gateway which is a layer-7 reverse proxy and load balancer.

Therefore, the rule of thumb should be to not require private endpoint for any app service. But if the private endpoints for the app services cannot be removed, there are a few options to try although none is guaranteed to work. These options are explained along with the concepts they make use of.

The first is that the virtual network in which the application gateway is deployed, can communicate with app services over either a service endpoint or a private endpoint. Care must be taken to ensure that this endpoint is setup in the subnet to which the application gateway is deployed in the virtual network. The presence of such an endpoint alone does not guarantee that the client will not get the dreaded 502/403 errors when trying to reach an app service through the gateway because by nature of the application gateway as a reverse proxy, its frontend communication with the client is split from the backend communication with the app service. Also, the app service might be deployed in its own virtual network independent from that of the application gateway. The app service and application gateway have independent public domains when they are initially setup with the app service having an azurewebsites.net domain by default and the application gateway having the cloudapp.net domain by default. Their public access and http/https-based operations makes them more web facing rather than infrastructure facing resources. The private links that they create with their respective virtual networks or a shared one whenever possible gives them a private ip address on the infrastructure side but provides no routing which must be established independently. This brings us to the second point.

The second is that the infrastructure can determine the routing of traffic between the gateway and the app service. If they are in two different virtual networks, those can be peered to allow communication originating anywhere in one network to reach anywhere else in the other network. Putting all app services as well as the gateway in one virtual network might solve that problem for the infrastructure but the app services might still be in different subnets. Virtual network peering allows many to many communications between the participating networks otherwise private endpoints enable one-to-one communication between the gateway and the app service.  Creating that private endpoint in the subnet to which the gateway is deployed is helpful to gather private inbound traffic to the gateway. This is reflected in the private addresses assigned to the resources. In the peered network, the resource will have a private address on each side of the peering. With a private endpoint only one private ip address is assigned to the resource. This same address can be specified as the representation of the app service to be included as the backend pool member of the application gateway just as much as the public ip address of the app service but its best to refer to the app service with the builitin drop-down option and the app-services’ public fully qualified domain name.

This brings us to the third point, that name, address, and routing determine the way gateway communicates with the app services and remains true regardless of whether it is over private network or public network. The name resolution is facilitated by an private DNS server when setting up a private endpoint for the app service to communicate with the gateway. The address is issues immediately on establishing a private endpoint in the subnet/vnet combination specified during the creation of the endpoint. The routing however is governed not just by the peering or the specific routing rules but also by network security groups and access control lists. This is where care must be taken to reuse the same NSG, ACL or policy across all participating subnets/vnets.  Also routing and network and for that matter private connectivity need not be established if we allow the built-in capabilities for routing that comes with service endpoints and Microsoft based routing.

Whether we use the first, second or the third option, it must be noted that the ways to validate the connections are with the help of probe and client calls and neither can be done away with. While the probe tests the backend responses, the client calls test the front-end. All other features of the application gateway maintained same, the communication with backend pool members public over public ip and private link connectivity have been mentioned in this article. While the IP connectivity allows us to rely on the network to route the traffic, the private link connectivity requires us to create inbounds and outbounds properly along with necessary routing and associated rules.

Reference: Article on how to author rewrite rules.

 

 

Configuring Azure Application Gateway

Azure Application Gateway is a web traffic load balancer that enables you to manage traffic to your web applications. It is a sophisticated resource comprising of several features such as a reverse proxy, firewall, http listener, backend pool members, http settings and rules for routing traffic to the backend. Configuring the Azure Application Gateway often takes time and effort with the overwhelming number of features and cases to cover. This article explains what to do and what not to do when routing traffic to a backend web application that involves users to login via Azure Active Directory. Such is the complexity of path-based routing, host header-based application redirections and location header rewrites, that this use case is often the most troublesome and requires call out.

When compared to Azure Front Door that offers global load balancing across regions and is preferred for direct connectivity to the internet, App Gateway is suitable for backends contained within a region or is not open to the public internet. Both provide web application firewall and App Gateway provides features such as SSL/TLS termination, autoscaling, zone-redundancy, static VIP, Ingress controller for Azure Kubernetes Service resources, URL based routing, multiple site hosting, redirection, session affinity, websocket, and http/2 traffic, connection draining, custom error pages, rewrite of http headers and URL, and sizing. It is preferable to choose the v2 sku that includes the WAF.

With a normal web application or function application that does not involve authentication and redirect from the Azure AAD login servers, the http settings are typically set for use in path-based routing with the port, protocol, path, timeout, and other parameters specified. The port is typically 443, the path is “/”, the timeout can be 30s and the host header is overridden by “picking a value from the backend pool” which is the web apps default endpoint, say appName.azurewebsites.net.

With the login-based applications that require users to authenticate with Azure AAD, the host header used to specify the redirect conflicts with the requirement of routing all traffic through the gateway. So, after the successful username-password entry by the user, they are redirected to the “appName.azurewebsites.net/” instead of a url based on the application gateway + “/<app-specific-path>”. If the host name override is specified, the path-based routing breaks.

The fix to have users who successfully login to access the app through the application gateway involves writing rewrite rules in addition to path-based routing rules where the location header in those successful login responses is rewritten to replace the appName.azurewebsites.net to application gateway endpoint.  Since all successful responses from the login server for an application redirect have Location header specified, the users upon login are successfully redirected to the web application via the application gateway url and their cookie is set correctly.

Since http settings cannot have both host-header override option enabled as well as disabled for regular and redirected access to the web application, it is suggested to have two sets of settings that map to the backend web application and setup independently one each for the distinct cases of regular and redirected access to the web application. The port associated with an http setting is unique to that setting, so different http settings will require different ports, say 443 and 8443. The web app as a backend pool member of the application gateway can get traffic from either set of rules to cover the regular versus redirected accesses.

The literature says that this combination of path-based routing rule-sets and rewrite based rule-sets can be avoided by using a combined routing that involves host-name pass-through setup such that the application gateway and the web application are put as part of the same web application but those who deploy application gateways know that they might not have control over the creation of a custom domain from the network, so the de facto way is to fall back to path-based routing and cover both the regular and redirected accesses to the web application.

Providing smart self-service capabilities:

 

The following are some use cases for building smart self-service portal for data science in a health-care organization.  

 

Use case 1) Making recommendations. 

 

Users often have a variety of choices when they come to request resources or infrastructure from public cloud. One way to help them would be to tell them what others like them have been leasing from the cloud provider. If we search a large group of people and smaller set with tastes like that of customers in terms of what they have been liking or borrowing, then we can make a ranked list of suggestions by finding the people who are similar and combining their choices to make a list. 

 

Different people and their preferences can be represented as a nested dictionary. To find similarity among people, we could use some sort of similarity measures based on Euclidean distance or Pearson correlation. We can then score the items by producing a weighted score that ranks the participants. 

 

Use case 2) Discovering groups. 

 

Users can be pigeonholed by clustering and categorizing them based on their requests.  By determining the frequency of a certain type of resource or infrastructure units in the requests made by them, we may be able to find those who are similar in their needs or have similar planning. Such a result could be very useful in searching, listing and discovering the themes in the vast number of ad hoc requests made over time. 

 

We can employ hierarchical clustering or K-means clustering to categorize the requests and relate personas to this leasing pattern. 

 

Use case 3) Searching and Ranking 

 

Searching and Ranking form the core of analytics for a user who wants to filter through support documents and literature. Very often one will get a buzzword to search for and help oneself, but s/he might not know the site layout or the categories to walk down to filter it. 

 

Such a user can be helped with a page ranking that gives a weighted score based on the fraction of in-references to out references. Other ways of arranging them or finding results based on clustering can also help. 

 

Use case 4) Optimization. 

 

Users may want the self-service capabilities to search for the lowest cost set of resources when there are many choices for their solution when there are many different solutions. Optimization finds the best solution to a problem by trying many different infrastructure solutions and workload combinations and scoring them to determine their best fit and optimum use. Right sizing the resources and determining the scope where the improvements or upgrades need to be made or the scale out of units to serve peak usage are some of the variations that can be optimized. 

 

A cost function can be used to determine the costs for different combinations. Then hill climbing or simulated annealing can be used to reduce the cost towards the optimum.

 

Use case 5) Document Filtering 

 

This use case is about training a classifier to classify documents that are published as resource usages and logs. A classifier for this purpose starts off very uncertain and increases in certainty as it learns which features are important for making a distinction.  By classifying the documents, we expect to find resources that require attention either immediately or soon. 

 

This will require a training set as well as a test set. The training set will be used to tune the classifier and the classifier will then be run on the test sample usually with a 80/20 split. If the features are independent of each other, they can be used in a naïve Bayes classifier. 

 

Use Case 6) Determine users that might benefit from premium infrastructure and service levels. 

 

To predict the likelihood that a user will invest more than others, we collect information and put it in say a table with the attributes that we pick from server logs, such as location, help articles read, pages viewed, and whether premium SKUs were chosen. 

 

Then, decision tree classifier can be used to construct a tree from real data that is easy to understand where each node tells how important it is and how it will impact the outcome. 

 

Use case 7) Building price models for premium service levels and their users. 

 

A model for this purpose will predict a price. Initially, the prices will be manually computed in a training set but it will be used to predict once the model has learned from the training set and is applied to a testing set. By finding the items that are like the item that interests the customer, the algorithm can average their prices and make a guess at what the price should be for this item.