Multitenancy and network virtualization:  

   

The previous article talked about Application Virtualization. Resources hosting the applications can also be partitioned into virtual networks. This article talks about network virtualization.

Virtual networks or vNets for short, are fundamental to locate resources such that they can securely communicate with each other, the internet, and on-premises networks. VNet is like a traditional network placed in a private datacenter but it brings additional benefits of scale, availability and isolation.

Communication between resources, communication of those resources with the internet and communication with on-premises resources, filtering network traffic, routing network traffic and integration with other cloud services are all made possible by virtual networks.

Outbound communication from resources to the internet is available by default. The inbound communication to a resource from the internet is achieved by assigning a public IP address or a public load balancer. Communication between resources can be achieved via a virtual network, a virtual network service endpoint and VNet peering. Communication with on-premises resources is achieved by Point-to-site virtual private network, Site-to-site VPN and Azure ExpressRoute. Network traffic can be filtered between subnets using either or both network security groups and network virtual appliances.

When these virtual networks are created in the cloud, the cloud providers automatically route traffic between the subnets, connected virtual networks, on-premises networks, and the internet by default. Overrides can be specified using the route tables and Border gateway protocol routes. Integrating cloud services to a virtual network enables private access to the service from virtual machines or compute resources in the virtual network by dedicating the services to a virtual network, using private link to access privately or by extending the virtual network to the service using a service endpoint. Service endpoint allows services resources to be secured to the virtual network.

There are limits to how many resources can be placed within a virtual network but there are options to increase those limits. Virtual network and subnets span all the availability zones in a region. Even if a resource is zonal, the virtual network does not need to be divided between the availability zones.

Network and storage are low-cost resources, but network administrators know that when they keep running out of IP addresses, these don’t seem to be true. The end-user might get a virtual network for free by requesting it from the cloud in a free tier but there are pricing applicable to premium tiers and SKUs.

Some of the best practices for virtual network are universal design principles. For example, the address-spaces must be ensured to be non-overlapping. The subnets should not cover the entire address space of the virtual network. There must be fewer large virtual networks rather than smaller numerous ones. The virtual networks must be secured by assigning Network security groups to the subnets underlying the vNet.

Name resolution applies to virtual networks. The name resolution to an IP address depends on whether there is a single instance or many instances of the multitenant application. For example, a CNAME for the custom domain of a tenant might have a value pointing to a multi-part subdomain of the multitenant application solution provider. Since this provider might want to set up proper routing to multiple instances, they might have a CNAME record for subdomains of their individual instance to route to that instance. They will also have an A name record for that specific instance to point to the IP address of the provider’s domain name. This chain of records resolves the requests for the custom domain to the IP address of the instance within the multiple instances deployed by the provider.

Validation of custom domains is a necessity for the tenants to be onboarded. Without validation, tenants might accidentally or maliciously park a domain name. Typos in custom domain names are encountered often. Parking leads to an error for others wanting to use their custom domain with the message that the domain name is already in use. Domain names especially within a self-service or automated process require a domain verification step. A CNAME record or a DNS TXT record might be added to reserve the domain name until the verification is completed. Private DNS domains as well as custom domain names are supported with private virtual networks. 

Reference: https://1drv.ms/w/s!Ashlm-Nw-wnWhLMfc6pdJbQZ6XiPWA?e=fBoKcN

 

Multitenancy and application virtualization:

 

Application virtualization enables applications to be available to end-users and their computers without requiring direct installation. It is made possible through a process called sequencing the application, which enables each application to run in its own self-contained virtual environment on the client computer. The sequenced applications are isolated from one another, and they can continue to interact with the client computer.

 

There is a client made available to leverage application virtualization and lets the end users interact with the application once it has been published. The client manages the virtual environment in which the applications run. Publishing is the process by which the application is made available to the client computer. This process copies the virtual application icons and shortcuts to the computer as well as the package definition and the file type association to the computer. When the content is made available to the end user’s computer, it behaves as if it is installed locally.

 

The package content can also be copied locally into one or more Application virtualization servers so that it can be streamed down to the clients on demand and cached locally. File servers and web servers can also be used as streaming servers or copied directly to a destination computer as in the case of an electronic software distribution system. In a multi-server implementation, maintaining the package content, and keeping it up to date on all the streaming servers requires a comprehensive package management system. When the size of the organization is large and the number of client computers is large and geographically spread out, a comprehensive packaging and publishing helps. Managing the package to ensure that the applications are available and accessible to all users is important.

 

Multitenant solutions have often found a need for an application store to collect all the applications that can be used by their users on their tenant resources. Those users download the application after consent and purchase and then use it locally. This experience can be improved with the application virtualization without involving the user for unnecessary setup and cleanup operations.

One of the biggest appeals of using this technology is its potential to enable the rewriting of existing commercial software. Rewrites and making the software available in the cloud via an application store might sound unappetizing or less glamorous than a new groundbreaking product but it aligns with the drive towards pushing commoditization of technology deeper into the cloud. This calls for actions to go beyond the marketplace store to share applications as a slice is reserved for commitments from the public cloud provider in the form equivalent to enterprise blocks and application patterns for repurposed, streamlined, and optimized workflows with extensibility for cross-cloud integrations.  

 

The Lessons from enabling search for multitenant applications:

Multitenant search solutions provide search capabilities to any number of tenants who cannot see or share data of any other tenant. Searching often requires content virtualization across different sources where the content can be considered a shared resource. Providing content isolation with search capabilities is the topic of this article.

A search service usually builds an index of the content so that the queries for the content can return results efficiently since the index is far easier and more efficient to lookup than traversing the entire content store. The search service can be considered equivalent to a database service and indeed search can be provided out of box from a database. All the multitenant considerations for database services hold for the search service as well. Each index within a search service has its own schema, which is defined by several customizable fields. Search services, indexes, fields and documents are infrastructure and there is little or no user control.

Some considerations for multitenant applications:

Multitenant applications must effectively distribute resources while preserving privacy between the tenants. A few considerations must be made when designing the architecture for such an application.

-          Tenant Isolation – No tenants must have unauthorized or unwarranted access to the data of other tenants

-          Cloud resource cost – any solution must remain cost-effective.

-          Ease of operation – the impact on the application’s operations and complexity is important.

-          Global Footprint – Applications may need to effectively serve tenants which are distributed across the globe

-          Scalability – applications may need to reconcile between maintaining a sufficiently low level of complexity and scaling to the number and size of workloads.

A uniform scope is assumed where each tenant is a whole instance of an application, but they can handle many smaller scopes. If the service per tenant and index per tenant are not sufficiently small scopes, then the index can be utilized for a finer degree of granularity. A field can be added to an index to make it behave differently to different client endpoints. This helps to achieve separate user accounts, separate permission levels, and even separate applications.

Reference: https://1drv.ms/w/s!Ashlm-Nw-wnWhLMfc6pdJbQZ6XiPWA?e=fBoKcN

 

 The Lessons from enabling search for multitenant applications:

Multitenant search solutions provide search capabilities to any number of tenants who cannot see or share data of any other tenant. Searching often requires content virtualization across different sources where the content can be considered a shared resource. Providing content isolation with search capabilities is the topic of this article.

A search service usually builds an index of the content so that the queries for the content can return results efficiently since the index is far easier and efficient to lookup than traversing the entire content store. The search service can be considered equivalent to a database service and indeed search can be provided out of box from a database. All the multitenant considerations for database services hold for the search service as well. Each index within a search service has its own schema, which is defined by several customizable fields. Search services, indexes, fields and documents are infrastructure and there is little or no user control. A search service can enable tenant isolation via one of the following three methods.

The application developer consumes one or more search services and divide their tenants among services, indexes or both. The methods to provide tenant isolation, scale, cost efficiency, ease of operations and reduced footprint have the following patterns:

1.       One index per tenant: This pattern has an index dedicated to a tenant within a search service that is shared with other tenants.

2.       One service per tenant: This pattern has its own dedicated service offering the highest levels of data and workload separation.

3.       A mix of both: Larger more active tenants are assigned dedicated services and the smaller ones are assigned individual indexes within the shared services.

In an index-per-tenant model, tenants achieve data isolation because all search requests and document operations are issued at an index level. The application layer directs tenant traffic to proper indexes while managing the resources at the service levels across all tenants. This model is simple for the application developers and allows them to oversubscribe the capacity among the application’s tenants. The tradeoff is the restrictions to handle peak load where most or all of the tenants are highly active. The index per tenant model provides the basis for a variable cost model. It may not be the most efficient for an application with a global footprint.

In the service-per-tenant model, each tenant has its own search service. In this model, each application achieves a maximum level of isolation for its tenants. Each service has dedicated storage and throughput for handling search request as well as separate API keys. This pattern is effective when each tenant has a large footprint, or the workload has little variability from tenant to tenant since resources are not shared.

With the mix for both index-per-tenant and service-per-tenant strategies, an application’s largest tenants can occupy dedicated services while the numerous less active, smaller tenants can occupy indexes in a shared service. It ensures that the largest tenants can derive high performance from the service, while protecting the smaller tenants from noisy neighbors. There is some planning required to classify tenants into those that required dedicated and those that can leverage a shared service.

The above patterns assume a uniform scope where each tenant is a whole instance of an application but they can handle many smaller scopes. If the service per tenant and index per tenant are not sufficiently small scopes, then the index can be utilized for a finer degree of granularity. A field can be added to an index to make it behave differently to different client endpoints. This helps to achieve separate user accounts, separate permission levels, and even completely separate applications.

Reference: https://1drv.ms/w/s!Ashlm-Nw-wnWhLMfc6pdJbQZ6XiPWA?e=fBoKcN

 

 The Lessons from the overuse of multitenancy – quotas and reservations:

The computing power and storage matters to the users more than how it is provisioned and where. The ease of requesting the compute, the cleanliness of instantiating and deleting the associated resource, the ability to migrate applications and services far outweigh any consideration of the vendor or the mechanism just so long as the performance, security and costs are comparable.

One of the classic indicators of overengineering is when there are many dials and knobs. There are two specific problems with these. First, it is an enormous burden for the users to figure out when and how to tune them optimally. Second, it increases the chance of oversight, improper use, and mistakes.

Consider the case brought forward by Alphonse Chapanis, a US Army Air Force aero medical lab psychologist who interviewed many pilots of the B-16 ‘Flying fortress’ bomber aircraft and were distressed to have landed a perfectly good plane after their mission only to see it suffer damage on stopping slowly. He determined that the pilots could not distinguish between the switches between the flaps and the landing gear that were co-located and retracted the landing gear when they meant to retract the flaps causing a nosedive and damage to the wings, propeller, and the underside of the aircraft. This incident review resulted in the switches having the landing wheel and wing flap design introduced to tell them apart and even to place them on opposite sides of the controller box. The savings from the prevented aircraft loss thereafter are a testament to the proper placement and use of controls.

Containers are governed by the quotas and limits of the underlying system to manage the shared resources. Solaris introduced partitions for different flavors, quotas and limits for file-systems and soon applications would start failing in one partition because they didn't have enough resources on this as they earlier had on the undivided host. One of the reasons Solaris zones suffered from application migration considerations from the operating system consumers is that they kept running into application failures due to limits. Both Linux containers and Solaris zones must be understood as operating system features and not as standalone products for virtualization technologies.

Linux containers have the advantage that they provide the same environment without the overhead of a separate kernel and hardware. This translates to lightning-fast availability of an isolated shell as well as their reduction of failure points. The hardware is not necessarily partitioned per virtual machine and the separation of kernel does play significantly in terms of performance because the emulation is different from the real especially when compared to a standalone and dedicated instance.

 Quotas often cause resource starvation and non-deterministic behavior of otherwise sound applications. Soft limits complicate the matter even further. This calls for a judicious use of leveraging multitenancy.

 Storage and data considerations for a Multitenant Provider 

The previous articles described the multitenancy in databases using schema-based consolidation and within-database virtualization by partitioning the data dictionary horizontally. These articles also described elastic pools, sharding patterns, row-level security and key management. 

Data for applications grows by leaps and bounds year after year. This article describes the storage and data considerations and requirements.

When the volume of data increases, multitenancy depends on a clear strategy to scale the data and storage resources and to apply automation to their management, 

Elastic pools share compute resources between several databases on the same server. This helps to achieve performance elasticity of each database. The sharing of provisioned resources across databases reduced their unit costs. There are built-in protections against noisy neighbor problems. The architectural approach must meet the levels of the scale expected from the system.

Resource and request quotas must be part of this decision.  If a single storage account is deployed to contain all of the tenant’s data, then exceeding a specific number of storage operations per second will reject the application’s requests and all of the tenants will be impacted. Monitoring and retry for services must be enabled.

When designing a solution that contains multitenant data services, there are different options and levels of data isolation, For example, there can be separate containers between tenants or the database and accounts can be shared between multiple tenants. When using Azure storage for blob data, separate blob containers or separate storage accounts for each tenant can be deployed.  When deploying resources for services they can be in a single shared Azure subscription, or they can be in multiple Azure subscriptions with one per tenant. Isolation must even be higher to meet security and compliance requirements or to avoid noisy neighbor problems. Keeping the architecture as simple as possible while meeting the requirements helps with growth and planning. Tenants will likely require customizations and using feature flags to independently author, test and deploy updates are necessary. Cross tenant management operations such as regular maintenance activities, monitoring and metering of tenants, reporting data from across isolated tenants enforcing a schema and how to deploy schema updates, considering high availability requirements, and migrating tenants needed to move to different types of service, deployment or perhaps another region are all operational considerations that must be planned for in the multitenancy approach.

Higher tenancy density and lower operating costs are desirable but not the only optimization parameter. Patterns can be judiciously chosen or they can be mixed and matched especially for creating service levels. It is a good practice to scale by using deployment stamps. When we work with shared infrastructure, there are several caveats to consider. If a single resource is used, it might impose scale restrictions and limits that might interfere with maximum scale and current and future limits. Measuring and monitoring for a single tenant might be difficult depending on the services used and their tiers. Choosing premium tiers can help overcome this difficulty.

Sharding pattern can scale to a large number of tenants with appealing cost effectiveness. Horizontal, vertical and functional data partitioning can be applied. Other patterns include dedicated storage containers for each tenant and geographically distributed containers patterns. 

Each storage product or its organizational unit might have features to support multitenancy.  For example, the previous article described multitenancy approach with row-level security, tenant-level encryption, resource pooling and governance, sharding and partitioning 

Authentication and authorization strategy must be planned. A valet-key pattern can provide clients with access to storage resources. Finally, consumption must be measured and costs must be assigned to tenants. 

  Databases as a Multitenant Provider 

 

The previous article described the multitenancy in databases using schema-based consolidation and within-database virtualization by partitioning the data dictionary horizontally. This article focuses on elastic pools, sharding patterns, row-level security and key management.

Elastic pools share compute resources between several databases on the same server. This helps to achieve performance elasticity of each database. The sharing of provisioned resources across databases reduced their unit costs. There are built-in protections against noisy neighbor problems.

Resource management in dense elastic pools is achieved by implementing resource governance.

Within a database there can be multiple resource pools and workload groups, with resource limits set at both pool and group levels. User workloads and internal workloads are classified into separate resource pools and workload groups. User workload on the primary and readable secondary replicas into a designated pool and partitioned workload groups for various internal workloads. Other pools and workload groups may be reserved for various internal workloads.

In addition, job objects may be used for process level resource governance and File Server Resource Manager may be reserved for storage quota management.

Resource governance is hierarchical in nature. From top to bottom, limits can be enforced at various levels using their level appropriate mechanisms starting with the operating systems, then the resource pools and the workload groups. Data I/O governance limits both the read and the write physical I/O against data files of a database. IOPS limits are set for each service level to minimize the noisy neighbor effect. This approach allows customers to use dense elastic pools to achieve adequate performance and major cost savings. The only shortcoming with this approach is that dense competition gives rise to significant resource contention which can impact internal processes. One of the following three mitigation actions can be chosen by the customers. First, the query workload can be tuned to reduce resource consumption. Second, the pool density can be reduced by moving some databases to another pool and 3. the pool can be scaled up to get more resources.

The Sharding pattern enables to scale the workloads across multiple databases. Tools provided by the databases support the management of shard maps which track the tenants assigned to each shard. They also initiate and track queries and management operations on multiple shards by using elastic jobs.

These jobs can be periodically executed against one or many databases to run queries and perform maintenance tasks. The scripts must be defined, maintained, and persisted across a group of databases

Row-level security is useful for enforcing tenant level isolation in sharded tables. Group memberships or execution contexts are used to control access to the rows in a database table. It simplifies the design and coding of security in the application and implements restrictions on data row access. This access restriction logic is based out of the database tier rather than one spanning the application tier. This system is made more reliable and robust by reducing the surface area of the security system.

End-to-end encryption of data at rest and in transit is achieved through encryption keys and separation of databases for each tenant and always enabling the always encrypted feature.

Storage and data approaches for multitenancy must consider scale, performance predictability, data isolation, complexity of implementation, complexity of management and operations, costs, patterns and anti-patterns and best practices.