The Lessons from enabling search for multitenant applications:

Multitenant search solutions provide search capabilities to any number of tenants who cannot see or share data of any other tenant. Searching often requires content virtualization across different sources where the content can be considered a shared resource. Providing content isolation with search capabilities is the topic of this article.

A search service usually builds an index of the content so that the queries for the content can return results efficiently since the index is far easier and efficient to lookup than traversing the entire content store. The search service can be considered equivalent to a database service and indeed search can be provided out of box from a database. All the multitenant considerations for database services hold for the search service as well. Each index within a search service has its own schema, which is defined by several customizable fields. Search services, indexes, fields and documents are infrastructure and there is little or no user control. A search service can enable tenant isolation via one of the following three methods.

The application developer consumes one or more search services and divide their tenants among services, indexes or both. The methods to provide tenant isolation, scale, cost efficiency, ease of operations and reduced footprint have the following patterns:

1.       One index per tenant: This pattern has an index dedicated to a tenant within a search service that is shared with other tenants.

2.       One service per tenant: This pattern has its own dedicated service offering the highest levels of data and workload separation.

3.       A mix of both: Larger more active tenants are assigned dedicated services and the smaller ones are assigned individual indexes within the shared services.

In an index-per-tenant model, tenants achieve data isolation because all search requests and document operations are issued at an index level. The application layer directs tenant traffic to proper indexes while managing the resources at the service levels across all tenants. This model is simple for the application developers and allows them to oversubscribe the capacity among the application’s tenants. The tradeoff is the restrictions to handle peak load where most or all of the tenants are highly active. The index per tenant model provides the basis for a variable cost model. It may not be the most efficient for an application with a global footprint.

In the service-per-tenant model, each tenant has its own search service. In this model, each application achieves a maximum level of isolation for its tenants. Each service has dedicated storage and throughput for handling search request as well as separate API keys. This pattern is effective when each tenant has a large footprint, or the workload has little variability from tenant to tenant since resources are not shared.

With the mix for both index-per-tenant and service-per-tenant strategies, an application’s largest tenants can occupy dedicated services while the numerous less active, smaller tenants can occupy indexes in a shared service. It ensures that the largest tenants can derive high performance from the service, while protecting the smaller tenants from noisy neighbors. There is some planning required to classify tenants into those that required dedicated and those that can leverage a shared service.

The above patterns assume a uniform scope where each tenant is a whole instance of an application but they can handle many smaller scopes. If the service per tenant and index per tenant are not sufficiently small scopes, then the index can be utilized for a finer degree of granularity. A field can be added to an index to make it behave differently to different client endpoints. This helps to achieve separate user accounts, separate permission levels, and even completely separate applications.

Reference: https://1drv.ms/w/s!Ashlm-Nw-wnWhLMfc6pdJbQZ6XiPWA?e=fBoKcN

 

 The Lessons from the overuse of multitenancy – quotas and reservations:

The computing power and storage matters to the users more than how it is provisioned and where. The ease of requesting the compute, the cleanliness of instantiating and deleting the associated resource, the ability to migrate applications and services far outweigh any consideration of the vendor or the mechanism just so long as the performance, security and costs are comparable.

One of the classic indicators of overengineering is when there are many dials and knobs. There are two specific problems with these. First, it is an enormous burden for the users to figure out when and how to tune them optimally. Second, it increases the chance of oversight, improper use, and mistakes.

Consider the case brought forward by Alphonse Chapanis, a US Army Air Force aero medical lab psychologist who interviewed many pilots of the B-16 ‘Flying fortress’ bomber aircraft and were distressed to have landed a perfectly good plane after their mission only to see it suffer damage on stopping slowly. He determined that the pilots could not distinguish between the switches between the flaps and the landing gear that were co-located and retracted the landing gear when they meant to retract the flaps causing a nosedive and damage to the wings, propeller, and the underside of the aircraft. This incident review resulted in the switches having the landing wheel and wing flap design introduced to tell them apart and even to place them on opposite sides of the controller box. The savings from the prevented aircraft loss thereafter are a testament to the proper placement and use of controls.

Containers are governed by the quotas and limits of the underlying system to manage the shared resources. Solaris introduced partitions for different flavors, quotas and limits for file-systems and soon applications would start failing in one partition because they didn't have enough resources on this as they earlier had on the undivided host. One of the reasons Solaris zones suffered from application migration considerations from the operating system consumers is that they kept running into application failures due to limits. Both Linux containers and Solaris zones must be understood as operating system features and not as standalone products for virtualization technologies.

Linux containers have the advantage that they provide the same environment without the overhead of a separate kernel and hardware. This translates to lightning-fast availability of an isolated shell as well as their reduction of failure points. The hardware is not necessarily partitioned per virtual machine and the separation of kernel does play significantly in terms of performance because the emulation is different from the real especially when compared to a standalone and dedicated instance.

 Quotas often cause resource starvation and non-deterministic behavior of otherwise sound applications. Soft limits complicate the matter even further. This calls for a judicious use of leveraging multitenancy.

 Storage and data considerations for a Multitenant Provider 

The previous articles described the multitenancy in databases using schema-based consolidation and within-database virtualization by partitioning the data dictionary horizontally. These articles also described elastic pools, sharding patterns, row-level security and key management. 

Data for applications grows by leaps and bounds year after year. This article describes the storage and data considerations and requirements.

When the volume of data increases, multitenancy depends on a clear strategy to scale the data and storage resources and to apply automation to their management, 

Elastic pools share compute resources between several databases on the same server. This helps to achieve performance elasticity of each database. The sharing of provisioned resources across databases reduced their unit costs. There are built-in protections against noisy neighbor problems. The architectural approach must meet the levels of the scale expected from the system.

Resource and request quotas must be part of this decision.  If a single storage account is deployed to contain all of the tenant’s data, then exceeding a specific number of storage operations per second will reject the application’s requests and all of the tenants will be impacted. Monitoring and retry for services must be enabled.

When designing a solution that contains multitenant data services, there are different options and levels of data isolation, For example, there can be separate containers between tenants or the database and accounts can be shared between multiple tenants. When using Azure storage for blob data, separate blob containers or separate storage accounts for each tenant can be deployed.  When deploying resources for services they can be in a single shared Azure subscription, or they can be in multiple Azure subscriptions with one per tenant. Isolation must even be higher to meet security and compliance requirements or to avoid noisy neighbor problems. Keeping the architecture as simple as possible while meeting the requirements helps with growth and planning. Tenants will likely require customizations and using feature flags to independently author, test and deploy updates are necessary. Cross tenant management operations such as regular maintenance activities, monitoring and metering of tenants, reporting data from across isolated tenants enforcing a schema and how to deploy schema updates, considering high availability requirements, and migrating tenants needed to move to different types of service, deployment or perhaps another region are all operational considerations that must be planned for in the multitenancy approach.

Higher tenancy density and lower operating costs are desirable but not the only optimization parameter. Patterns can be judiciously chosen or they can be mixed and matched especially for creating service levels. It is a good practice to scale by using deployment stamps. When we work with shared infrastructure, there are several caveats to consider. If a single resource is used, it might impose scale restrictions and limits that might interfere with maximum scale and current and future limits. Measuring and monitoring for a single tenant might be difficult depending on the services used and their tiers. Choosing premium tiers can help overcome this difficulty.

Sharding pattern can scale to a large number of tenants with appealing cost effectiveness. Horizontal, vertical and functional data partitioning can be applied. Other patterns include dedicated storage containers for each tenant and geographically distributed containers patterns. 

Each storage product or its organizational unit might have features to support multitenancy.  For example, the previous article described multitenancy approach with row-level security, tenant-level encryption, resource pooling and governance, sharding and partitioning 

Authentication and authorization strategy must be planned. A valet-key pattern can provide clients with access to storage resources. Finally, consumption must be measured and costs must be assigned to tenants. 

  Databases as a Multitenant Provider 

 

The previous article described the multitenancy in databases using schema-based consolidation and within-database virtualization by partitioning the data dictionary horizontally. This article focuses on elastic pools, sharding patterns, row-level security and key management.

Elastic pools share compute resources between several databases on the same server. This helps to achieve performance elasticity of each database. The sharing of provisioned resources across databases reduced their unit costs. There are built-in protections against noisy neighbor problems.

Resource management in dense elastic pools is achieved by implementing resource governance.

Within a database there can be multiple resource pools and workload groups, with resource limits set at both pool and group levels. User workloads and internal workloads are classified into separate resource pools and workload groups. User workload on the primary and readable secondary replicas into a designated pool and partitioned workload groups for various internal workloads. Other pools and workload groups may be reserved for various internal workloads.

In addition, job objects may be used for process level resource governance and File Server Resource Manager may be reserved for storage quota management.

Resource governance is hierarchical in nature. From top to bottom, limits can be enforced at various levels using their level appropriate mechanisms starting with the operating systems, then the resource pools and the workload groups. Data I/O governance limits both the read and the write physical I/O against data files of a database. IOPS limits are set for each service level to minimize the noisy neighbor effect. This approach allows customers to use dense elastic pools to achieve adequate performance and major cost savings. The only shortcoming with this approach is that dense competition gives rise to significant resource contention which can impact internal processes. One of the following three mitigation actions can be chosen by the customers. First, the query workload can be tuned to reduce resource consumption. Second, the pool density can be reduced by moving some databases to another pool and 3. the pool can be scaled up to get more resources.

The Sharding pattern enables to scale the workloads across multiple databases. Tools provided by the databases support the management of shard maps which track the tenants assigned to each shard. They also initiate and track queries and management operations on multiple shards by using elastic jobs.

These jobs can be periodically executed against one or many databases to run queries and perform maintenance tasks. The scripts must be defined, maintained, and persisted across a group of databases

Row-level security is useful for enforcing tenant level isolation in sharded tables. Group memberships or execution contexts are used to control access to the rows in a database table. It simplifies the design and coding of security in the application and implements restrictions on data row access. This access restriction logic is based out of the database tier rather than one spanning the application tier. This system is made more reliable and robust by reducing the surface area of the security system.

End-to-end encryption of data at rest and in transit is achieved through encryption keys and separation of databases for each tenant and always enabling the always encrypted feature.

Storage and data approaches for multitenancy must consider scale, performance predictability, data isolation, complexity of implementation, complexity of management and operations, costs, patterns and anti-patterns and best practices.

 

 

 

Databases as a Multitenant Provider

Persistence of data mandates a database for the guarantees of Atomicity, Consistency, Isolation and Durability as well as for querying. Tenants can be served their own databases in which case the database becomes a multi-tenant data platform and the technology stack for handling the data might vary from tenant to tenant.  Groups of database can be put into a container to manage them as a single unit. The benefits of a multitenant database include the following:

1.       High consolidation density where containers of database makes it easier to manage databases than the conventional single database per tenant

2.       Rapid provisioning and cloning using SQL where the database can be created, populated and cloned using SQL scripts.

3.       Rapid patching and upgrades where the patching of individual databases inside a container can be accomplished by plugging out and plugging in the database. The patching of the container is also fast.

4.       Converting and consolidating existing databases into containers so that they can be managed as a unit.

5.       Resource management between pluggable databases in a container

Multitenancy brings standardization which reduces the operating expense. Cost often increases with variations in size and traffic on databases. Customers benefit when the databases can scale out and remain elastic. Both operational expense and capital expense reduces.

It is also possible to host distinct application backends into the same database and this approach is called schema-based consolidation. This reduces the number of databases and increases the consolidation density. There are a few caveats to this approach which include 1. Name collision hampers schema-based consolidation, 2. It brings weak security 3. Per application backend point-in-time recovery is prohibitively difficult. 4. Resource management between application backends is difficult, 5. Patching for a single application backend is disallowed and 6. Cloning a single application backend is not permitted.

The difference between dedicated database architecture and this multitenant architecture is that the latter brings true within-database virtualization. This is implemented by partitioning the data dictionary horizontally. Within database virtualization removes the drawbacks of schema-based consolidation. The physical separation between the container and the databases brings pluggability. This unplug/plug action set brings a new paradigm for patching. The sharing model and the background processes is the same for conventional, pluggable databases and application backends. The container or root is the new tenet of the multitenant architecture. Just as operating systems orchestrate provisioning tasks, SQL statements executed against the root implement them for pluggable databases.

In this case, multitenancy becomes a deployment choice with neither the application backend nor the client code required to change.

Reference: Multitenancy: https://1drv.ms/w/s!Ashlm-Nw-wnWhLMfc6pdJbQZ6XiPWA?e=fBoKcN     

 

 Databases as a Multitenant Provider 

 

The previous article described the multitenancy in databases using schema-based consolidation and within-database virtualization by partitioning the data dictionary horizontally. This article focuses on elastic pools, sharding patterns, row-level security and key management.

Elastic pools share compute resources between several databases on the same server. This helps to achieve performance elasticity of each database. The sharing of provisioned resources across databases reduced their unit costs. There are built-in protections against noisy neighbor problems.

Resource management in dense elastic pools is achieved by implementing resource governance.

Within a database there can be multiple resource pools and workload groups, with resource limits set at both pool and group levels. User workloads and internal workloads are classified into separate resource pools and workload groups. User workload on the primary and readable secondary replicas into a designated pool and partitioned workload groups for various internal workloads. Other pools and workload groups may be reserved for various internal workloads.

In addition, job objects may be used for process level resource governance and File Server Resource Manager may be reserved for storage quota management.

Resource governance is hierarchical in nature. From top to bottom, limits can be enforced at various levels using their level appropriate mechanisms starting with the operating systems, then the resource pools and the workload groups. Data I/O governance limits both the read and the write physical I/O against data files of a database. IOPS limits are set for each service level to minimize the noisy neighbor effect. This approach allows customers to use dense elastic pools to achieve adequate performance and major cost savings. The only shortcoming with this approach is that dense competition gives rise to significant resource contention which can impact internal processes. One of the following three mitigation actions can be chosen by the customers. First, the query workload can be tuned to reduce resource consumption. Second, the pool density can be reduced by moving some databases to another pool and 3. the pool can be scaled up to get more resources.

The Sharding pattern enables to scale the workloads across multiple databases. Tools provided by the databases support the management of shard maps which track the tenants assigned to each shard. They also initiate and track queries and management operations on multiple shards by using elastic jobs.

These jobs can be periodically executed against one or many databases to run queries and perform maintenance tasks. The scripts must be defined, maintained, and persisted across a group of databases

Row-level security is useful for enforcing tenant level isolation in sharded tables. Group memberships or execution contexts are used to control access to the rows in a database table. It simplifies the design and coding of security in the application and implements restrictions on data row access. This access restriction logic is based out of the database tier rather than one spanning the application tier. This system is made more reliable and robust by reducing the surface area of the security system.

End-to-end encryption of data at rest and in transit is achieved through encryption keys and separation of databases for each tenant and always enabling the always encrypted feature.

Storage and data approaches for multitenancy must consider scale, performance predictability, data isolation, complexity of implementation, complexity of management and operations, costs, patterns and anti-patterns and best practices.

 

 

 Sample program to add claim to token in delegated auth use case:

using System.IO;

using IdentityClaim = Microsoft.IdentityModel.Claims.Claim;

using IdentityClaimTypes = Microsoft.IdentityModel.Claims.ClaimTypes;

using IdentityClaimsPrincipal = Microsoft.IdentityModel.Claims.ClaimsPrincipal;

using ClaimsIdentityCollection = Microsoft.IdentityModel.Claims.ClaimsIdentityCollection;

            IClaimsIdentity claimsIdentity = new ClaimsIdentity(Thread.CurrentPrincipal.Identity);

            var claimValue = string.Format("claim://{0}@{1}", TargetResourceRole.PrivilegedDeploymentOperator, "sample-resource-folder-test");

            var identityClaim = new IdentityClaim(IdentityClaimTypes.Role, claimValue);

            claimsIdentity.Claims.Add(identityClaim);

            ClaimsIdentityCollection claimsIdentityCollection = new ClaimsIdentityCollection(new List<IClaimsIdentity>() { claimsIdentity });

            var newIcp = IdentityClaimsPrincipal.CreateFromIdentities(claimsIdentityCollection);

            Thread.CurrentPrincipal = newIcp;

The  above example uses the Microsoft.IdentityModel namespace to describe the elevation of privilege to run some code.

Now for the delegated auth use case:

           string homeSecurityTokenService = ConfigurationManager.GetSetting("HomeSecurityTokenService");

           string SecurityTokenServiceRealm = ConfigurationManager.GetSetting("SecurityTokenServiceRealm");

           string serviceName = ConfigurationManager.GetSetting("ServiceName");

           var serverHomeSecurityTokenService = new ServerHomeSecurityTokenService(

                    new Uri(SecurityTokenServiceRealm),

                    homeSecurityTokenService,

                    null);

           var serviceIdentity = new ServiceIdentity(

                serviceDnsHostName: targetDnsName,

                serviceNames: new string[] { serviceName });

           WebSecurityTokenAuthenticator authenticator = new WebSecurityTokenAuthenticator(serverHomeSecurityTokenService, serviceIdentity);

           ClaimsIdentityCollection collection = authenticator.Authenticate(authorizationHeader, resourceName);

           var claimValue = string.Format("claim://{0}@{1}", TargetResourceRole.PrivilegedDeploymentOperator, payload.Properties.Folder);

           collection.Add(new ClaimsIdentity(new List<Claim>() { new Claim(ClaimTypes.Role, claimValue) }));

var authContext = new Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext(

                        tokenIssuanceUrl, true);

StringBuilder sb = new StringBuilder();

collection.ForEach(x => x.Claims.ForEach(c => sb.Append(c.Value + ",")));

var claims = sb.ToString().Trim(',');

var authenticationResult = 

authContext.AcquireTokenAsync(resourceName, clientCredential.ClientId, new Uri("https://DstsInternalNativeClient"), new PlatformParameters(PromptBehavior.Auto), userIdentifier, extraQueryParameters, claims, synchronizationContext);

var newDelegatedToken = authResult.AccessToken;