The previous article talked about workflows and multitenant systems. This article talks about innovations in cloud-based multitenant systems or solutions for short which include those for industry specific solutions, authoritative marketplace offerings, fleet deployment of agents, and solution-based automation.

In this section, we focus on transitions specifically. When control passes from the tenant workload to the multitenant infrastructure and back, there is an opportunity to add routines that can not only safeguard the state of the caller but also improve the statistics and bookkeeping withing the solution. It is even possible to introduce a tag or inject color into the data stream so that its propagation throughout the cloud can be made visible. This improves forensics as well as the detection of resources that are underutilized for advice towards application optimization.  

Similarly adding headers before and after data segments from specific callers enables the study of those data manipulations by all parties during its lifetime. This study could include stack captures that are authoritative and comprehensive. 

A lot of information can be obtained when specific bookkeeping is added to sequences or patterns of usages or by specific callers. Since this additional bookkeeping might introduce regressions in performance optimizations for the application, it becomes important to turn it on for as granular a session as possible and for the duration specified by the administrator. In this regard, feature flags, variables, and dynamic behavior from the code will be helpful in the isolation of the control and data path under investigation.  

Finally, system performance and behavior capture has traditionally been curated for manual inspection. With the advent of AI and the popularity of data mining techniques, this machine data could automate analysis that draws insights and makes recommendations to the application authors. This strategy could involve cross-application comparisons and associations, historical trends, and collaborative filtering. Some common scenarios are described here.

Conventional software engineering practice involves the use of profiling as a means of studying avenues for application optimizations. A multitenant solution hosted in the public cloud is uniquely positioned toward this goal. Not only does the public cloud have complete visibility into the utilization of cloud resources and profiling, but it can also draw insights into application performance with the help of models that weren’t possible earlier on-premises. The cloud-based solution can introduce a significant number of resources for short periods of time on bursts of analysis activities, so they remain unparalleled in elasticity elsewhere. By incorporating the control and feedback loop directly within the cloud-based solution on behalf of the tenant and their applications, the solution can offer better impact in terms of shaping the tenant resources and the workloads they host, for the future.  Finally, public cloud provides an excellent paradigm for multitenant solutions to imbibe as an infrastructure provider for their tenants

 Workflows:

This article focuses on some of the best practices for working with workflows that deploy services.  The tenets are:

1.       Reusability – many of the activity from the library of activities for one workflow can and will be reused for another. Very few workflows might have differences in doing tasks that were not covered by the global collection of activities. There should not be any difference between an activity that appears in bootstrapping and its invocation during redeployment/ rehosting in the new environment. Only the parameter values will change for this.

2.       Dependencies – many of the dependencies will be implicit as they originate from system components and services information. A workflow might additionally specify dependencies via the standard way in which workflows indicate dependencies. These will be on a case-by-case basis for tenants since it adds overhead to other services, many of whom are standalone. Implicit dependencies can be articulated in the format specified by the involved components.

3.       Splitting – Workflows are written for on-demand invocation from the web interface or by the system, so there might be more than one for a specific deployment scenario. It is best to include both the bootstrapping and the redeploy in the main workflow for the specific scenario, but they will be mutually exclusive during their respective phases and remain idempotent.

4.       Idempotency – All workflow steps and activities should be idempotent. If there are conditionals involved, they must be part of activities. The signaling and receiving notifications of dependent workflows if any must be specifically called out.

5.       Bootstrapping – This phase is common to many services and usually requires at least a cluster/set of servers to be made ready but there might be activities that require the service stamp to be deployed even if it is not configured along with necessary activities to do one time preparation such as getting secrets. Until the VIPs are ready, the redeployment cannot be kicked off.  Bootstrapping might involve preparations for both primary and secondary where applicable.

6.       Redeployment or rehosting – This phase involves configuration since the bootstrapping is usually for a stamp and this stage converts it into a deployment for a service. Since it involves reconfiguration, it can be for both primary and secondary and typically done inside the new cloud. It is best to parameterize as much as possible.

7.       Naming convention – Though workflows can have any names inside the package that the owning teams upload, it is best to follow a convention for the specific scenario of one workflow calling another. Standalone single workflows do not have this problem. Even in the case when there are many workflows, a prefix/suffix might be helpful. This applies to both work workflows and activities.

8.       System workflow – Requiring separate workflows for bootstrap and redeployment via a system defined workflow to allow system to inject system defined activities between bootstrap and redeploy is a nice-to-have but the less intrusion into service deployment the better. This calls on the service to do their own tracking via passing parameter values between workflows and activities. A standard need not be specified for this, and it can be left to the discretion of the services.

The above list is not intended to be complete but focuses on the strengths of those that have worked well

 

Considerations during migration of Active Directory Domain services objects across forests via tool or cloud service for Active Directory Migration 

Upgrade and migration are two distinct but popular and mainstream operations for working with Active Directory objects and Organizational Units. Migration is usually to a new forest from say domain A to domain B. There are many considerations to migration. 

1.       Migration is about users not machines. It must be focused on users and groups. Machines come along with the users. 

2.       It is not easy to move all the things pertaining to a user all at once. This must be done one after another.  

3.       When organizations want to migrate, they envision something but what turns out at the end is usually different because when the migration is long, the landscape of objects changes from such things as acquisitions and mergers.  

4.       IT is wide and complex spread out across time-zone and geography so migration must scale to many objects 

5.       Migration pattern always involves a scale up and scale down as needed.  The drawing board for migration planning might project a best-case scenario as one where the pilot takes a constant time to build, and the business has no ups or downs but this is never uniform. There may be early adopters, if at all the pilot goes as planned, then there might be blackouts, followed by intense sprints and then the dwindling in terms of stragglers. These varying migration patterns can all be accommodated by self-service which is critical to enable many customers. 

6.       It is best to require users to self-serve themselves because their situation might vary from one to another.  

7.       There must not be any damages caused due to migration. Before and after the migration they must remain working and especially after migration, there must not be any more fixes to be made. 

8.       Prerequisite checking is important towards this purpose. Flights and dry runs are part of the migration process. The migration process is successful only when it has been carefully planned out. 

Expectations must be softened for Active Directory migrated objects. The security identifier for a user, for example, is not going to be the same. That is generated by a domain controller and a new domain has its own controller. The process of issuing a security identifier involves a Relative Identifier Master or RID master which has a Flexible Single Master Operations role. This is a domain scoped role. Each RID master role can allocate active and stand-by Relative identifiers.  The RID master can move multiple objects across domains within the same forest. Since it is a single master, when it is down, no new objects can be created. This might seem severe but in established organizations, creation of new objects is low so the brief downtime can be tolerated. If the role is seized and the original domain controller is brought back online, duplicate RIDs may be introduced. 

The following PowerShell commands can help: 

Get-ADForest <domain> | Format-Table SchemaMaster, DomainNamingMaster, PDCEmulator, RIDMaster, InfrastructureMaster 

If the source and target DC are not the RID master, the error is  
move-adobject : The requested operation could not be performed because the directory service is not the master for that type of operation

 

Support and sustained engineering:  

The previous articles talked about Licensing and purchasing models. This article talks about support and sustained engineering.

Sustaining engineering for multitenant applications is about maintenance after release hence ‘sustaining’ in the way tenants and customers use these and provide feedback with their usage. The difference between sustaining engineering for a software product versus a multitenant application is one of on-premises deployment versus software-as-a-service. Nevertheless, the company that makes the software product is focused on innovations and improved valued additions that come in waves of software releases to tenants. While many are seamlessly upgraded to the newer versions since they have no ownership of the infrastructure, solution providers cannot always guarantee backward compatibility and the resources they provision might themselves not be supported in the same way as they were. These and other external dependencies are often addressed with releases which are versioned to indicate which is older and which is newer. The older versions are upgraded to the newer on existing systems or the newer versions are installed on newer systems. As the software maker focuses on the next release, sustaining engineers focus on maintaining the existing released versions for specific tenant usages.   

The typical rule of thumb for how many versions to maintain is usually determined based on the usages by customers. Some companies maintain all their application versions as far back as the earliest if there are customers who have purchased them and want to actively use them. Others choose to discontinue the maintenance on select older versions so long as both the customers and the company have an agreed exit strategy. Usually, the customers may be eased into newer versions. There are several factors that play out into the extent of maintenance engaged by a software maker such as revenue, customer base, market segment, costs, resources, media, etc. and it is not uncommon to find two or three versions being maintained.  Sustaining is all about this art and science of maintaining released software versions and often engages with customers throughout their usages. It is interesting to note that customers can run into issues of their own accord with any of these versions and not just when the software maker has put out a release the customer wants to use. That is why software sales and sustaining are both ongoing commitments for a software maker while being fundamentally different. 

Nowhere in the industry has there been a better service-level agreement articulation as the warranty and support that comes with the application both for the multitenant application and those applications that are sold via the application store. This is not just legal language. It is one where the software maker is offering a tiered approach to what the customer has paid for and is required to pay for.  

On one end of the spectrum, early multitenant applications have long held on to a difficult bargain for the customer where they were required to pay for the updates and upgrades to their purchases so that their operations could continue without outages. Large commercial multitenant applications even had a wake-up call from the industry to say that this simply cannot go on and there must be resources pitched in to improve the efficiency and experience around the engagement. 

On the other end of the spectrum, cloud service and outsourced business processes have largely muted the discussions on software maintenance with most error data gathering activities and corrections happening independent of the businesses concern. Even the billing has changed to being all-inclusive in the pay as you go approach with the clouds taking over the total cost of ownership and leaving merely the application optimizations to the businesses. 

Somewhere in between, the industry is required to balance and invest in such agreements across and the applications that they use. The maintenance plan and support are drawn out by the multitenant solution providers to best suit their customers and internal schedule. 

 

 

 Licensing:

The previous articles talked about Licensing with a multitenant application. This article continues to discuss a few more aspects.

The lifecycle of group-based licenses can be managed in Azure Active Directory. This is called entitlement management. Using groups to manage licenses for applications helps to configure periodic access reviews and allows other employees to request membership in the group.

For example, an access package can be created to allow employees to gain access to Office licenses such that group members can be reviewed annually, and new employees can request licenses with their manager's approval.

Azure AD entitlement management itself requires Azure AD Premium P2 license and Enterprise Mobility plus Security EMS ES approval.

The steps to create the access package involve the following steps: 1) the basics for the access package such as name, description, and catalog type must be specified. 2) the resources for the access package must be specified as groups and teams with roles as members. and 3) the requests for the access package must be configured to include approvals and their manner. 4) The requestor information must be collected, 5) the lifecycle for the access package must be configured and 6) finally, the access package must be created and reviewed.

Users with individual licensing can be migrated to use groups. There is a caveat here that a situation where users temporarily lose their currently assigned licenses during migration must be avoided. Any process that may result in the removal of licenses should similarly be avoided. The recommended migration process involves 1) using existing automation to manage license assignment and removal for users. 2) creation of a new licensing group to make sure all the required users are added as members. 3) the required licenses should be assigned to those groups 4) the licenses should be applied to all users in those groups and 5) a check must be performed that no license assignments failed. License assignment errors can be found by finding users in an error state in a group.

Common errors encountered with Licensing involve the following:

1) a situation where there are not enough licenses – this can be mitigated by purchasing more licenses for the product or freeing up unused licenses from other users or groups. Available licenses can be viewed.

2) a situation where there are conflicting service plans. Some service plans are configured in a way that they can’t be assigned to the same user as another related service plan This can be resolved by disabling one of the plans. 

3) a situation where other products depend on this license. A product might have a service plan that requires another service plan in another product to function. This can be mitigated by making sure that the required plan is still assigned to users through some other method or that dependent services are disabled for those users.

4) a situation where the usage location is not allowed. Before a license can be assigned to a user, the usage location property must be specified for the user. When this is violated, an error occurs. This can be resolved by removing users from unsupported locations from the license group.

5) a situation where the proxy addresses are duplicated. when users in the organization specify the same proxy address twice and the group-based licensing tries to assign a license to such a user, it fails. This error must be solved on the user side and the license processing must be forced on the group after the remediation.

6) a situation where the Azure AD mail and Proxy Addresses attribute changes. Some proxy address calculations can trigger attribute changes. These must be investigated on a case-by-case basis.

7) a situation where a concurrency exception occurs in the audit logs. This comes from a concurrent license assignment of the same license to a user. Retrying the process will resolve this issue and there will not be any action required from the customer to fix this issue.

8) a situation where more than one product license must be assigned to a group. We can see users who failed to get assigned and check which products are affected by this symptom.

9) a situation where a licensed group is deleted. All licenses assigned to the group must be deleted before the group can be deleted.

10) a situation where licenses for products with prerequisites must be managed – some products are add-ons and they require a pre-requisite service plan to be enabled for a user or group before they can be assigned a license. The add-on license can be assigned to a group, where the group also contains the prerequisite service plan

11) a situation where group licensing processing can be forced to resolve errors especially for freeing up some licenses

12) a situation where the user licensing processing can be forced to resolve errors such as the duplicate proxy error described above.

When the number of servers or the number of users is large, volume licensing options might be available. This is the practice of selling a license authorizing one piece of software to be used on a large number of computers or by a large number of users. Software training for volume licensing customers might be made available by way of training and certification solutions. A customized software purchase program that grants discounted access to training and certification solutions.
 

 

 Licensing: 

The previous articles talked about Branding with a multitenant application. This article discusses Licensing next. 

Licensing depends on the application, users and tenants. Some applications enforce licensing to be based on users or logins but not servers. Licensing also affects purchasing model.   

There are several ways to license a multitenant store. For example, managed database instances can come with licensing on per core basis or per user basis or a combination of server plus client access licenses (CAL). The editions of the product for these licenses can vary from Enterprise, Standard-per core, Standard- server + CAL, Developer, Web, and Express.  
 
There are two purchasing models:  
1) Virtual v-core based purchasing model - this model provides a choice between a provisioned compute tier and a serverless compute tier. With the provisioned compute tier, the amount of computer resources is always provisioned for the workload. With the serverless compute tier, the autoscaling of the compute resources is specified instead. With autoscaling, the databases are paused and resumed with charges only for storage during period of inactivity. 

2) the database transaction unit-based purchasing model – this model provides bundled compute and storage packages balanced for common workloads. The compute sizes are declared in DTUs for single databases and elastic DTUs for elastic pools. The vCore based model allows us to independently choose compute and storage resources. Customers prefer DTU-based for simple, preconfigured resource options and vCore-based for value flexibility, control, and transparency. 

Storage costs are calculated differently based on each purchasing model. Storage is included in the price of the DTU. It is possible to add extra storage in the standard and premium tiers of a cloud SQL database. 

When the number of servers or the number of users is large, volume licensing options might be available. This is the practice of selling a license authorizing one piece of software to be used on a large number of computers or by a large number of users. Software training for volume licensing customers might be made available by way of training and certification solutions. A customized software purchase program that grants the discounted access to training and certification solutions. 

 

The previous articles on Multitenant Applications talked about organization of data particularly keys and values. This article talks about branding.

Branding is perhaps most noticeable when users sign-in to the multitenant solution.  When the sign-in process can include the company logo and customized experiences based on browser language, it creates a powerful branding for the organization over the multitenant solution that is re-used across the organizations.

The technique behind such flexibility in customization involves an inventory of all user interface properties, such that their customizations can be looked up and applied to compose a unique and holistic branded page to the users. This inventory consists of the text, its culture and locale, references to the images to be used, stylesheets and other assets and where to locate them on the rendered page. Even dark mode or light mode can be set as default and not rely on users’ preferences. The inventory makes it easy to add customizations for new tenants and organize their variations over the defaults.

The tenant provider composes the page by referring to the tenant ID obtained from the domain name and request headers. Tenants might want to bring their own domain names. This might be important for business purposes. It might also be for technical purposes such as they supply their own TLS certificates which bear subject names.

The name resolution to an IP address depends on whether there is a single instance or many instances of the multitenant application. For example, a CNAME for the custom domain of a tenant might have a value pointing to a multi-part subdomain of the multitenant application solution provider. Since this provider might want to set up proper routing to multiple instances, they might have a CNAME record for subdomains of their individual instance to route to that instance. They will also have an A name record for that specific instance to point to the IP address of the provider’s domain name. This chain of records resolves the requests for the custom domain to the IP address of the instance within the multiple instances deployed by the provider.

Host header resolution is also significant. All the web components need to be aware of how to handle the requests that arrive with the provider’s domain name in their host request header. Each tenant’s domain name might be a subdomain or a custom domain and this adds operational overhead to the onboarding of tenants. Host headers can also be rewritten by say the Azure FrontDoor so that the web server receives a single Host header. The example of Azure FrontDoor also propagates the original value of the host header in a X-Forwarded-Host header so the multitenant application can properly resolve the tenant.

Custom branding appears after the users’ sign-in. Some of the prerequisites for custom branding includes licenses for Azure Active Directory Premium 1 and 2, Office 365 for office applications, and Microsoft 365 for keeping the user signed in. When users sign into the organization’s tenant-specific applications, such as https://outlook.com/woodgrove.com, or when passing a domain variable such as https://passwordreset.microsoftonline.com/?whr=woodgrove.com All branding elements are optional. Default settings will remain, if left unchanged. Images have different image and file-size requirements. Language, background image, banner logo, username hint and sign-in page text can all be part of the branding strategy.

If the branding must appear before sign-in, technologies like pagelets can be used so that the server has the chance to discern and render the page without relying on out-of-box default pages.

By retrieving the proper assets from the inventory of customizations, a multitenant solution can compose the user interface page with the proper branding experience for users.