Friday, September 30, 2022

 

Recovery and Replication of data in Multitenant Applications:

 

This is a continuation of the detailed article on recovery options. In next section, we discuss replication options.

 

The example used here is a full disaster recovery scenario for a multitenant SaaS application implemented with the database per tenant model. The concepts introduced here are geo-restore and geo-replication.

 

GeoReplication helps to create a continuously synchronized readable secondary database for a primary database. It is preferable although not required to have the secondary database in a different region. Since this kind of secondary database is merely readable, it is called a georeplica. This option serves to perform quick recovery of individual databases in case of a regional disaster or a large-scale outage. Once georeplication is setup, a geo-failover helps maintain continuity of business.

 

There can be at most four geo-secondaries for a single primary. Multiple geo-secondaries are tactical redundancy. Additional secondaries can also be used to scale out read-only workloads. If there’s only one secondary and it fails, the application is exposed to higher risk until a new secondary is created.

Each geo-secondary can be a single database or a database in an elastic pool. The elastic pool choice for each geo-secondary database is separate and does not depend on the configuration of any other replica. Each elastic pool is contained within a single logical server. Database names must be unique in a pool so multiple geo secondaries cannot share the same pool.

 

A geo-secondary that has finished initial seeding can be failed over on demand by the user. If the primary is unavailable, only the unplanned geo-failover can be used. The geo-secondary becomes the new primary. Once the outage is mitigated, the system makes the recovered primary a geo-secondary. Linking of all secondaries to the new primary is automatically done and replication relationships are reconfigured. After the outage that caused the geo-failover is mitigated, it may be desirable to return the primary to its original region.

Preparing for a geo-failover involves validating that the authentication and network access for the secondary server are properly configured. The backup retention policy on the secondary database matches that of the primary. This setting is not part of the database, and it is not replicated from the primary. The default configuration of a geo-secondary has a default PITR retention period of 7 days.

 


Thursday, September 29, 2022

 

This article discusses using the checklist for architecting and building multitenant solutions. Administrators will find that this list is familiar to them. 

 

The checklist is structured around business and technical considerations as well as the five pillars of the Azure well-architected framework.  These pillars include 1) Reliability, 2) Security, 3) Cost Optimization, 4) Operational Excellence, and 5) Performance efficiency.  The elements that support these pillars are Azure well-architected review, azure advisor, documentation, patterns-support-and-service offers, reference architectures and design principles. Out of these, cost optimization is one of the primary benefits of using the right tool for the right solution. It helps to analyze the spend over time as well as the effects of scale out and scale up. The Azure Advisor can help improve reusability, on-demand scaling, reduced data duplication, among many others. Performance is usually based on external factors and is very close to customer satisfaction. Continuous telemetry and reactiveness are essential to tuned up performance. The shared environment controls for management and monitoring create alerts, dashboards, and notifications specific to the performance of the workload. Performance considerations include storage and compute abstractions, dynamic scaling, partitioning, storage pruning, enhanced drivers, and multilayer cache.

Operational excellence comes with security and reliability. Security and data management must be built right into the system at layers for every application and workload. The data management and analytics scenario focus on establishing a foundation for security. Although workload specific solutions might be required, the foundation for security is built with the Azure landing zones and managed independently from the workload. Confidentiality and integrity of data including privilege management, data privacy and appropriate controls must be ensured. Network isolation and end-to-end encryption must be implemented. SSO, MFA, conditional access and managed service identities are involved to secure authentication. Separation of concerns between azure control plane and data plane as well as RBAC access control must be used.

The checklist for business considerations include 1. understanding what kind of solution is being created such as business-to-business, business-to-consumer, or enterprise software 2. Defining the tenants in terms of number and growth plans, 3. Defining the pricing model and ensuring it aligns with the tenants’ consumption of Azure resources. 4. Understanding whether we need to separate the tenants into different tiers and based on the customer’s requirements, deciding on the tenancy model. Finally, promoting the multitenant solution in the commercial marketplace.

The technical considerations emphasize design and service-level objectives, as well as the scale of the solution. It also suggests applying Chaos engineering to test the reliability of the solution. The security considerations involve Zero Trust and least privilege principles.


 

Wednesday, September 28, 2022

 

Recovery and Replication of data in Multitenant Applications:

This is a continuation of the detailed article on recovery options. In next section, we discuss replication options.

The example used here is a full disaster recovery scenario for a multitenant SaaS application implemented with the database per tenant model. The concepts introduced here are geo-restore and geo-replication.

GeoReplication helps to create a continuously synchronized readable secondary database for a primary database. It is preferable although not required to have the secondary database in a different region. Since this kind of secondary database is merely readable, it is called a georeplica. This option serves to perform quick recovery of individual databases in case of a regional disaster or a large-scale outage. Once georeplication is setup, a geo-failover helps maintain continuity of business.

Geo-replication can also be performed for database migration with minimum downtime and application upgrades by creating an extra secondary as a fail back copy during application upgrades. An end-to-end recovery requires recovery of all components and dependent services. All components are resilient to the same failures and become available within the recovery time objective of the application. Designing cloud solutions for disaster recovery include scenarios using two Azure regions for business continuity with minimal downtime or using regions with maximum data preservation or to replicate an application to different geographies to follow demand.

Automatic asynchronous replication requires transactions to be committed on the primary database before they are replicated. It is used for creating a geo-secondary. When the geo-secondary is created, the replica is populated with the data of the primary database. This process is known as seeding. After it has been created and seeded, updates to the primary database are automatically and asynchronously replicated to the replica.

An application can access the geo-secondary replica to execute read-only queries. The security principal used to access the geo-secondary can be same as the one used for primary or different.

A planned geo-failover switches the roles of the primary and geo-secondary databases after completing full data synchronization. A planned failover does not result in data loss. Since the replication is done based on well-known log shipping techniques, it duration depends on the size of the log at the origin. This kind of failover is applicable to performing disaster recovery drills, relocating the database to a different region, returning the database to the primary region after the outage has been mitigated.

On the other hand, unplanned geo-failover immediately switches the geo-secondary to the primary role without any synchronization with the primary. Any transactions committed on the primary but not yet replicated to the secondary are lost. Only when the primary is not available, should the geo-failover be unplanned.

There can be at most four geo-secondaries for a single primary. Multiple geo-secondaries is tactical redundancy. Additional secondaries can also be used to scale out read-only workloads.


Tuesday, September 27, 2022

 

Recovery and Replication of data in Multitenant Applications:

This is a continuation of the detailed article on Azure Monitor and Azure Monitor Logs. These are different services and Logs are incredibly useful for troubleshooting and notifications. In next section, we discuss recovery options.

The example used here is a full disaster recovery scenario for a multitenant SaaS application implemented with the database per tenant model. The concepts introduced here are geo-restore and geo-replication.

A geo-restore can be used to recover the catalog and tenant databases from automatically maintained geo-redundant backups into an alternate recovery region. After the outage is resolved, geo-replicate can be used to repatriate the changed databases to their original region.

A database can be restored to an earlier point in time within its restoration period. This works for any service tier or compute size for the restored database. If the database is restored into an elastic pool, there must be sufficient resources in the pool to accommodate the database. There is no charge incurred during the restoration and the restored database is charged at normal rates after that.

A point-in-time restore does not support cross-server restoration and it cannot restore a geo-secondary database. Hyperscale databases are not subject to a backup frequency and must be restored on demand. A restored database can be used to replace the original database by renaming it. If the database is restored only for its data, a recovery script must extract and apply that data to the original database.

A restore operation on a long-term backup can be performed form the logical server via the user interface, command line, programmability interface or scripts. It is not applicable to Hyperscale databases.

A deleted database can be restored to the deletion time, or an earlier point in time or an earlier point of time on the same server. A geo-restore can perform cross-server cross-region restore from the most recent backups. It is typically done when the database is restored or the entire region is inaccessible.

 There is usually a delay when a backup is taken and when it is geo-restored and the restored database can be upto one hour behind the original database. Geo-restore relies on automatically created geo-replicated backups with a recovery point objective of up to 1 hour and an estimated recovery time objective (RTO) of upto 12 hours. It does not guarantee that the target region will have the capacity to restore the database after a regional outage, because a sharp increase in demand is likely. Therefore, it is most used for small databases. Business continuity for larger databases is ensured via auto-failover groups. It has a much lower RPO and RTO and the capacity is guaranteed.

GeoReplication helps to create a continuously synchronized readable secondary database for a primary database. It is preferable although not required to have the secondary database in a different region. Since this kind of secondary database is merely readable, it is called a georeplica. This option serves to perform quick recovery of individual databases in case of a regional disaster or a large-scale outage. Once georeplication is setup, a geo-failover helps maintain continuity of business.

Geo-replication can also be performed for database migration with minimum downtime and application upgrades by creating an extra secondary as a fail back copy during application upgrades. An end-to-end recovery requires recovery of all components and dependent services. All components are resilient to the same failures and become available within the recovery time objective of the application. Designing cloud solutions for disaster recovery include scenarios using two Azure regions for business continuity with minimal downtime or using regions with maximum data preservation or to replicate an application to different geographies to follow demand.


Monday, September 26, 2022

 Recovery in Multitenant Applications:

 

This is a continuation of the detailed article on Azure Monitor and Azure Monitor Logs.  Azure Monitor and Azure Monitor Logs are different services and Logs are incredibly useful for troubleshooting and notifications.

This article talks about Recovery

The example used here is a full disaster recovery scenario for a multitenant SaaS application implemented with the database per tenant model. In this case, a geo-restore can be used to recover the catalog and tenant databases from automatically maintained geo-redundant backups into an alternate recovery region. After the outage is resolved, georeplicate can be used to repatriate the changed databases to their original region.

Geo-restore is about recovering any database from a backup in Azure SQL Database, including Hyperscale databases. We can restore a database from a backup in Azure SQL Managed Instance.

Backups can also be taken on a scheduled basis and this helps to protect databases from user and application errors, accidental database deletion, and prolonged outages. This built-in capability is available for all service tiers and compute sizes of Azure SQL Managed instances. The automated recovery helps in these cases: 1. Recovering to a specific point in time within the retention period, 2. Recovering to the deletion time for a deleted database. 3. Recovering to the time of a recent backup and 4. Recovering to the point of the most recent replicated backups.

If Long-term retention is configured, even the long-term retention backup can be used to restore databases.

Several factors affect the recovery time. These include: the size of the database, the compute size of the database, the number of transaction logs involved, the amount of activity that needs to be replayed to recover to the restore point, the network bandwidth if restoring to a different region, the number of concurrent restore requests that are processed in the target region.

GeoReplication helps to create a continuously synchronized readable secondary database for a primary database. It is preferable although not required to have the secondary database in a different region. Since this kind of secondary database is merely readable, it is called a georeplica. This option serves to perform quick recovery of individual databases in case of a regional disaster or a large-scale outage. Once georeplication is setup, a geo-failover helps maintain continuity of business. It can be initiated programmatically or manually. If a stable connection endpoint is required with automatic geo-failover, an Auto-failover group can be used. It provides endpoint redirection. Auto-failover groups provide read-write and read-only listener endpoints. The connection string for the application does not change.

Sunday, September 25, 2022

 

Logging in Multitenant Applications:

 

This is a continuation of the detailed article on Azure Monitoring but with an emphasis on logs.  Azure Monitor and Azure Monitor Logs are different services and Logs are incredibly useful for troubleshooting and notifications.

 

When the cloud monitoring service such as Azure Monitor logs is used to monitor elastic pools and databases for a multitenant application, typically only one account and namespace is needed for the entire integration unlike keyvaults, storage accounts, application instances and databases that proliferate in size and number with the number of tenants. A single geographical region and instance of the Azure Monitor logs is sufficient for the multitenant application solution. 

 

Azure Monitor logs support monitoring thousands of elastic pools and hundreds of thousands of databases.  Azure monitor logs provide a single monitoring solution which can integrate monitoring of different applications and Azure services across multiple Azure subscriptions.

 

A single cloud resource such as an Azure SQL Database can have monitoring and alerting made available via the Portal. It is not convenient to query these logs for large installations or for a unified view across resources and subscriptions. Azure Monitor Logs can collect logs from various resources and their services. The SQL Analytics solution provides several predefined elastic pool and database monitoring and alerting views and queries. It also provides a custom view designer.

 

Platform diagnostics data can be created by simulating a workload on the tenant. Provisioning a batch of tenants prior to the simulation is recommended because it will be close to the real-world scenario.

Also, the Log Analytics Workspace and the Azure SQL Analytics solution must be installed and configured prior to the workload. Azure Monitor Logs is different from Azure Monitor and collects logs and telemetry data in a Log Analytics workspace. The workspace can be created in a dedicated resource group. With the help of the Azure Portal, a Log Analytics workspace and the SQL Analytics solution can be activated. It might take a couple of minutes before the solution is active. The titles or individual databases can then be opened in a drop-down menu.

 

The Portal allows for date range-based filtering, and it shows the pools and databases on the server. Pool level metrics are also available. Monitoring and alerting in the Azure monitor logs are based on queries over the data in the workspace and not specific to a resource. This enables us to               query across databases and subscriptions. Alert rules can also be set up in Azure Monitor Logs. The billing is based on the data volume in the workspace. A free workspace is limited to 500MB/data. After that limit is reached, the data is no longer added to the workspace. This is not the case with the premium tiers.

 


 

Saturday, September 24, 2022

 Multitenant application and monitoring:  

Monitoring is not divested from a multitenant solution provider’s concern. In fact, unlike applications and databases that proliferate across tenants and vary in number and size, monitoring must be a consistent story and experience across tenants and thereby deserves to be studied separately. 

Surely, Public cloud services for monitoring and log analytics can be leveraged for a cloud native multitenant application and we can start from there before we change the topology to include on-premises deployments. As with all multitenant applications deployments, they can come in all forms and modes of deployments spanning on-premises, public cloud and hybrid environments.  

A cloud native monitoring service helps us leverage the best practices of monitoring solutions in the cloud and reduce the complexity of managing something native to the multitenant application while allowing both the monitoring service and the multitenant application to scale independently and remain highly available. A cloud monitoring service is also a complete solution for collecting, analyzing, and acting on the telemetry from the multitenant application solution in the cloud environment.  In the case of Azure, this monitoring program comprises of an application performance management system called the Application Insights, the host monitoring system called the VM Insights and Container Insights, the Log Analytics solution which allows drill down into the monitoring data, smart alerts, and automated actions which help support operations at scale, and visualizations with dashboard and workbooks. The data collected from this comprehensive solution becomes part of Azure Monitor Metrics.   

Azure Monitoring is not only about metrics, but it is also about logs, and it allows us to gather insights, visualize, analyze, respond, and integrate. The monitoring data platform works for both metrics as well as logs. While events and traces become part of logs, metrics are numerical values that quantify application performance at a given point in time.  The metrics store and its visualization with metrics explorer and the log data and its filtering with Log Analytics are just applications dedicated to their respective data. The Azure Monitor uses the Kusto query language that is suitable for simple log queries that also include advanced functionalities such as aggregations, joins, and smart analytics. Kusto benefits from both SQL and Splunk querying practices.   

One of the most interesting aspects of a cloud monitoring service of interest to a multitenant solution provider is that it collects metrics from the multitenant solution, the Guest OS, Azure resources, subscriptions, and of course tenants which pretty much covers the depth and breadth of the systems involved. Additionally, Alerts and Autoscale help determine the appropriate thresholds and actions that become part of the monitoring stack, so the data and the intelligence are together and easily navigated via the dashboard. The dashboards available for monitoring provide a variety of eye-candy charts that better illustrate the data to the viewers than the results of the query. Workbooks provide a flexible canvas for data analysis and the creation of rich visual reports in the Azure Portal.  The analysis is not restricted to just these two. Power BI remains the robust solution to provide analysis and interactive visualizations across a variety of data sources and it can automatically import log data from Azure monitor. Azure Event Hubs is a streaming platform and event ingestion service which permits real-time analytics as opposed to batching or storage-based analysis. APIs from the Azure monitor help with reading and writing data as well as configure and retrieve alerts.   

Let us take a closer look at how the monitoring data is gathered and analyzed internally within the cloud. The architecture behind Azure Monitoring is a diagnostics pipeline.  This pipeline consists of an ingestion gateway, a delivery service, a distributed graph, a normalizer service and scrubber, logs to metrics converter, and an uploader to a global database. The pipeline supports ingestion, streaming, transformations, and querying. This is its hallmark. All these paths are supported end-to-end via the pipeline without any interference from each other.   

The idea behind the monitoring pipeline is one of queuing and pub-sub mechanisms. Logs and metrics flow from gateways to storage queues, where blobs are listened for, scrubbed, forwarded to event hubs, and uploaded to different destinations such as CosmosDB, Azure Data Lake Storage (ADLS), and delivery services. The rate of flow to the queues can be throttled and the schema hints can be propagated to the storage where the schema and notifications power the analytics. The metrics accumulation in an MDM facilitates the logic for throttling and rate adjustments while the schemas are mostly published and queried via Kusto.   

Configurations for different storage containers, queues, and hubs are defined between the collection and the delivery services. These are called Monikers and it is a pairing of Event hub and storage account. The ingestion service is responsible for connecting the monitoring agent with its storage account. The use of this service reduces the number of monikers, the number of blob writes to storage, and the complexity of the distributed graph representation. The storage is billed in terms of transactions and what would earlier take hundreds of transactions and blob writes, would require only tens of transactions using the ingestion or ingress service. It can also aggregate the blobs before writing them to the storage account.   

The corresponding egress service is the delivery service and can be considered an equivalent of Apache Kafka. It comes with a set of producer and consumer definitions and this pub-sub service operates at the event level. There is an application programmability interface provided for consumers who would like to define the monikers instead of the control on the events. The setting up of monikers determines where and how the data is delivered and the use of monikers reduces the bill in an equivalent way to how the ingress did. The destinations are usually Kusto clusters and event hubs. The delivery service forms the core of the pipeline with agents and ingestion pouring data to storage defined by monikers. At the other end of the pipeline are the event hubs and Kusto clusters.   

Collection and Storage have prerequisites. For example, when virtual machines are created, they automatically have a monitoring agent (MA) installed. This agent reaches out to a collection service with an intent to write and define a namespace. The handshake between the monitor and the agent gives the agent the configuration necessary to direct its data to a destination Moniker which can scale automatically for the storage account.   

Unlike the collection and the storage, which are automatically provisioned, the delivery and the paths are set up by the customer using the application programmability interfaces in the extensibility SDK associated with the delivery services. The delivery service then concerns itself merely with the resolving of monikers, the listening on the monikers, the filtering of the events, and its delivery to the Kusto clusters and event hubs. If the destination is unreachable or unavailable, the data is handed off to the snapshot delivery service which reads the delivery service namespaces for retries. The data is never put in memory when the delivery service forwards the data to a cache under a namespace key. The snapshot delivery service acts as the standby destination in place of the unreachable one.   

The access to the data is controlled by the storage access service key also called a blob token that is issued at the time of writing the blob so that the destination can use that token to import the data and handle the single-cast or multi-cast as appropriate to event handlers or appropriate Kusto Cluster. Data copying and rewriting is avoided by merely exchanging the payload information and blob tokens with the delivery service absorbing the process of fetching the permissions from GCS on behalf of the customer.   

The role of the distributed graph might be standing out in the form of a question at this point. It is a service that is used for searching logs and for transformations. It consists of a front-end service and a backend service with each individual component within the FE and BE cloud services as individual micro-services performing a specific task. The front-end service allows the customers to set up query conditions such as job scope and interval period.    

All the monitoring services are region-bound and can be repeated in other regions. Availability within the region such as for disaster recovery purposes requires the use of availability zones. The backend service merely schedules the workers for the appropriate handling of the logs to the customer’s storage account.   

Many miscellaneous activities are specific to the data and whether the data is logs or metrics such as scrubbing, logs to metrics transformations, normalization, and uploading which are handled by dedicated services and serve to enhance the pipeline described so far. The monitoring architecture is generic and one that requires queues, blobs, collections, schedulers, pub-subs, producer-consumers accounts, analysis and reporting stacks and their configurations.  

Most of the resources for Azure monitoring are region scoped. This enables Azure Monitoring to be set up in each region.  Some shared data and resources across these regions may exist in a dedicated region which would power use cases of monitoring via the Azure portal.  

Azure Monitoring also performs continuous monitoring which refers to processes and tools for monitoring each phase of the DevOps and IT operations lifecycles. It helps to continuously ensure the health, performance and reliability of the application and the infrastructure as it moves from deployment to production. It builds on Continuous Integration and Continuous deployment which are ubiquitously embraced by organizations for software development. Azure Monitoring is a unified monitoring solution that provides transparency to the application, runtime, host and cloud infrastructure layers. As a continuous monitoring tool, Azure Monitor allows gates and rollback of deployments based on monitoring data. Software releases to the services hosted in the cloud and have very short software development cycles and must pass through multiple stages and environments before it is made public.  Monitoring data allows any number of environments to be introduced without sacrificing the controls for software quality and gated release across environments. The data not only allows thresholds to be set but also alerts so that appropriate action may be taken. As the software makes its way to the final production environment, the alerts increase in levels and become more relevant and useful for eliminating risks from the production environment.  

It may be argued that tests and other forms of software quality control achieve that as the software goes through the CI/CD pipeline. While this is true, the software quality is enhanced by monitoring data because it is not intrusive or vulnerable to flakiness that many tests are prone to in different environments. The monitoring data, its visualization with dashboards need to be set only once even as the code and tests change over time. The investments in continuous monitoring and its implications boost the planning and predictability of software releases.  

When the cloud monitoring service such as Azure Monitor logs is used to monitor elastic pools and databases for a multitenant application, typically only one account and namespace is needed for the entire integration unlike keyvaults, storage accounts, application instances and databases that proliferate in size and number with the number of tenants. A single geographical region and instance of the Azure Monitor logs is sufficient for the multitenant application solution. 

 

Friday, September 23, 2022

 

Tenancy in Active Directory

Multitenant applications can switch between single tenant deployment mode to multitenant deployment mode. The Active Directory can handle both tenancies and this document discusses how to do that. Single tenant applications must enable sign-in to the home tenant. Multitenant applications are available to users to sign-in to both their home tenant and other tenants.

Public clouds like Azure enable application to be configured for single-tenancy or multi-tenancy via the management portal. The “Accounts in any organizational directory” must be specifically set. The AppID URI of the application must be unique.  Global uniqueness is guaranteed by requiring the AppId URI to have a hostname that matches a verified domain of the Active Directory tenant.

When these applications are developed, there can be a number of challenges from the different policies that can be configured by the administrators.

The best practices for developing sign-in experience on multitenant applications include the following:

1.       The application must be tested in a tenant that has conditional access policies configured.

2.       The principle of least user access must be honored to ensure that the application only requests permission that it needs.

3.       Appropriate names and descriptions for any permissions must be provided. This helps users and administrators know what they are agreeing to before they give consent.

When the sign-ins are configured to be accepted from any Active Directory tenant, it makes the application multi-tenant. If the application is using an existing proprietary account system or from other cloud providers, Active Directory sign-in can still be added via OAuth2, OpenID connect, or SAML integration if the application adds a published UI control on their sign-in page.

In a single-tenant application, the sign-in requests are sent to the tenant’s sign-in endpoint. In the multitenant application, the originating tenant from which the user is attempting to sign-in is not known. Instead, requests are sent to an endpoint that multiplexes across all the Azure AD tenants. When the Microsoft’s identity platform receives a request on the /common endpoint, it signs the user in and discovers the tenant she belongs to and sets this value in the issuer field in the token response to the application. The application can read the issuer field in the token response and take it to correspond to the user’s tenant. The application must handle the validation of the token before reading this field from the multiplexer. There could also be multiple issuer values. Each public cloud AD tenant has a unique issuer value of the form of a URI containing a GUID. A single tenant application can validate the tenant by matching the GUID in the response, but a multitenant application only receives a templatized URI from the multiplexer and is unable to validate it this way. It must maintain a registry of valid tenants and check the issuer value or the tid claim value in the token. It can also choose to admit based on userId registry and ignore the tenant Id altogether.

 


Thursday, September 22, 2022

 Some rules, applications, and guidelines for multitenant application extension development: 

When multitenant application extensions are developed, there are a few best practices that can be called out. This article covers some of them. 

Some of the common pitfalls in multitenant application development include the following: 

  1. Prefix/suffix missing. This is required to ensure a healthy app ecosystem that avoids name collisions. 

  1. DataClassification missing or set incorrectly – There is a tool to detect these, and it is not hard to automate. 

  1. Required translation files missing – These are required for specifying additional languages. 

  1. Missing permission sets - The least privilege execution policy requires proper permission set to be granted. 

  1. Permission errors – These must not be shown unless it entails a necessary action for the user. 

  1. Missing application area tagging – tenant applications can only be categorized with tagging 

  1. Usage category not set – This is required for search because the property helps to provide hints 

  1. Business open/close for a tenant implies a common handler into the invocation of code for the tenant. These must be properly maintained. 

  1. Upgrade procedures – An application can be upgraded properly if the standard operating procedure is followed. 

  1. Use logic owning the artifacts – they should not be updated or maintained if there is no ownership 

  1. Testing with elevated privileges hides the errors users might encounter. 

  1. Testing does not cover a specific scenario because the documentation for the scenario is not proper. 

  1. The configuration and environment settings can alter the behavior of the product and they must be set properly before testing. 

The tenant application extension lifecycle also differs significantly enough from mainstream single-tenant applications to be called out here: 

  1. Migration – Data might need to be migrated between extensions. The process applies to both large- and small-scale data migrations. Upgrade can be treated as a large-scale data migration. Small scale data migrations are where a few objects are hand-picked between extensions. The migration’s direction is dependent on the dependency graph. 

  1. Translations can be applied to multiple properties and these captions are scoped to their entities. They can be changed from multiple places and might live in different artifacts, but they are associated with the entities they are for. 

  1. Tests can be isolated as database transactions in the tenant namespace. The difference between a normal run versus a test run is that a failure in the former means that it stops while a failure in the latter means that it can skip to the next. 

  1. Publishing and installing an extension requires an external registry and installation in individual tenant namespaces. The extension then becomes available to the users in the client. 

  1. Updating and upgrading differ in the behavior of the code before and after. Upgrade strives to maintain backward compatibility while update can replace wholesome. 

  1. Deprecation best practices involve the use of conditional directives to surround the code to be obsoleted.