Performance for multitenant application (continued)

This is a continuation of the articles on multitenancy with the most recent one linked https://1drv.ms/w/s!Ashlm-Nw-wnWhLZnYUBoDUNcjAHNwQ?e=NAo7vM. This article continues to focus on performance

Large volumes of web service calls can cause stability and performance issues. It is important to understand the operational limits and to scale such that the load always falls under the limit. External applications can handle the HTTP Status codes 429 for too many requests and 504 for gateway timeout.

Handling status code 429 requires the client to adopt a retry logic while providing a cool off period. Retries can be regular interval, incremental interval, exponential backoff, and randomization. Status code 504 requires the client to refactor the long running request to execute within the time limit by splitting the request into multiple requests. Then the potential 429 codes can be handled by a backoff strategy. A common pattern is to implement a queue in the external application to flatten the spikes in the traffic. If the request gets a 429, it is put back in the queue and one of the retry strategies is applied.

 

Reports can be efficiently written. They can pertain to a single instance of an entity like an invoice or they can be more analytical in nature by performing joins across multiple instances.  Faster reports can be enabled by performing read scale-out to read data from a read-only copy of the database, by using partial records to reduce the data loaded from the database, by using AL queries to optimize the way data is read from the database, and using word layouts and RDL layouts which can result in slower performance with document reports, especially for actions related to the user interface.

 

If a data lake or a data warehouse is involved, there are typically two types of data extractions: 1. a historical load with all the data from a given point-of-time and 2. delta loads on what changed since the historical load. The fastest and least disruptive way to get  ahistorical load is usually a SQL BACPAC file that can be restored on a private database server.  The fastest or the least disruptive way to get delta loads is to setup API queries configured with read scale-out and use of data audit field LastModifiedOn.

 

Performance for multitenant application

This is a continuation of the articles on multitenancy with the most recent one linked https://1drv.ms/w/s!Ashlm-Nw-wnWhLZnYUBoDUNcjAHNwQ?e=NAo7vM. This article focuses on performance.

 

The multitenant application discussed so far has an application server and a database. Performance is improved by 1. Writing efficient pages, 2. Efficient web services, 3. Efficient reports, 4. AL performance patterns, 5. Efficient data access, 6. Testing and validating performance, 7. Tuning the development environment 8. And using the AL profiler to analyze performance.

 

Efficient pages are written by using patterns that get a page to load faster. These include: avoiding unnecessary recalculation, 2. Doing less work, 3. And offloading the UI thread. Caching the data and refreshing the cache regularly avoids recalculation.  This saves time each time the page is loaded. Querying objects are notorious for recalculation since they reach the database each time. Caching the results from an API works significantly better.

 

Reducing the amount of work also speeds things up.  A simple page with few UI elements can also be ease of use and navigation.  Removing calculated fields from lists if they aren’t needed  and removing the field definition  or page extension definition improves loading of pages that list data.

 

Creating dedicated lookup pages instead of the normal pages when dropdown like logic is involved, and removing triggers and fact boxes will help because a default page will render all controls

Offloading the UI thread with say page background tasks can get a more responsive and faster UI. Custom controls that require heavy duty logic can also be avoided.

 

Avoiding expose of calculated fields, avoiding heavy duty logic in pre and post handlers of getting records, refactoring the page and its code so that values are persisted can reduce performance hits. It is not recommended to use temp tables if there are many records. Fetching and inserting each record in a temp table without caching data can be detrimental to performance. If the number of  records exceeds a hundred, this antipattern is easy to detect.

 

Parent and child records need not be inserted in parallel. This condition causes locks on parent and integration record tables because parallel calls try to update the same parent record. It is best to do it incrementally by allowing one to finish before another or by putting them in a transaction batch.

 

A deprecated protocol can be avoided. OData version 4 and APIs have best performance. API queries and pages are faster with newer technology stacks.

 

API pages and API queries are better than exposing ui pages as web service endpoints. If the latter must be implemented, then triggers need to run for all the records returned from the server. If we want OData endpoints that work as data readers, we can use API queries. OData has a few performance callouts such as limiting the set with $filter and $top if there’s an expensive $expand, using a transaction batch and read-only data access intent.

 

 

Synchronization and Data security for multitenant databases

This is a continuation of the articles on multitenancy with the most recent one linked https://1drv.ms/w/s!Ashlm-Nw-wnWhLZnYUBoDUNcjAHNwQ?e=NAo7vM. This article focuses on data synchronization and security. 

The customer’s data is stored in a separate business database, each of which is a tenant in the deployment. By separating application from data, the same stack can be deployed to many customers with centralized maintenance of the application and the isolation of each tenant.  The application database contains the tables that define an application.

Data and applications are partitioned and isolated from one tenant to another in a multitenant application. The application data is in a dedicated application database, separate from the business data in the tenant database. The revert to a single-tenant deployment mode requires the revert of the separation of application data and business data and the merging of the two databases. A single tenant can be folded from the multitenant mode and leaving the remaining tenants in the existing deployment by adding the application tables to the relevant tenant database and leaving the original application database unchanged. If the application data is separated into a dedicated database, but it is not used in a multitenant deployment mode, the databases can still be merged.

Tenant database must be synchronized with the application database in both modes of deployment – single tenancy as well as multitenancy.

Synchronization can occur in the following ways: 1) continuously, which is typical for transactional replication, 2) On demand, which is typical for merge replication. And 3) On a schedule, which is typical for snapshot replication. Replication happens by way of state or operations transfer which is like the well-known log shipping and mirroring options. State gets overwritten on the destinations whereas operations are re-executed on the destination. 

 

If there are ten customers to the current solution, each of them can get a tenant database dedicated to store the company’s business data. The knowledge about the shared application is then stored in a dedicated application database. When this solution is upgraded but the deployment mode is not changed, we can still have a single database that has one or more companies in it. We can also choose to extract the application tables to an application database but still have one business data database that has one or more companies in it. 

 

A security system must be in place to control which objects or tables a user can access within each database. We can specify the type of access each user has to these objects and tables and whether they are able to read, modify or enter data. Permissions can be granted at both the table level and the record level. It should be easy to query the permissions assigned to a user and to find out who can access a particular database. Typically, there are four different levels of security – Database, Company, Object, and Record. The first layer of security is database security when the credentials are checked even before the database is opened. Logins can be native to the database server or integrated with the operating system. The user is granted access only after the login is authenticated. Authorization is determined from role-based access control. A database can have several companies. Each company can use its own tables or share the tables with other companies. User’s access to companies is controlled by permissions sets. Object level security is determined by the permissions within the permission set. Permissions can be defined for all types of objects such as table data, table, pages, report, code units, data import and export units, query, and system. The lowest level of security is the record level security which limits the access that a user has to the data in a table. It is implemented by creating security filters on table data.

 

 

Switching modes:

This is a continuation of the articles on multitenancy with the most recent one linked https://1drv.ms/w/s!Ashlm-Nw-wnWhLZlxLBXSWQZmcirVA?e=ugcdca . This article focuses on migrating between deployment modes. 

It’s possible to convert a solution from a multitenant deployment mode to single-tenancy. This article describes some of the steps.

Data and applications are partitioned and isolated from one tenant to another in a multitenant application. The application data is in a dedicated application database, separate from the business data in the tenant database. The revert to a single-tenant deployment mode requires the revert of the separation of application data and business data and the merging of the two databases. A single-tenant can be folded from the multitenant mode and leaving the remaining tenants in the existing deployment deployment by adding the application tables to the relevant tenant database and leaving the original application database unchanged. If the application data is separated into a dedicated database, but it is not used in a multitenant deployment mode, the databases can still be merged.

Application code instances and configurations per tenant are still easily honored because as the applications instances are decreased the multitenant mode ensured that the infrastructure was shared. This made it easier to scale in or out and continues to work for migrating back into the single tenant mode.

Data, on the other hand, requires immense protection, isolation and compliance standards. It is not easy for the shared infrastructure to interpret the data belonging to a tenant. The multitenant solution provider can move or migrate the data one by one but it cannot make changes to syntax, semantics or format.

Even when the data is allowed to be read for operational purposes, the size of the data can vary from a few GB to TeraBytes. In such cases, even a migration of data from one table to another poses a long running operation that must be carefully planned and executed. It’s best to move a single record at a time in an idempotent manner and with failure detection and corrections so that when the operation succeeds for one single record, it can be repeated for all the records in the table.

When we refer to a tenant, we refer to it by the tenant ID but it is also possible to refer to them by the hostnames for the tenants in the deployment. A tenant specific sub-domain is setup in this case. The tenant host name, mytenant.myservice.com must be specified as an alternative in the tenant configuration.  The URL can specify the tenant ID and the tenant host name if we specify the hostnames as an alternative IDs for tenants. 

 

 

Multitenant Application example:

Let us take an example of an application that can be deployed in a single tenant mode and a multitenant mode. In a multitenant deployment, the information about the application is stored in a separate database. The customer’s data is stored in a separate business database, each of which is a tenant in the deployment. By separating application from data, the same stack can be deployed to many customers with centralized maintenance of the application and the isolation of each tenant.  The application database contains the tables that define an application.

If there are ten customers to the current solution, each of them can get a tenant database dedicated to store the company’s business data. The knowledge about the shared application is then stored in a dedicated application database. When this solution is upgraded but the deployment mode is not changed, we can still have a single database that has one or more companies in it. We can also choose to extract the application tables to an application database but still have one business data database that has one or more companies in it. In both scenarios, we have not migrated to multitenancy, but in the second scenario, we have prepared our solution so that we can move to multitenancy at a later point.

When a multitenant solution is deployed, a relationship is activated between the server instance by mounting the tenant to the instance. Similarly, to disconnect a tenant, the dismount needs to be performed. When the tenants are mounted, the tenant configuration is stored in the tenants table of the application database that is connected to the instance. If we connect additional server instances to the same application database, then the server instances will automatically inherit the tenant configurations from the application database. This means that the existing tenants will be automatically mounted to the new server instance.  If we mount or dismount an instance, then we have to perform the operation on one of the instances. The other instances will automatically detect and update the changes.

When we refer to a tenant, we refer to it by the tenant ID but it is also possible to refer to them by the hostnames for the tenants in the deployment. A tenant specific sub-domain is setup in this case. The tenant host name, mytenant.myservice.com must be specified as an alternative in the tenant configuration.  The URL can specify the tenant ID and the tenant host name if we specify the hostnames as an alternative IDs for tenants.

 

Multitenant Applications and the process of upgrading:

This is a continuation of the articles on multitenancy with the most recent one linked MultitenantApplicationsPart84.docx. This article focuses on synchronizing data between publishers and subscribers in a multitenant deployment.

Synchronization can occur in the following ways: 1) continuously, which is typical for transactional replication, 2) On demand, which is typical for merge replication. And 3) On a schedule, which is typical for snapshot replication. Replication happens by way of state or operations transfer which is like the well-known log shipping and mirroring options. State gets overwritten on the destinations whereas operations are re-executed on the destination.

When a subscription is synchronized, different processes occur based on the different types of replication. Snapshot synchronization is one where the Distribution Agent reapplies the snapshot at the Subscriber so that schema and data at the subscription database is consistent with the publication database. Modifications to data or schema when made at the publisher, require a new snapshot to be generated in order to be propagated. Transactional replication is one where the distribution agent transfers, updates, inserts, deletes and any other change from the distribution database to the subscriber. Merge replication is one where the merge agent uploads the changes from the subscriber to publisher and then downloads changes from the publisher to the subscriber. Conflicts are detected and resolved. Data is converged and the publisher and all subscribers eventually end up with the same data values. If conflicts were detected and resolved, work that was committed by some of the users is changed to resolve the conflict according to defined policies

Synchronizing a push subscription can be done on demand by connecting the source such as a publisher, a subscriber, or a monitor agent in the middle, to a tool that that can initiate the synchronization to the destination. Status of the synchronization is usually displayed and will show completed when done. A pull synchronization only happens from the subscriber side.

Synchronization can also be scheduled by defining a synchronization schedule that controls the replication agent will run. Distribution agents and merge agents can run continuously, run on demand and run on a schedule.

Reference: https://1drv.ms/w/s!Ashlm-Nw-wnWhLMfc6pdJbQZ6XiPWA?e=fBoKcN 

 

 

Multitenant Applications and the process of upgrading:

This is a continuation of the articles on multitenancy with the most recent one linked here. This article focuses on upgrading in a multitenant deployment.

Application upgrade and data upgrade are two different things. The data upgrade can be done even after the application has been upgraded. In this case, the old tenants are mounted on the server instance and then the data upgrade is performed.

Data conversion tools are usually provided to assist with the data upgrade process. In this case, the old data with the old version’s table and field structure is converted so that it can work with the new version’s table and field structure.

Before any upgrade occurs, a bunch of prerequisites must be met. Usually, the dry run of an upgrade can indicate potential issues. A tool like the upgrade advisor can assist with this discovery before the actual upgrade is performed. Prerequisites might include converting certain components in the older version, exporting permissions and permission sets from the old deployment, exporting encryption keys from the old deployment, preparing for transitioning, and installing multitenant application components.

The upgrade of the application code is not expected to break any of its functionality. In the multitenant mode, the application code is upgraded first, followed by creating a database on the new platform, then upgrading the application to the database and upgrading the data by mounting the tenant on the application database.

A tenant database can be prepared for data upgrade by first backing it up, followed by uninstalling the software components, and dismounting the tenant from the old server instance. When the data is upgraded on the tenant, it must be mounted on the server instance.

The tenant database might be different from the application database. In such cases, some synchronization is required. Synchronization can occur in the following ways: 1) continuously, which is typical for transactional replication, 2) On demand, which is typical for merge replication. And 3) On a schedule, which is typical for snapshot replication. Replication happens by way of state or operations transfer which is like the well-known log shipping and mirroring options. State gets overwritten on the destinations whereas operations are re-executed on the destination.

Post upgrade tasks include the importing of permission and permission sets, importing encryption keys, and uploading the customer license.

Reference: https://1drv.ms/w/s!Ashlm-Nw-wnWhLMfc6pdJbQZ6XiPWA?e=fBoKcN