This is a continuation of a series of articles on crowdsourcing application and including the most recent article. The original problem statement is included again for context.     

 

Social engineering applications provide a wealth of information to the end-user, but the questions and answers received on it are always limited to just that – social circle. Advice solicited for personal circumstances is never appropriate for forums which can remain in public view. It is also difficult to find the right forums or audience where the responses can be obtained in a short time. When we want more opinions in a discrete manner without the knowledge of those who surround us, the options become fewer and fewer. In addition, crowd-sourcing the opinions for a personal topic is not easily available via applications. This document tries to envision an application to meet this requirement.     

 

The previous article continued the elaboration on the usage of the public cloud services for provisioning queue, document store and compute. It talked a bit about the messaging platform required to support this social-engineering application. The problems encountered with social engineering are well-defined and have precedence in various commercial applications. They are primarily about the feed for each user and the propagation of solicitations to the crowd. The previous article described selective fan out. When the clients wake up, they can request their state to be refreshed. This perfects the write update because the data does not need to be sent out. If the queue sends messages back to the clients, it is a fan-out process. The devices can choose to check-in at selective times and the server can be selective about which clients to update. Both methods work well in certain situations. The fan-out happens in both writing as well as loading. It can be made selective as well. The fan-out can be limited during both pull and push. Disabling the writes to all devices can significantly reduce the cost. Other devices can load these updates only when reading. It is also helpful to keep track of which clients are active over a period so that only those clients get preference.  In this section, we talk about the retry storm antipattern. 

 

This antipattern occurs in social engineering applications when] I/O requests fail due to transient errors and services must retry their calls. It helps overcome errors, throttle and rate limits and avoid surfacing and requiring user intervention for operational errors. But when the number of retries or the duration of retries is not governed, the retries are frequent and numerous, which can have a significant impact on performance and responsiveness. Network calls and other I/O operations are much slower compared to compute tasks. Each I/O request has a significant overhead as it travels up and down the networking stack on local and remote and includes the round trip time, and the cumulative effect of numerous I/O operations can slow down the system. There are some manifestations of the retry storm.

Reading and writing individual records to a database as distinct requests – When records are often fetched one at a time, then a series of queries are run one after the other to get the information. It is exacerbated when the Object-Relational Mapping hides the behavior underneath the business logic and each entity is retrieved over several queries. The same might happen on writing for an entity. When each of these queries is wrapped in their own retry, they can cause severe errors. 

Implementing a single logical operation as a series of HTTP requests. This occurs when objects residing on a remote server are represented as a proxy in the memory of the local system. The code appears as if an object is modified locally when in fact every modification is coming with at least the cost of the RTT. When there are many networks round trips, the cost is cumulative and even prohibitive. It is easily observable when a proxy object has many properties, and each property get/set requires a relay to the remote object. In such a case, there is also the requirement to perform validation after every access.

Reading and writing to a file on disk – File I/O also hides the distributed nature of interconnected file systems.  Every byte written to a file on amount must be relayed to the original on the remote server. When the writes are several, the cost accumulates quickly. It is even more noticeable when the writes are only a few bytes and frequent. When individual requests are wrapped in a retry, the number of calls can rise dramatically.

There are several ways to fix the problem. They are about detection and remedy. The remedies include capping the number of retry attempts and preventing retrying for a long period of time. The retries could include an exponential backoff strategy that increases the duration between successive calls exponentially, handle errors gracefully, use the circuit breaker pattern which is specifically designed to break the retry storm. Official SDKs for communicating to Azure Services already include sample implementations of retry logic. When the number of I/O requests is many, they can be batched into coarse requests. The database can be read with one query substituting many queries. It also provides an opportunity for the database to execute it better and faster. Web APIs can be designed with the REST best practices. Instead of separate GET methods for different properties, there can be a single GET method for the resource representing the object. Even if the response body is large, it will likely be a single request. File I/O can be improved with buffering and using cache. Files may need not be opened or closed repeatedly. This helps to reduce fragmentation of the file on disk.

When more information is retrieved via fewer I/O calls and fewer retries, the operational necessary evil becomes less risky but there is also a risk of falling into the extraneous fetching antipattern. The right tradeoff depends on the usages. It is also important to read-only as much as necessary to avoid both the size and the frequency of calls and their retries. Sometimes, data can also be partitioned into two chunks, frequently accessed data that accounts for most requests and less frequently accessed data that is used rarely. When data is written, resources need not be locked at too large a scope or for a longer duration. Retries can also be prioritized so that only the lower scope retries are issued for idempotent workflows.

 

 

 

This is a continuation of a recent article on multi-tenancy.

The previous article introduced multi-tenancy with an emphasis on dedicated and shared resource models. It suggested that provisioning of resources, their lifetime, scope, and level can be isolated in the client’s perspective and run deep into the infrastructure as necessary. The infrastructure makes choices on the costs, but the clients want separation and protection of resources, privacy, and confidentiality of their data as well as tools for control and data plane management The benefits that the infrastructure provides with multi-tenancy include common regulatory controls, governance and security framework as well as a scheduled sweep of resources.

We took the opportunity to discuss Azure storage service as an example of a service that implements multi-tenancy. The design choices for the components at different levels that articulated multi-tenancy were called out. Specifically, the adaptive algorithm and the movement of partitions is equally applicable  to dedicated resources from a SaaS. Multi-tenant resource providers today ask for their clients to choose regions because it provides high availability given that resources have little or no difference when used from one region or another.

A case was made for service class differentiation from a multi-tenant service. Resources can be grouped and tiered to meet different service level classifications, but this can also go deeper into the provisioning logic of the multi-tenant service.  Service class differentiation can be achieved with quality-of-service implementation and costing strategies.

Quality of Service guarantees are based on sharing and congestion studies. To study congestion, resource requests are marked for distributor to distinguish between different classes and new distribution policies to treat requests differently. QoS guarantees provide isolation for one class from other classes. For example, when allocating different bandwidths to each application flow by a router, bandwidth may not be efficiently used. QoS guarantees that while providing isolation to different classes, it is preferred to utilize the resources as efficiently as possible. Since most multi-tenancy providers have an infrastructure capacity that cannot be exceeded, there is a need to control admission. Tenants can declare their requirements, and the multi-tenant service provider can block if it cannot provide the resources. 

The costing strategy has been favored to be on a pay-as-you-go basis because this improves the value for the client while giving them the incentive to reclaim resources and keep their costs down further.  Costing and monitoring go hand in hand so a certain amount visibility into usage is demanded from the infrastructure.  Continuous operation for each phase of the DevOps and IT operations lifecycles is necessary. Health, performance, and reliability of the provisioned resources play a critical role in their usage and affect billing. Continuous monitoring of API is also possible via Synthetic monitoring. It provides proactive visibility into API issues before customers find the issues themselves. This is automated probing then ensures end-to-end validation of specific scenarios. The steps to setup a Synthetic monitoring includes onboarding, provisioning, and deployment.

The costs can make service levels more appealing when the costing strategy involves a weighted cost analysis. By making the costing be based on granular activities, the organization of costs to categories is removed and separate focus on all the labor involved in the cloud operations is removed. A virtual data warehouse and a star schema can help to capture all dimensions including time and space to perform aggregations for better querying, reporting and visualizations.

 This is a continuation of a series of articles on crowdsourcing application and including the most recent article. The original problem statement is included again for context.     

 

Social engineering applications provide a wealth of information to the end-user, but the questions and answers received on it are always limited to just that – social circle. Advice solicited for personal circumstances is never appropriate for forums which can remain in public view. It is also difficult to find the right forums or audience where the responses can be obtained in a short time. When we want more opinions in a discrete manner without the knowledge of those who surround us, the options become fewer and fewer. In addition, crowd-sourcing the opinions for a personal topic is not easily available via applications. This document tries to envision an application to meet this requirement.     

 

The previous article continued the elaboration on the usage of the public cloud services for provisioning queue, document store and compute. It talked a bit about the messaging platform required to support this social-engineering application. The problems encountered with social engineering are well-defined and have precedence in various commercial applications. They are primarily about the feed for each user and the propagation of solicitations to the crowd. The previous article described selective fan out. When the clients wake up, they can request their state to be refreshed. This perfects the write update because the data does not need to be sent out. If the queue sends messages back to the clients, it is a fan-out process. The devices can choose to check-in at selective times and the server can be selective about which clients to update. Both methods work well in certain situations. The fan-out happens in both writing as well as loading. It can be made selective as well. The fan-out can be limited during both pull and push. Disabling the writes to all devices can significantly reduce the cost. Other devices can load these updates only when reading. It is also helpful to keep track of which clients are active over a period so that only those clients get preference.  In this section, we talk about the busy frontend antipattern. 

 

This antipattern occurs when there are many background threads that can starve foreground tasks of their resources which decreases response times to unacceptable levels. There is a lot of advantages to running background jobs which avoids the interactivity for processing and can be scheduled asynchronously. But the overuse of this feature can hurt performance due to the tasks consuming resources that foreground workers need for interactivity with the user, leading to a spinning wait and frustrations for the user. It appears notably when the foreground is monolithic compressing the business tier with the crowdsourcing application frontend. Runtime costs might shoot up if this tier is metered. A crowdsourcing application tier may have finite capacity to scale up. Compute resources are better suitable for scale out rather than scale up and one of the primary advantages of a clean separation of layers and components is that they can be hosted even independently. Container orchestration frameworks facilitate this very well. The Frontend can be as lightweight as possible and built on model-view-controller or other such paradigms so that they are not only fast but also hosted on separate containers that can scale out.

 

This antipattern can be fixed in one of several ways. First the processing can be moved out of the application tier into an Azure Function or some background api layer. If the application frontend is confined to data input and output display operations using only the capabilities that the frontend is optimized for, then it will not manifest this antipattern. APIs and Queries can articulate the business layer interactions. The application then uses the .NET framework APIs to run standard query operators on the data for display purposes. 

 

UI interface is designed for purposes specific to the application. The introduction of long running queries and stored procedures often goes against the benefits of a responsive application. If the processing is already under the control of the application techniques, then they should not be moved.  

Avoiding unnecessary data transfer solves both this antipattern as well as chatty I/O antipattern. When the processing is moved to the business tier, it provides the opportunity to scale out rather than require the frontend to scale up. 

 

Detection of this antipattern is easier with the monitoring tools and the built-in supportability features of the application layer. If the front-end activity reveals significant processing and very low data emission, it is likely that this antipattern is manifesting. 

 

Examine the work performed by the Frontend in terms of latency and page load times which can be narrowed down by callers and scenarios, may reveal just the view models that are likely to be causing this antipattern 

 

Finally, periodic assessments must be performed on the application tier. 

 

This is a continuation of a series of articles on Microsoft Azure from an operational point of view that surveys the different services from the service portfolio of the Azure public cloud. The most recent article on Service Fabric discussed an infrastructure for hosting. In this article, we explore the Dataverse and solution layers.

Microsoft Dataverse is a data storage and management system for the various Power Applications so that they are easy to use with Power Query. The data is organized in tables some of which are built-in and standard across applications, but others can be added on a case-by-case basis for applications. These tables enable applications to focus on their business needs while providing a world class, secure and cloud-based storage option for the data that are 1. Easy to manage, 2. Easy to secure, 3. Accessible via Dynamics 365, has rich metadata, logic and validation, and come with productivity tools. Dynamics 365 applications are well-known for enabling businesses to quickly meet their business goals and customer scenarios and Dataverse makes it easy to use the same data across different applications. It supports incremental and bulk loads of data both on a scheduled and on-demand basis.

Logic and validation performed on the Dataverse include business rules, business process flows, workflows, and business logic with code although they are by no means limited to just these. Dataverse is another option when compared to connectors for external data sources and AI processing stacks.

Solutions are used to transport applications and components from one environment to another or to add customizations to an existing application. It can comprise applications, site maps, tables, processes, resources, choices, and flows. It implements Application Lifecycle management and powers Power Automate. There are two types of solutions (managed and unmanaged) and the lifecycle of a solution involves create, updates, upgrade and patch. The managed properties of a solution govern which components are customizable. A solution can be created using the navigation menu of the Power Apps. Microsoft AppSource is the marketplace where solutions tailored to a business need can be found.

Managed and unmanaged solutions can co-exist at different levels within a Microsoft Dataverse environment. They form two distinct layer levels. What the user sees as runtime behavior, comes from the active customizations of an unmanaged layer which in turn might be supported by a stack of one or more user-defined managed solutions and system solutions in the managed layer.  Managed solutions can also be merged. The solution layers feature enables one to see all the solution layers for a component.

This is a continuation of a series of articles on Microsoft Azure from an operational point of view that surveys the different services from the service portfolio of the Azure public cloud. The most recent article on Service Fabric discussed an infrastructure for hosting. In this article, we explore the Dataverse and solution layers.

Microsoft Dataverse is a data storage and management system for the various Power Applications so that they are easy to use with Power Query. The data is organized in tables some of which are built-in and standard across applications, but others can be added on a case-by-case basis for applications. These tables enable applications to focus on their business needs while providing a world class, secure and cloud-based storage option for the data that are 1. Easy to manage, 2. Easy to secure, 3. Accessible via Dynamics 365, has rich metadata, logic and validation, and come with productivity tools. Dynamics 365 applications are well-known for enabling businesses to quickly meet their business goals and customer scenarios and Dataverse makes it easy to use the same data across different applications. It supports incremental and bulk loads of data both on a scheduled and on-demand basis.

Logic and validation performed on the Dataverse include business rules, business process flows, workflows, and business logic with code although they are by no means limited to just these. Dataverse is another option when compared to connectors for external data sources and AI processing stacks.

Solutions are used to transport applications and components from one environment to another or to add customizations to an existing application. It can comprise applications, site maps, tables, processes, resources, choices, and flows. It implements Application Lifecycle management and powers Power Automate. There are two types of solutions (managed and unmanaged) and the lifecycle of a solution involves create, updates, upgrade and patch. The managed properties of a solution govern which components are customizable. A solution can be created using the navigation menu of the Power Apps. Microsoft AppSource is the marketplace where solutions tailored to a business need can be found.

Managed and unmanaged solutions can co-exist at different levels within a Microsoft Dataverse environment. They form two distinct layer levels. What the user sees as runtime behavior, comes from the active customizations of an unmanaged layer which in turn might be supported by a stack of one or more user-defined managed solutions and system solutions in the managed layer.  Managed solutions can also be merged. The solution layers feature enables one to see all the solution layers for a component.

 

 

 

 

 

This is a continuation of a series of articles on crowdsourcing application and including the most recent article. The original problem statement is included again for context.     

 

Social engineering applications provide a wealth of information to the end-user, but the questions and answers received on it are always limited to just that – social circle. Advice solicited for personal circumstances is never appropriate for forums which can remain in public view. It is also difficult to find the right forums or audience where the responses can be obtained in a short time. When we want more opinions in a discrete manner without the knowledge of those who surround us, the options become fewer and fewer. In addition, crowd-sourcing the opinions for a personal topic is not easily available via applications. This document tries to envision an application to meet this requirement.     

 

The previous article continued the elaboration on the usage of the public cloud services for provisioning queue, document store and compute. It talked a bit about the messaging platform required to support this social-engineering application. The problems encountered with social engineering are well-defined and have precedence in various commercial applications. They are primarily about the feed for each user and the propagation of solicitations to the crowd. The previous article described selective fan out. When the clients wake up, they can request their state to be refreshed. This perfects the write update because the data does not need to be sent out. If the queue sends messages back to the clients, it is a fan-out process. The devices can choose to check-in at selective times and the server can be selective about which clients to update. Both methods work well in certain situations. The fan-out happens in both writing as well as loading. It can be made selective as well. The fan-out can be limited during both pull and push. Disabling the writes to all devices can significantly reduce the cost. Other devices can load these updates only when reading. It is also helpful to keep track of which clients are active over a period so that only those clients get preference.  In this section, we talk about the busy frontend antipattern. 

 

This antipattern occurs when there are many background threads that can starve foreground tasks of their resources which decreases response times to unacceptable levels. There is a lot of advantages to running background jobs which avoids the interactivity for processing and can be scheduled asynchronously. But the overuse of this feature can hurt performance due to the tasks consuming resources that foreground workers need for interactivity with the user, leading to a spinning wait and frustrations for the user. It appears notably when the foreground is monolithic compressing the business tier with the crowdsourcing application frontend. Runtime costs might shoot up if this tier is metered. A crowdsourcing application tier may have finite capacity to scale up. Compute resources are better suitable for scale out rather than scale up and one of the primary advantages of a clean separation of layers and components is that they can be hosted even independently. Container orchestration frameworks facilitate this very well. The Frontend can be as lightweight as possible and built on model-view-controller or other such paradigms so that they are not only fast but also hosted on separate containers that can scale out.

 

This antipattern can be fixed in one of several ways. First the processing can be moved out of the application tier into an Azure Function or some background api layer. If the application frontend is confined to data input and output display operations using only the capabilities that the frontend is optimized for, then it will not manifest this antipattern. APIs and Queries can articulate the business layer interactions. The application then uses the .NET framework APIs to run standard query operators on the data for display purposes. 

 

UI interface is designed for purposes specific to the application. The introduction of long running queries and stored procedures often goes against the benefits of a responsive application. If the processing is already under the control of the application techniques, then they should not be moved.  

Avoiding unnecessary data transfer solves both this antipattern as well as chatty I/O antipattern. When the processing is moved to the business tier, it provides the opportunity to scale out rather than require the frontend to scale up. 

 

Detection of this antipattern is easier with the monitoring tools and the built-in supportability features of the application layer. If the front-end activity reveals significant processing and very low data emission, it is likely that this antipattern is manifesting. 

 

Examine the work performed by the Frontend in terms of latency and page load times which can be narrowed down by callers and scenarios, may reveal just the view models that are likely to be causing this antipattern 

 

Finally, periodic assessments must be performed on the application tier. 

 

Azure Service Fabric Cluster Instance and Replicas:

This is a continuation of the Azure Service Fabric articles with most recent one as included here. The Azure Service Fabric instance allows specifying the TargetReplicaSize and the MinReplicaSetSize for a stateful service. The first determines the number of replicas that the system creates and maintains for each replica set of a service. The MinReplicaSetSize is the minimum allowed number of replicas for each replica set of a service.

These two parameters allow a configuration that permits two concurrent failures to occur without the partition going in a quorum loss. That situation can happen when there’s one planned failover upgrade bringing node or replica down and one unplanned failover such as when node crashes.

If the TargetReplicaSetSize = 5, MinReplicaSetSize = 3, then without failures, there will be five replicas in the view of the replica set and if failures occur, then the ServiceFabric will allow a decrease until it reaches MinReplicaSetSize.

ServiceFabric uses the majority quorum of the number of replicas maintained in this view and that is the minimum level of reliability for the operation of the cluster. If the replicas follow below this level, then further writes will be disallowed. Examples of suboptimal configurations involve a quorum loss of TargetReplicaSetSize = 3 and MinimumReplicaSetSize = 2 or when both are equal.

Stateless services do not need to specify replicas. They can have their instances scaled to the same count as in a replica set size. An instance of a stateless service is a copy of the service logic that runs on one of the nodes of the cluster. An instance within a partition is uniquely identified by its instanceId.

The lifecycle of an instance is modeled in a cycle of transitions between Inbuild stage, Ready stage, Closing stage and Dropped stage with an occasional transition from Ready to Dropped.

The Inbuild stage of a ClusterResourceManager determines the placement for the instance and enters its lifecycle. The instance is started on the node. When it exits it transitions to the ready state.  If the application host or node for this instance crashes, it transitions to the dropped state.

In the closing state, the Service Fabric is in the process of shutting down the instance on a node. When it completes the shutdown, it transitions to the dropped state. In the dropped state, the metadata maintained by the Service Fabric is marked to be deleted.

A replica of a stateful service is a copy of the service logic running on one of the cluster nodes. The replica lifecycle has a few additional stages: down, opening and stand-by. The down state is when the replica code is not running. The opening state is when ServiceFabric needs to bring the replica back up again. A standby stage is entered when a replica was down and is open now but has not entered the replica set. If the keep duration expires, the standby is discarded.

The role of a replica determines its function in the replica set which includes Primary, ActiveSecondary, IdleSecondary, None and Unknown.