Configuration Automation part 3.

This article is a continuation of the part 1 article on Configuration Automations and part 2 that describes the syntax of the configuration templates.  It describes the process of finding and loading configurations.

Most application configurations are composable. This means that they can be declared in separate files and can be referenced one from the other. Configuration key values must be unique so that their values are deterministic during the execution. Organization of files for application configuration is subject to DevOps considerations and best practices such as for minimizing the impact on changes to code or environment.

Application configuration files are local to the application. The configuration is loaded from the local first before resolving from remote. Remote configuration sources can include configuration repositories such as with the operating system, remote store or others. They can also include services that are injected as dependencies into the application so that they can be read from a dedicated store that is not on the same host as the application.

The process of resolving the configuration keys is dependent on the system but the visibility into its final value at the time of execution can be found from logs. Most applications will log their configurations at least at the time of usage for diagnostics. This helps with determining whether they were intended to be set but it does not provide a way to change them. One way to change configuration keys is to store them in a repository outside the application so that they can be fetched dynamically by a component service of the application that was registered to be used at application initialization time.

The use of dynamic retrieval of configuration key values is a concept that allows the alteration of application behavior without having to restart it. This is helpful to change the behavior on subsequent requests to the application after initialization, but it does not give any indication on the trigger that caused the configuration to change. Audit is one way in which these changes can be effectively found but they happen post-mortem.

Consequently, configuration services often use a publisher-subscription method for listening to changes so that appropriate handlers can take actions. This eliminates code or expensive and inefficient polling services. Events are pushed through an Event Grid to subscribers. Common App configuration event scenarios include refreshing application configuration, triggering deployment or any configuration-oriented workflow. There is a difference between polling mechanisms and event subscriptions where each has its advantage and can be seen with the help of the size of the changes made to the configuration store. When the changes are infrequent, but the scenario requires immediate responsiveness, event-based architecture can be especially efficient.  The publisher and subscribers get just the data that has changed and as soon as the change happens which enables downstream systems to be reactive and multiple subscribers to receive the notifications at once. They are also relieved from implementing anything more than a message handler. There is also the benefit that comes from the scope of the change that is passed down to the subscribers, so they get additional information on just what configuration has changed.  If the changes become frequent, the number of notifications is large leading up to performance bottleneck with variations in the queue size and delays in the messages.  Instead, when the changes span a lot of keys, it is best to get those changes in bulk. A polling mechanism can get changes in batches over time and then process through all those changes. It can even find only the updates that were made from the time of the previous polling.  This enables incremental updates at the destination. Since a polling mechanism is a loop that perpetually finds changes, if any, and applies them to the destination, it can work in the background even as a single worker. A polling mechanism is a read-only operation and therefore it does not need to fetch the data from the store where the configuration is being actively updated. It can even fetch the data from a mirror of the configuration store. Separation of the read-write store from a read-only store helps improve the throughput for the clients that update the configuration store. Read-only access is only for querying purposes and with a store that is dedicated to this purpose, the configuration store can deploy a suitable technology to host the read-only store that can assist with queries. It is recommended that both the source and the destination of the configuration store changes be made better suited to their purpose 

 

Configuration Automations Part 2.

This article is a continuation of the previous article on Configuration Automations and describes the syntax of the configuration templates.  

The templates can declare variables and logic that expand to actual configurations when substituted. As common to declarative syntax across declarative frameworks such as for user interface resources, infrastructure resources, and container orchestration frameworks, most begin and end with a specific delimiter.

The W3 organization specifies a template language for xml and xquery that involves a QName. Template arguments can be described in a key-value pairs where the key is a proper key name. These must be uniquely identified for their usage.

Templates can also import modules and open data files. The import modules take a namespace to avoid conflict from other modules. The data files can be opened as a string or an xml node using a built-in intrinsic. This can be used to assign the content to a variable. With the help of this variable, it is possible to browse an element by walking the hierarchy checking for each level if it exists and then finding the value corresponding to the name. A text file can be opened as a sequence of strings as well.

XPath is the language for navigating the xml hierarchy to identify a node. It uses curly braces as delimiters and those delimiters so those cannot be used inside the XPath embedded in a template. XQuery is the language for querying the nodes with the criteria specified. XQuery embedded in an attribute value of the template will call the builtin implicitly and this will return the node itself if it were an attribute that was queried or the concatenated text of the node and its descendants or if the input contains multi-nodes, their return value will be joined and delimited by white space. If the XQuery is embedded in an element text, it will keep its original result and in the case of attribute node will return the surrounding elements’ attribute while in the case of element node will become the surrounding elements’ child. As an example, a

<root>

<item name=”A” value=“1”>X</item>

<item name=”B” value=”2” >Y</item>

<item name=”C” value=”3”>Z</item>

</root> 

where xml content can be used with a template specifying an attribute as:

<xml><test result=”{root/item/@*}”</test>

and this will result in

“A 1 B 2 C 3”

And an element as:

<test result=”{root/item}” ></test>

And this will result in

<test result=”X Y Z”></test>

Templates can also include references to other entities in element texts and attribute values and there’s some difference between this format and X-Query.

One of the challenges with template syntax that is often encountered is the use of quotes for enclosing literals. Single quotes and double quotes cannot be used together in a string literal but concat operator can be used to build a string from parts that are enclosed in both.

There are extensions available that can take a template in one form and translate it into another. For example, a JSONiq extension is designed to write JSON templates naturally.

 

 

Configuration templates:

Configuration for software applications is essential to control their behavior. It includes secrets, endpoints, connection strings, certificate references and many other settings. When the same software is deployed to different environments, the values change. Multiple configurations specific to environments begin to form and teams start leaning towards a configuration store. As with all stores, a suitable service can automate the computations associated with the store and make the results available over the web, so a configuration service forms.

The variations between environments in configurations become easier to manage only when the repetitions are moved to a single template. This form of configuration splits the multiple copies of configurations into a template and values set of files. There can be many values file, but their size will be small, and they will be easier to read. The idea of templates to generate configmaps for deployment is rampant throughout the industry on both windows and Linux environments and hosting infrastructures that leverage this operating system.

The process of adding a new configuration to a deployed application sometimes involves several manual steps. After the configuration value is set, it must be deployed for consumption in every cluster, environment, and machine function. This involves running a command using a client for every cluster, environment, and machine function where the target application is deployed. The client requires a configuration template and configuration value file as inputs.

As mentioned, the configuration value file needs to be unique for every cluster, environment, and machine function combination. One way to automate it is to generate the configuration values file so that the human involvement to edit it is removed and there is little chance for typo. Tools are often used for this purpose.

While this solves a set of problems, it does not eliminate the need to run the tool many times for creating configuration values in each distinct deployment.  Consequently, some form of batching also makes its way into automations. The batch command merely repeats the invocations once for each deployment so that the human involvement to run them again and again is avoided. The output of the batch command is also easy to read with the set pattern of invocations but with a different parameter for the generated configuration values file.

This covers the deployment related automations for configuration but it does not solve the syntax and semantics associated with the templates. While Xpath for xml and similar for Json are popular ways to store data and browse them, the syntax can be used just like a programming language with repetitions using loops and counters.  Some templates provide a veritable library of builtin intrinsics and functions to use and thus make them more succinct.

Service Fabric (continued)     

Part 2 compared Paxos and Raft. Part 3 discussed SF-Ring and Part 4 discussed its architecture. This article describes how to configure Service Fabric clusters

The Service Fabric managed cluster SKU is a starting point for creating a Service Fabric cluster. There are several other ways to configure it. These include:

1.       Adding a virtual machine scale set extension to a node type - Each node type in a service fabric managed cluster is backed by a virtual machine scale set (vmss). It enables us to add vmss extensions that are small applications that provide post-deployment configuration and automation on Azure VMs. Extensions take mandatory parameters and simplify the installation.

2.       Configuring cluster availability zone spanning – Service fabric clusters can span availability zones in the same region which provide low latency network and high availability. Azure availability zones are only available in select regions.

3.       Configuring cluster network settings – Service Fabric managed clusters are created with a default network configuration which consists of a load balancer, public IP, VNet with one subnet allocated, and a NSG configured for essential cluster functionality. The settings of this network can be configured with NSG rules, RDP access, Load balancer config, IPV6, custom virtual network, custom load balancer, accelerated networking, and auxiliary subnets.

4.       Configuring a node type for large vmss – which are created by specifying the multiplePlacementGroups in added to the nodeType definition. Managed cluster node types set this property to false to keep fault and upgrade domains consistent within a placement group but it can be set for large vmss.

5.       Configuring managed identity - which are specified by a property vmManagedIdentity that has been added to node type definitions and contains a list of identities that may be used.

6.       OS and Data encryption – The disk encryption options are selected with the recommendation that the encryption be set at host. This improves on Azure disk encryption by supporting all OS types and images, including custom images.

7.       Autoscaling - which gives great elasticity and enables addition or reduction of nodes on demand on a secondary node type. The rule for the workload can be configured and autoscaling handles the rest.

8.       Scaling a node type – A service fabric managed cluster node type is scaled with portal, ARM template or PowerShell. We can also configure autoscale for a secondary node type if a fully automated solution is needed.

9.       Enabling Automatic OS image upgrades – This is a choice for the user even though the Service Fabric managed clusters manages the vmss resources.

10.   Modifying the OS SKU for a nodeType – This is helpful for scenarios such as migrating from earlier version of Windows to later or from switching one SKU to another.

11.   Configuring placement properties – These are used to ensure that certain workloads run only on certain node types in the cluster

12.   Managing the disk type SKU such that managed disks are used for all storage types so that the disk type and size are not provisioned.

13.    Configuring the cluster upgrade options such as wave deployment for automatic upgrades

This rounds up the configurations for the service fabric cluster.

Service Fabric (continued)    

Part 2 compared Paxos and Raft. Part 3 discussed SF-Ring and Part 4 discussed its architecture. This article describes monitoring and diagnostics.

Service Fabric provides monitoring at various levels – Application Monitoring, Cluster Monitoring, Infrastructure monitoring etc.

Application monitoring allows us to study the performance of features and components of an application. The responsibility of application monitoring is on the users developing an application and its services. It answers questions on how much traffic is flowing to the application, the success of the calls made to the services, the actions taken by the users on the application, if the application is throwing unhandled exceptions and whether the services are running fine within their containers.

Service Fabric makes applications resilient to hardware failures. Failures are rapidly detected, and workloads go through failover to other nodes. Cluster monitoring is critical to this end. There are a comprehensive set of events out of the box. These events can be accessed through the event store or the operational channel. Service Fabric events are available from a single ETW provider with a set of relevant filters to separate them. EventStore provides events available in the Service Fabric Explorer and through the REST API.

These events illustrate activities taken by the platform on different entities such as Nodes, Applications, Services, Partitions, etc. If a node were to go down, the platform would emit a NodeDown event and a tool of choice would use that event to generate notifications. These events are available both on Windows and Linux clusters.

The platform includes a health model which provides extensible health reporting for the status of the entities in a cluster. Each node, application, service, platform, replica or instance has a continuously updatable health status. Whenever the health of a particular entity transitions, a corresponding event would also be emitted. Queries and alerts can be setup on a dashboard with these events.

Users can also override heath for entities. If the application is going through an upgrade and the validation tests were failing, then the events can be written to the Service Fabric Health using the Health API to indicate that the application is no longer healthy and the Service Fabric will automatically rollback the upgrade.

Watchdogs are available as a separate service that watches health and load across the services, pings endpoints and reports unexpected health events in the cluster. This can help prevent errors that may not be detected and are based only on the performance of a single service. Watchdogs are also a good place to host code that performs remedial actions that don’t require user involvement such as archiving older log entries

Infrastructure monitoring is also referred to as performance monitoring since it pertains to system performance and depends on many factors. These are typically measured through performance counters. They can come from a variety of sources including the operating system, the .Net framework, or the service fabric platform itself. Performance counters are also available for Reliable Services and Actors programming models.

Application Insights is used for application monitoring, Diagnostic agent is used for cluster monitoring and Azure monitor logs is used for infrastructure monitoring.

 

Service Fabric Summary continued Part 3. 

Part 1 of Service Fabric Summary focuses on the principle and the purpose behind the design decisions in Service Fabric. Part 2 focuses on usage.  Part 3 in this article focuses on efficiency.

Service Fabric provides an infrastructure to build, deploy, and upgrade microservices efficiently with options for auto scaling, managing state, monitoring health, and restarting services in case of failure. It helps developers and administrators to focus on the implementation of workloads that are scalable, reliable, and manageable by avoiding the issues that are regularly caused by complex infrastructures. The major benefits it provides include deploying and evolving services at very low cost and high velocity, lowering costs to changing business requirements, exploiting the widespread skills of developers, and decoupling packaged applications from workflows and user interactions.   

 

Service Fabric follows an application model where an application is a collection of microservices. The application is described in an application manifest file that defines the different types of service contained in that application, and pointers to the independent service packages. The application package also usually contains parameters that serve as overrides for certain settings used by the services. Each service package has a manifest file that describes the physical files and folders that are necessary to run that service, including binaries, configuration files, and read-only data for that service. Services and applications are independently versioned and upgradable. 

 

A package can deploy more than one application but if one service fails to upgrade, the entire application is rolled back. For this reason, the microservices architecture is best served by multiple packages. If a set of services share the same resources and configuration or have the same lifecycle, then those services can be placed in the same application type. 

Service Fabric programming models can be chosen whether the services are stateful or stateless. 

 

A stateless service is chosen when it must scale, and the data or state can be stored externally.  There is also an option to run an existing service as a guest executable and it can be packaged in a container will all its dependencies. Service Fabric models are both container and executable as stateless services.  

 

Service Fabric is also useful for scenarios that require low-latency reads and writes, such as in online gaming or instant messaging. Applications can be built to be interactive and stateful without having to create a separate store or cache. Gaming and instant messaging are some examples of this scenario. 

Applications that must reliably process events or streams of data run well on Service Fabric with its optimized reads and writes. Service Fabric supports application processing pipelines, where results must be reliable and passed on to the next processing stage without any loss. These pipelines include transactional and financial systems, where data consistency and computation guarantees are essential. 

Stateful applications that perform intensive data computation and require the colocation of processing (computation) and data in applications benefit from Service Fabric as well. Stateful Service Fabric services eliminate that latency, enabling more optimized reads and writes. As an example, real-time recommendation selections for customers that require a round trip-time latency of less than hundred milliseconds are handled with ease. 

Service Fabric also supports highly available services and provides fast failover by creating multiple secondary service replicas. If a node, process, or individual service goes down due to hardware or other failure, one of the secondary replicas is promoted to a primary replica with minimal loss of service. 

Individual services can also be partitioned where services are hosted by different hosts. Individual services can also be created and removed on the fly. Services can be scaled from a few instances on a few nodes to thousands of instances on several nodes and dialed down as well. Service Fabric helps with the complete life cycles. 

 

 

Capacity and Scaling are two different considerations for Service Fabric and must be reviewed individually. Cluster capacity considerations include Key considerations include initial number and properties of cluster node types, durability level of each node type, which determines Service Fabric VM privileges within Azure infrastructure, and reliability level of the cluster, which determines the stability of Service Fabric system services and overall cluster function 

A cluster requires a node type. A node type defines the size, number, and properties for a set of nodes (virtual machines) in the cluster. Every node type that is defined in a Service Fabric cluster maps to a virtual machine scale set aka VMSS. A primary node type is reserved to run critical system services. Non-primary node types are used for backend and frontend services. 

Node type planning considerations depend on whether the application has multiple services or if they have different infrastructure needs such as greater RAM or higher CPU cycles or if any of the application services need to scale out beyond hundred nodes or if the cluster spans availability zones. 

The durability characteristics of the cluster are offered as durability tiers. A durability level designates the privileges that Service Fabric VMs have with the underlying Azure architecture which can allow override of infrastructure requests such as reboot, reimage or migration that impacts the quorum requirements. 

Scalability considerations depend on whether scaling is required for the number of nodes of each node type or if it is required for services. 

Migration plan for a cloud service to Service Fabric:

 

The preparation for the migration involves the following:

1)      Defining the Service Fabric cluster: The planning for a Service Fabric cluster will culminate with an Azure Resource Manager (ARM) template that defines the node types and their numbers.  Sample template for creating a Service Fabric cluster is included here.

2)      Developer workstation requirements: The workstation requirements are important for application development. These include Docker for windows to containerize and test the applications prior to deployment, Visual Studio Docker support for Docker compatibility, and compute resources when workstations do not have 8GB of memory. Azure DevTest Labs service is an option for such compute.

3)      Networking requirements: Service Fabric deployments for large enterprises require careful planning for the following:

a.       Reachability over VPN: Enterprises extend their corporate network with a private address space to an Azure subscription. ExpressRoute or Site-to-Site VPN provides secure on-premises connectivity and must have corresponding resources instantiated for viewing and managing via the Azure Portal.

b.       NSG rules: Enterprises must secure their inbound and outbound network traffic through third-party firewall appliances or with Azure Network Security Group rules or both.

c.       Address Space: The Service Fabric cluster has tight control over address space requirements and subnets. These will also be described under the placement constraints.

4)      Containerizing existing Windows applications: Docker images can run an operating system, an IIS server and the windows application to be migrated to Service Fabric. Proper choice of base line image, Docker layers, customizations and hardening are required to secure the containerized image. A sample Dockerfile has been included here for this purpose. The base image could be a locked-down custom enterprise image that enforces enterprise requirements but often it is an official signed image from a public image repository and as an alpine version at that.

5)      Cluster configuration – node types:  A Service Fabric cluster may have one or mode nodeTypes which loosely map to an Azure Virtual Machine Scale Set (VMSS) each. A VMSS is a set of Virtual Machines (VM)s with the same set of properties. Each cluster must have at least one node type-primary node type, which runs the Service Fabric system services. The application containers cannot be run on the primary node type. The sizing of the primary node type is directly proportional to the size of the cluster. T-shirt sizing of Service Fabric cluster is available and can be referenced via monikers such as Standard_D2s_v3. Application and Data intensive compute can form the other node types. Choice of node types can be determined based on the application that is being migrated. If the Service Fabric cluster starts out with being dedicated to a single application, it can have just two node types – primary and application. Each scale set can be scaled in or out independently and does not require extensive planning and testing because the actual size can grow or shrink driven by monitoring and metrics.