Cost comparisons between standalone products and cloud native solutions

 

Introduction:

This article is a TCO calculator for a comparison of cost between an isolated storage appliance and one native to public cloud computing

Description:

Many datacenter products are sold as separate isolated standalone appliances which start out as lean and mean to fit on a single host and eventually justify their own expansion to several racks. The backend processing for many IT operations is delegated to these appliances. For example, object storage is one such example where each organization can choose to have a private cloud storage.

This is a comparison of the features and their relative price comparisons as low or high:

Feature/Subsystem	Standalone appliance	Cloud native DIY solution
Organization	Multi-layered and multi-component monolithic application which requires significant bare metal libraries – High	This is staged and pipelined execution including several pre-built Azure resources - Low
Cluster based architecture for scale out	Involves deploying specific types of components to control and data nodes with costs for coordinator – High	State based reconciliation of control plane resources including scale out and replicas – Low
Microservices for each of the components for ease of integration, testing and programmability	Each component targets the same core storage layer which if distributed between clusters relies on message-based consistency algorithms. Depending on code organization, maintenance and individual component health, the costs for shipping releases of software are accumulated over timeframes. High	Each service can be included into an app service and a plan while components are replaced by efficient use of resources. Packing, unpacking multi-layer blob and user-access-resolution independent layers are replaced by pipelined services that add minimal code to existing resources. Message broker, passing, pub-sub and other routines are eliminated in favor of dedicated products like service bus while the algorithm remains the same. Code reduction and independent release results in cost savings- Low
Since the user namespace hierarchy, user object management, web user interface and virtual data centers are implemented independently as layers, the flexibility to provide business functionalities can remain shallow and restricted to upper layers or frontend	Behind the scenes, the system architecture facilitates the changes to be restricted to frontend or middle tier including data access. Most features can be added in a single shot feature delivery. But the cost often includes metadata changes that might also be persisted to the store. Most features that require persistence reuse the store. High	Behind the staged pipeline and region-based storage accounts, the feature implementations do not rely on anything more than a message queue and a database. Custom logic can be added via extensions and functions that are easy to add without impacting the rest of the organization. Low
DIY libraries and code	Significant investment – High	Little or no investment – leveraging available resources- Low
Objects owned by a virtual data center within a replication group will need to be replicated.	Code must be written to replicate readable objects from one virtual data center to another. Three nodes might be chosen from a pool of cluster nodes for the writes. For example: the storage engine records the disk locations of the chunk in a chunk location index and the disk locations corresponding to the chunk are written to three different disks/nodes. The index locations are chosen independently from the object chunk locations. The VDC needs to know the location of the object. Directories such as for location of objects might be designated for different purposes.  Cost: High	Syncing across availability zones is built into the Azure resources. Although this might not be exposed to the resource invokers, they are welcome to create regions for read-write and read-only. Cosmos DB for instance supports automatic replication across regions. If a storage engine layer must be written on top of the cloud resources, it may still have to write its own replication but usages involving existing data stores can leverage an Azure store, cache or CDN with automatic replication. Cost: Low
Query execution engine	A storage engine could have standard query operators for the query language if the entire data were to be considered as enumerable. In order to collapse the enumeration, efficient lookup data structures such as Bplus tree are used. These indexes can be saved right in the storage for enabling faster lookup later. Cost: High	Unlike preparation, resolving, compilation, plan creation, plan optimization and caching of plans, objects and their heuristics, the cloud services provide simpler indexing and searching capabilities that transcend even document types let alone documents. Besides the operational advantages of using these services from the cloud, this simplifies the search experience. Cost: Low
Analysis engine The reporting stack has always been a read-only stack which made it possible to interchange analysis stacks independent from the strict or eventually consistent writes.	A storage engine with its own reporting stack is a significant investment for that product even if the query interfaces are exposed as standard query operators Cost: High	Many analytical stacks can easily connect to the storage via existing and available connectors reducing the need for integration. Services for analysis from the public cloud are rich, robust and very flexible to work with. Cost: Low

 

Conclusion:

The use of a TCO calculator realizes the reimagining of a storage appliance built for the cloud so that the footprint on premises of individual organizations is minimized.

 

This is a continuation of an article that describes operational considerations for hosting solutions on Azure public cloud.

This article focuses on support for compute in Azure.

Compute requirement of a modern cloud app typically involves load balanced compute nodes that operate together with control nodes and databases.

VM Scale sets provide scale, customization, availability, low cost and elasticity.

VM scale sets in Azure resource manager have a type and a capacity. App deployment allow VM extension updates just like OS updates.

Container infrastructure layering allows even more scale because it virtualizes the operating system. While traditional virtual machines enable hardware virtualization and hyper V’s allow isolation plus performance, containers are cheap and barely anything more than just applications.

Azure container service serves both Linux and windows container services. It has standard docker tooling and API support with streamlined provisioning of DCOS and Docker swarm.

Azure is an open cloud because it supports open-source infrastructure tools such as Linux, ubuntu, docker, etc. layered with databases and middleware such as hadoop, redis, mysql etc., app framework and tools such as NodeJS, java, python etc., applications such as Joomla, drupal etc and management applications such as chef, puppet, etc. and finally with devops tools such as jenkins, Gradle, Xamarin etc.

Job based computations use larger sets of resources such as with compute pools that involve automatic scaling and regional coverage with automatic recovery of failed tasks and input/output handling.

Azure involves a lot of fine grained loosely coupled micro services using HTTP listener, Page content, authentication, usage analytic, order management, reporting, product inventory and customer databases.

 

Efficient Docker image deployment for intermittent low bandwidth connectivity scenarios requires the elimination of docker pulling of images. An alternative deployment mechanism can compensate for the restrictions by utilizing an Azure Container Registry, Signature Files, a file share, an IOT hub for pushing manifest to devices. The Deployment path involves pushing image to device which is containerized. The devices can send back messages which are collected in a device-image register. An image is a collection of layers where each layer represents a set of file-system differences and stored merely as folders and files. A SQL database can be used to track the state of what is occurring on the target devices and the Azure based deployment services which helps with both during and after the deployment process.

 

Resource groups are created to group resources that share the same lifecycle. They have no bearing on the cost management of resources other than to help with querying. They can be used with tags to narrow down the interest. There is metadata stored about the resources and it is stored in a particular region. Resources can be moved from one resource group to another or even to another subscription. Finally, resource groups can be locked to prevent actions such as delete or write by users who have access.

As with its applicability to many deployments, Azure Monitor provides tremendous insight into operations of Azure Resources. It is always recommended to create multiple application insights resources and usually one per environment. This results in better separation of telemetry, alerts, work items, configurations, and permissions. Limits are spread such as web test count, throttling, data allowance etc. and it also helps with cross-resource queries.

 

 

 

This post is a continuation of a series of articles on Azure such as this one: https://1drv.ms/w/s!Ashlm-Nw-wnWhKdsT81vgXlVl38AfA?e=cZz5jq . This describes the Azure architecture for startups:

Azure saves significant costs for big businesses, but it is a compelling value provider for startups as well. The core startup stack architecture makes use of factors that differentiate the requirements for startups. These refer to speed, cost and options where the business needs change rapidly and a sound architecture and system design can minimize the impact to the system while incrementally building the solutions.

Many startups start with a single monolithic application because they don’t have the requirements mature enough to warrant complex microservices pattern. This is best served by an architecture that involves an Azure App Service to provide a simple App Server to deploy scalable applications without configuring servers, load balancers or other infrastructure, an Azure database such as PostgreSQL for RDBS storage without the hassle for maintenance and performance tuning, Azure Virtual Network to segment virtual network traffic and keep internal services protected from internet threat, a GitHub Actions which helps build a continuous Integration and continuous deployment pipeline, a blob storage for unstructured data, a CDN or content delivery network for distributing data throughout the global network with reduced latency and Azure monitor that analyzes happenings across the applications infrastructure.

The core startup stack components are layered minimally such that the product can get off the ground and into the hands of the customers. 80 percent of the startups will use a stack comprising of a storage or persistence layer, a compute or static content layer and the CDN to distribute it to different clients.

With few customers at the start, a CDN might seem premature but adding a CDN serves two-fold in terms of avoiding significant costs to retrofit later on and providing a façade behind which api and architecture can be refined.

The AppServer is where the code runs. This platform should make deployments easy, while requiring the least possible operational input. The app server should scale horizontally. With the help of PaaS, challenges concerning the traditional use of bare-metal, web server, virtual machines etc. are avoided.

Static content does not have to reside on the app server.  The proper use of a CI/CD pipeline can build and deploy static assets with each release. Most production web frameworks deploy static assets.

When the app is running, it needs to store data in a database from online transactions processing. Using the managed instance of a relational database reduces the operational overhead and improves app optimizations.

Log aggregation is extremely useful to debugging and troubleshooting. Frequent integration avoids divergent code bases that lead to merge conflicts.

Leveraging the GitHub Actions, many of the compliance, regulatory, privacy and bounty program processes can be automated. These automations can similarly be expanded to relieve pain points for the startup during the initial growth phase.

 

 

Planning for onboarding a financial calculator service to Azure public cloud:

 

Problem statement: This article leverages an Azure industry cloud example to onboard a FinTech Service to the Azure public cloud. While there are many examples from the Azure industry clouds and verticals, this article is specifically for onboarding a financial calculator service that acts a broker between external producers and consumers. With this case study, we attempt to do a dry run of the principles learned from the Azure Cloud adoption framework and make the migration in the most efficient, reliable, available and cost-effective manner.

Article: Onboarding a service such as a financial calculator in the Azure public cloud is all about improving its deployment, using the proper subscription, planning for capacity and demand, optimizing the Azure resources, monitoring the service health, setting up management group, access control, security and privacy of the services and setting up the pricing controls and the support options. We look at these in more detail now.

Proper subscription: Many of the rate limits, quotas and the availability of services are quite sufficient in the very first tier of subscription. The Azure management console has a specific set of options to determine the scale required for the service.

Resource and Resource groups: The allocation of a resource group, identity and access control is certainly a requirement for the onboarding of a service. It is equally important to use the pricing calculator and TCO calculator in the Azure public cloud to determine the costs. Some back of the envelope calculation in terms of the bytes per request, number of requests per second, the latency, recovery time, recovery point, MTTR, MTBF help with determining the requirements and the resource management.

Optimizing the Azure resources: This is automated. If we are deploying a python Django application and a node.js frontend application, then it is important to make use of api gateway, load balancer, proxy and scalability options, certificates, domain name resources etc. The use of resources specific to the service as well as those that enhance its abilities must be methodically ruled off from the checklist that one can draw from the Azure management portal.

Monitoring the service health: Metrics specific to the financial calculator service in terms of the size of the data processed, the mode of delivery, the number of requests submitted to the system, the load on the service in terms of the distribution statistics and other such metrics will help determine if the service requires additional resources or when something goes wrong. Alerts can be set up for the thresholds so we can remain passive until we get an alert.

Management group, Identity and access control: Even if there is only one person in charge of the service, the setting up of a management group, user and access control formalizes and detaches that person so that anyone else can take on the administrator role.  This option will also help set up registrations and notifications to that account so that it is easier to pass the responsibility around.

Security and privacy: The financial calculator service happens to be a stateless transparent financial proxy which does not retain any data from the customer, so it does not need any further actions towards security and privacy. TLS setup on the service and use of proper certificates along with domain names will help keep it compute resource independent.

 Advisor: Azure has an advisor capability that advises on the efficiencies possible with the deployment after the above-mentioned steps have been taken. This helps in streamlining the operations and reducing cost.

Conclusion: The service onboarding feature is critical towards the proper functioning of the service both in terms of its cost and benefits. When the public cloud knowledge center articles are followed up meticulously for the use of the Azure management portal in the deployment of the service, the service is guaranteed to improve its return on investment.

 

 

 

 This is a continuation from the previous post.

1.       Adoption:  The cloud adoption plan at enterprise scale warrants significant investments in the creation of a new business logic. A migration plan moves those workloads to the cloud with the following three approaches: lift and shift, lift and optimize, or modernize. The migration scenarios, best practices and process improvements come with sufficient literature. Another area of emphasis is innovation. Unlike migration, this can provide the greatest business value by unlocking new technical skills and expanded business capabilities.

2.       Govern: The Governance in the Microsoft Cloud Adoption Framework for Azure is an iterative process. As cloud estates change over time, so do the cloud governance processes and policies especially for an organization that has been heavily invested in on-premises infrastructure. With the ability to create an entire virtual data center with a few lines of code, the paradigm left shifts in favor of automation. The governance benchmark tool goes a long way in realizing this vision.

3.       Manage: Cloud Management delivers strategy using planning, readiness and adoption and drives the digital assets towards tangible business outcomes. This form of management requires articulated business commitments, a management baseline, its subsequent expansion, and advanced operations and design principles.

4.       Secure:  The security in the Microsoft cloud adoption framework is a journey. It involves incremental progress and maturity and does not have a static destination.  Its end state can be envisioned, and this provides guidance for periodic assessments, alignments and definite results. Organizations like the NIST, The Open Group, and the Center for Internet Security provide standards to this effect. Establishing the security roles and responsibilities helps with resources for this deeply technical discipline.

5.       Organize: Cloud adoption cannot happen without well-organized people. Successful adoption is the result of properly skilled people doing the appropriate types of work. The approach to establish and maintain the proper organizational structure involves a. defining the type, b. cloud functions that adopt and operate the cloud, c. defining the teams that can provided various cloud functions and d. coming up with the Responsible, Accountable, Consulted and Informed (RACI) matrix.

6.       Resources: The public cloud comes with several tools and templates for each of the above stages such as cloud journey tracker, strategy and plan template, readiness checklist, governance benchmark assessment, migration discovery checklist, solution accelerators, operations management workbook, RACI diagram, and such others.

Conclusion:  The cloud adoption framework lays the roadmap for a successful cloud adoption journey.

 

 

This article is a continuation of a series of articles on Azure Cloud architecture, design patterns, technology choices and migration guidance. Specifically, it discusses Cloud Adoption Framework. This framework discusses the Cloud Adoption Framework.

The cloud adoption framework brings it together for the cloud adoption journey. It involves:

1.       Getting Started:  A great starting point aligns with an existing scenario for which the starting guides have already been published.  For example, an organization trying to accomplish a certain goal might want to choose the cloud adoption scenario that best supports their strategy, examine antipatterns across methodologies and their solutions, align foundational concepts to onboard a person, project, or team, adopt the cloud to deliver business and technical outcomes sooner, improve controls to ensure proper operations of the cloud or establish teams to support adoption and operations. Specific cloud adoption scenarios include hybrid and multi-cloud adoptions, modern application platform scenario, SAP adoption scenario, desktop virtualization, and cloud adoption for the retail industry. Cloud adoption requires technical change, but it is never restricted to IT. Other teams might want to migrate existing workloads to the cloud, build new products and services in the cloud, or might want to be unblocked by environment design and configuration. A solid operating model improves controls. The Azure Advisor fits this role.

2.       Strategy: A cloud adoption strategy describes it to the cloud technicians and makes a case to the business stakeholders. This is efficiently done only when it encompasses the following a. the motivations are defined and documented, b. the business outcomes are documented, c. the financial considerations are evaluated, and d. the technical considerations are understood. There is a published strategy-and-plan-template that builds out the cloud adoption strategy and it helps to capture the output of each of these steps.

3.       Plan: The cloud adoption framework must come with a plan that translates the goals from the strategy document to something that can be executed in the field. The collective cloud teams must put together Specific-Measurable-Achievable-Relevant-and-Timebound action items that capture the prioritized tasks to drive adoption efforts and maps to the metrics and motivations defined in the cloud adoption framework. This includes a. digital estate, b. initial organizational alignment, c. skills readiness plan, and d. cloud adoption plan.

4.       Readiness: Before a plan is enacted, some preparation is required. In this case, the following exercises guide us through the process of creating a landing zone to support cloud adoption. These include: a setup guide that familiarizes tools and processes involved, a landing zone that establishes code based starting point for the infrastructure and environment, its subsequent expansion to meet the platform requirements specified in the plan and finally, the best practices, that validates landing zone modifications against the best practices to ensure the best configurations.

 

This is a continuation of the article introduced here: https://1drv.ms/w/s!Ashlm-Nw-wnWhKdsT81vgXlVl38AfA?e=qzxyuJ. Specifically, it discusses the technology choices from the Application Architecture guide for Azure Public Cloud.

1.       Choosing a candidate service:

a.       if full control of the compute is required, a virtual machine or its scaleset is appropriate.

b.       If it has HPC workload, Azure Batch is helpful.

c.       If it has a microservice architecture, it has an Azure App Service.

d.       If it has an event-driven workload with short-lived processes. Azure functions suit the task.

e.       If a full-fledged orchestration is required, Azure Container Instances are helpful.

f.        If a managed service is needed with .Net framework, Azure Service Fabric is helpful.

g.       If Spring boot applications are required, Azure Spring Cloud is helpful.

h.       If RedHat Openshift is required, Azure RedHat Openshift is dedicated to this purpose.

i.         If a managed infrastructure is required, Azure Kubernetes Service does the job.

There are two ways to migrate on-premises compute to the cloud:

The first involves the ‘lift and shift’ pattern which is a strategy for migrating a workload to the cloud without the redesigning ans is also called ‘rehosting’ pattern.

The second involves refactoring an application to take advantage of the cloud native features.

2.       Microservices architecture can have further options for deployment.  There are two approaches here:  The first involves a service orchestrator that manages services running on dedicated nodes (VMs) and the second involves a serverless architecture using Functions-as-a-service (FaaS). When Microservices are deployed as binary executables aka Reliable Services, the Reliable Services Programming Model makes use of Service Fabric Programming APIs to query system, report health, receive notifications on configuration and code changes, and discover other services. This is tremendously advantageous for building stateful services using so called Reliable Collections. AKS and Mesophere provide alternative infrastructures for deployment.

3.       The Kubernetes at the edge compute option is excellent to keep operational costs low, easily configure and deploy a cluster, find flexibility with existing infrastructure at the edge, and for running a mixed node cluster with both Linux and Windows nodes. Out of the three options for leveraging Kubernetes with Baremetal, K8s on Azure Stack edge and AKS on HCI, the last option is the easiest to work with.

4.        Choosing an identity service involves comparing the options for self-managed Active Directory services, Azure Active Directory, and managed Azure Active Directory Domain Services. Azure Active Directory does away with one’s own directory services and leveraging the cloud provided one instead.

5.       Choosing a data store is not always easy. The term polyglot persistence is used to describe solutions that use a mix of data store technologies. Therefore, it is important to understand the main storage models and their tradeoffs.

6.       Choosing an analytical data store is not always about Big Data or lambda architectures with its incremental data processing speed serving layer and batch processing layer, although they both require it. The driving force varies on a case-by-case basis.

7.       Similarly, AI/ML services can be leveraged directly from the Azure Cognitive Services portfolio, but its applicability varies on a case-by-case basis as well.