This is a continuation of the articles on Azure Data Platform and discusses Data Lakes:

Access Control Model:

This section covers the access control model in Azure Data Lake Storage Gen2 which supports the following mechanisms: shared key authorization, shared access signature authorization aka SAS, role-based access control aka RBAC, attribute-based access control aka ABAC and access control lists aka ACL. Out of these the RBAC and ACLs on one hand and shared-key and SAS on the other hand are complimentary. The former has no effect on the latter. Shared-key and shared access both grant access to a user without the need for an identity. RBAC grants coarse-grained access to users such as for read-write of all data.  ABAC refines RBAC role assignments by adding conditions. ACLs grant fine-grained access such as a write access to a specific directory or file. The security principals recognized by the Azure are a user, group, service principal or managed identity  that is defined in the Azure Active Directory. A permission set grants coarse level access such as a read or write access to all the data in a storage account or all the data in a container. Permission sets are granted based on the roles and some well-known roles are Owner, Contributor, Reader, and Storage Account Contributor. The first three can access the data and all four can manage the storage account. They cannot grant access to other security principals but they can provide shared-key and shared access signatures except for the reader role. The order of resolving access grant or denial is RBAC first, ABAC next and ACLs last and applies to all operations such as list, get, create, update, and delete. Security groups are particularly useful to add ACLs. For example, if Azure Data Factory aka ADF ingests data into a folder named /LogData and specific service engineering team upload data in a container and various users analyze the data, then we create a LogsWriter group and LogsReader group to enable these activities. The POSIX permissions assigned to the directory will show rwx permissions assigned the LogWriter group and r-x permissions assigned to the LogReader group. The service principal or the managed service identity aka MSI for ADF will be part of the LogWriter group but the service principal or MSI for Databricks will be part of the LogsReader group. Groups facilitate addition or removal of members without disturbing the assignments. They also help to avoid exceeding the maximum number of role assignments per subscription and the maximum number of ACL entries per file or directory. The limits are 4000 Azure role assignments in a subscription and 32 ACL entries per file or directory.

Premium tier:

ADLS Gen2 supports premium tier and premium block blob storage accounts that are ideal for Big Data analytics and workloads that require low latency and support a high number of operations such as in the case of machine learning. The premium tier supports hierarchical namespace that accelerates big data analytics workloads and enables file-level access control lists. It also supports Azure Blob file system drive for Hadoop.

 The topic discussed here are some key considerations for the use of Azure Data Lake Storage Gen2 which is a highly scalable and cost-effective data lake solution for big data analytics.

The first and foremost consideration is that a data lake is not appropriate where a data warehouse is suitable.  They are complimentary solutions that work together to help us derive key insights from our data. A data lake is a store for all types of data from various sources besides the unstructured versus structured data differentiation.  A retail company can store the past 5 years’ worth of sales data in a data lake, and in addition they can process, and they can process data from social media to extract the new trends in consumption and intelligence from retail analytics solutions and then generate a highly structured data set that is suitable to store in the warehouse. ADLS Gen2 can offer faster performance and Hadoop compatible access with the hierarchical namespace, lower cost, and security with fine grained access controls and native AAD integration. It can become a hyperscale repository that can store up to petabytes of data and hundreds of Gbps in throughput.

Other considerations include what will the data lake store and for how long, what portion of the data is used in analytical workloads, who needs access to what parts of the data lakes, what are the various analytical workloads and what am I going to run on the data lake, what are the types of workloads that will use the data, what are the transaction patterns and the analytics patterns and what is the budget I’m working with.

Organization and managing of data in the data lake is also a key concern for this single data store. Some users of this store  can claim end-to-end ownership of the pipeline and others have a central team that manages, operates and governs the data lake. This calls for different topologies of the data lake such as centralized or federated data lake strategies, implementing with single or multiple storage accounts, and with globally shared or regionally specific footprints.

When the customers of the data lake are both internal and external, the scenarios may be subject to different requirements from security and compliance perspectives, query patterns, access requirements and cost and billing. Storage accounts can be created in different subscriptions for development and production environments and subject to different SLAs. The account boundaries determine the management of logical sets of data in a unified or isolated manner. Subscription limits and quotas apply to resources used with the Azure data lake such as the VM cores and ADF instances. Managed identities and SPNs must have different privileges than those who are merely reading the data.

Backup and archive is done with the help of scripts that create action and filter objects to apply blob tiering to block blobs matching a certain criteria and storage accounts must be provisioned in a brand-new resource group and storage account based upon input variables.

 

 

Data Lake is often used with Azure Data Factory to support Copy operations. It allows for downloading of files and folders directly from the Lake and to the order of thousands of files from folders, but the upload is best done with the help of Azure Data Factory which supports even upload of zip file and the in-transit extraction of files before they are stored in the Data Lake. ADF happens to support a variety of file formats such as Avro, binary and text to name a few and uses all available throughput by performing as many reads and writes in parallel as possible.

Data Lakes work best for partition pruning of time-series data which improves performance by reading only a subset of data.  The pipelines that ingest time-series data often place their files with a structured naming such as /DataSet/YYYY/MM/DD/HH/mm/datafile_YYYY_MM_DD_HH_mm.tsv.

The Hitchhiker’s Guide to Data Lake from Azure recommends that monitoring be set up for effective operations management of this cloud resource. This will help to make sure that it is available for use by any workloads which consume data contained within it. Key considerations include auditing the data lake in terms of frequent operations, having visibility into key performance indicators such as operations with high latency and understanding common errors.  All of the telemetry will be available via Azure Storage Logs which is easy to query with Kusto query language.

Common queries are also candidates for reporting via dashboards for Azure Monitor resource. For example,

1.       Frequent operations can be queried with:

StorageBlobLogs

| where TimeGenerated > ago(3d)

| summarize count() by OperationName

| sort by count_ desc

| render piechart

 

2. High latency operations can be queried with:

StorageBlobLogs

| where TimeGenerated > ago(3d)

| top 10 by DurationMs desc

| project TimeGenerated, OperationName, DurationMs, ServerLatencyMs, ClientLatencyMs = DurationMs – ServerLatencyMs

 

3. Operations causing the most error are caused by:

  StorageBlobLogs

| where TimeGenerated > ago(3d) and StatusText !contains "Success"

| summarize count() by OperationName

| top 10 by count_ desc

 

and these can be reported on the dashboard.

The differentiation between a Healthcare Insurance Industry Cloud and a general purpose cloud solution for health-care data.

 Introduction: This article discusses whether Healthcare and InsuranceTech require different industry clouds and whether they can be directly hosted as cloud solutions. Indeed, the cloud has all the resources that allows any infrastructure to be created for any company and purpose regardless of size. This begs the question of what’s special about Healthcare & InsuranceTech.

First, there is Fast Healthcare Interoperability Resources (FHIR) which is a healthcare data standard and an information network that lets us link data across systems and a communication network that lets us exchange data between systems. Healthcare IT Systems often don’t share the same data models. In fact, as more data becomes digitized, incompatibilities and resolving them becomes more expensive and time-consuming.

Second, there is the Health Insurance Portability and Accountability Act of 1996 and associated laws that establish requirements for the use, disclosure, and safeguarding of protected health information (PHI). HIPAA applies to doctor’s offices, hospitals, health insurers, and other healthcare companies – that create, receive, maintain, transmit or access PHI. HIPAA and Health Information Technology for Economic and Clinical Health Acts include rules for 1. Privacy for safeguards to protect PHI without authorization and 2. Security for administrative, technical, and physical safeguards and 3. a Breach Notification for whenever there is a breach of unsecured PHI occurs.

Third, there is the consent management that involves managing data related to consent and privacy across configuration management for the consent store, data related to the permissions granted by the users and managed resources that include user data mappings and data related to the resources in the form of attributes.

Lastly, there must be an overt display of security and compliance controls that span both observability and security controls. As an example of compliance, the Cybersecurity Maturity Model Certification attempts to prevent the theft of intellectual property and sensitive information from all industrial sectors due to malicious cyber activity. FedRamp High and FedRamp moderate both pertain to account management, monitoring, and role-based access controls and have different impact levels. HIPAA HITRUST 9.2 targets both privilege management and role-based access control. 

The difference between Healthcare and InsuranceTech can be compared to technologies that are customer-facing and backend processing.  On one hand, the customer facing data must be aggregated from various sources and a service or cloud greatly abstracts and simplifies this handling and on the other hand, insurance companies increase the competition by introducing various innovations, analysis and ultimately new options when processing these data. Many industry clouds from different cloud providers have agreed on the need for a dedicated API that can be consumed from various providers, clients and their devices. Insurance IT companies seek to develop new business capabilities with rapid application development and new machine learning models that mandate the need for dedicated pipelines. Even the cloud resources these two use, differ in their purpose. Healthcare benefits from FHIR API and the InsuranceTech prefers data lakes and pipelines.

 Some essentials for an integration pipeline include: 

1. Data Ingestion: data passed to the blob storage and the metadata queued in a job database for downstream processing. 

2. 
   

2. Central components:  

1. messaging queue: all components interact with these which allows scalability.  

2. config store: enables stable pipeline with configuration driven variability 

3. job store: keeps track of various job executions 

4. 
   

3. Job poller/scheduler: this picks up and drops a message in the message queue, for example, Kafka. 

4. 
   

4. Consumers: these must be horizontally scalable where the data is staged in tenant specific database. 

5. 
   

5. Databases: these must be horizontally scalable where the data is transformed and stored in suitable multi-tenant databases. 

6. 
   

6. Data warehouses/ data lake: the data must be denormalized with dimensions along say tenants to support multitenant data sources. 

7. 
   

7. Analytics stack: The Data Lake/warehouse is the source for the analysis stack and preferably U-SQL based for leveraging existing skills 

8. 
   

8. ML stack: also leverages the data lake/warehouse but with emphasis on separation of training and test datasets and the execution and feedback loop for a model. 

9. 
   

9. Monitoring, performance, and security: these include telemetry, auditing, security and compliance, aging and lifecycle management. 

 The Design Security for data:

One of the ways to secure data is to control access.  These involve generating access keys, shared access signatures, and Azure AD access. Access keys provide administrative access to an entire resource such as   storage accounts. Microsoft recommends these are only used for administrative purposes. Shared access signatures are like tokens that can be used to generate granular access to resources within a storage account. Access is provided to whoever or whatever has this signature. Role based access control can be used to control access to both the management and data layer. Shared access signatures can be generated to be used in association with an Azure AD identity, instead of being created with a storage account access key.  Access to files from domain joined devices are secured using the Azure AD identities that can be used for authentication and authorization. 

Another way to secure the data is to protect the data using storage encryption, Azure Disk Encryption, and Immutable storage. Storage encryption involves server-side encryption for data at rest and Transport Layer Security based encryption for data in transit. The default encryption involves an Account Encryption Key which is Microsoft managed but security can be extended using customer-managed keys which are usually stored in a Key Vault. The volume encryption encrypts the boot OS and data volumes to further protect the data.

Some controls are available from the databases and data storage products. For example, a database inside a SQL Server is secured with the help of a SQL login.  This SQL login is created inside the master database and then it is mapped to a database user associated with the database to be protected and then the data can be accessed with this credential. Every database will require to create its own mapping for that same SQL login. If the authentication is to be based on Azure Active Directory, then the mechanism varies slightly between a cloud SQL managed instances and a managed SQL database. Both of them require Azure AD-based administrator for the product and then the user accounts to work with the databases. When working with SQL managed instances, we give the read access to the Azure AD tenant to the admin account, map the user accounts to SQL logins in the master database similar to the traditional SQL Server approach and then create database users that is mapped one-on-one to the SQL logins. When working on the managed SQL database, all we need is the database user mapped to the Azure identity and so the entries in the master database and the read access to the admin account are completely avoided.

Azure SQL Services also provide ways to protect the data. These include Transparent Data Encryption, Always Encrypted, and Dynamic Data Masking. The encryption for the database is like that for the storage discussed just before the discussion on Azure SQL products in that the database is stored on the Azure storage infrastructure which guarantees the encryption at rest and the TLS guarantees the encryption in transit. The encryption supports Bring Your Own Key aka TDE protectors which are managed by the customers and saved in a Key Vault which helps meet compliance requirements around access and audit. A managed identity enables the Azure SQL Database service to access the Key Vault. With Always Encrypted, the data is encrypted within columns within the tables when storing, say credit card information. An encryption key is used for one or more columns and the data is not in the clear even for administrators. When the data needs to be hidden even from a genuine user, data masking is used so that only partial information is displayed, say the last four digits of the credit card number.

These are some of the ways to secure data on the Azure Data Platform.

 

This is a continuation of the discussion on Azure Data Platform.

GitHub actions are highly versatile and composable workflows that can be used to respond to events in the code pipeline. These events can be a single commit or a pull-request that needs to be build and tested or the deployment of a code change that has been pushed to production. While automations for continuous integration and continuous deployment are well-established practices in DevOps, GitHub Actions goes beyond DevOps by recognizing events that can include the creation of a ticket or an issue. The components of a GitHub actions are the workflow triggered for an event and the jobs that the workflow comprises of. Each job will run inside its own virtual machine runner, or inside a container and has one or more steps that either run a script or an action that can be reused.

A workflow is authored in the .github/workflows/ directory and specified as a Yaml file. It has a sample format like so:

name: initial-workflow-0001

run-name: Trying out GitHub Actions

on: [push]

jobs:

  check-variables:

    runs-on: ubuntu-latest

    environment: nonProd

    steps:

      - name: Initialize

        uses: actions/setup-node@v3

        with:

          node-version: '16'

      - name: Display Variables

        env:

             variable: 0001

        if: github.ref == ‘refs/heads/main’ && github.event_name == ‘push’

        run: |

            echo 000-“$variable”

            echo 001-“${{ env.variable }}”

And whenever there is a commit added to the repository in which the workflows directory was created, the above workflow will be triggered. It can be viewed with the Actions tab in the repository where the named job can be selected, and it steps expanded for the activities performed.

With this introduction, let us check out a scenario for using it with the Azure Data Platform. Specifically, this scenario calls for promoting the Azure Data Factory to higher environments with the following steps:

1.       Development data factory that is created and configured with GitHub.

2.       Feature/working branch is used for making changes to pipelines, etc.

3.       When changes are ready to be reviewed, a pull request is created from the feature/working branch to the main collaboration branch.

4.       Once the pull request is approved and changes merged with the main collaboration branch, the changes are published to the development factory.

5.       When the changes are published, data factory saves its ARM templates on the main publish branch (adf_publish by default). A file called ARMTemplateForFactory.json contains parameter names used for resources like key vault, storage, linked services, etc. These names are used in GitHub Actions workflow file to pass resource names for different environments.

6.       Once the GitHub Actions workflow file has been updated to reflect the parameter values for upper environment and changes pushed back to GitHub branch, the GitHub Actions workflow is started manually and changes pushed to the upper environment.

Some observations to make are as follows:

GitHub is configured on development data factory only.

Integration Runtime names and types need to stay the same across all environments.

Self-hosted Integration Runtime must be online in upper environments before deployment or else it will fail.

Key Vault secret names are kept the same across environments, only the vault name is parameterized.

Resource naming needs to avoid spaces.

Secrets stored in GitHub Secrets section.

There are two environments – dev and prod in this scenario.

Dev branch is used as collaboration branch.

Feature1 branch is used as a working branch.

Main branch is used as publish branch.