Using Azure Data Factory to upload and transform data to Azure Data Lake Storage Gen2.

This is a continuation of the articles on Azure Data Platform as they appear here and discusses data security and compliance for ADF and Data Lakes.

ADF can work with many types of data sources and ingest files and folders of varying size and number and to the tune of petabytes in size. Microsoft Purview can be used to govern, protect, and manage data estates. It provides integrated coverage, helps address the fragmentation of data across rest and transit. This kind of solution helps an organization to protect sensitive data across clouds, applications and devices and identify data risks and manage regulatory compliance requirements.  It helps to create an up-to-date map of the entire data estate that includes data classification and end-to-end lineage, identify sensitive data and create a secure environment for data consumers to find valuable data and generate insights about how the data is stored and used. With data in ADF and data lake, such a report is very helpful to meet compliance with standards such as SOC, ISO, HiTrust, FedRamp and HIPAA.

ADF can be connected to Microsoft Purview. There are two options to do so:

1.       Connect to Microsoft Purview account in Azure Data Factory and

2.       Register Data Factory in Microsoft Purview

3.       Complete with Azure Monitor based alerts

The prerequisites are an Owner or Contributor role on the ADF to connect to a Microsoft Purview Account and ADF to have a system assigned managed identity enabled. The connection takes merely the Azure subscription to locate the Purview account and one of them is selected. The connection information is stored in the ADF resource. ADF’s managed identity is used to authenticate lineage push operations from the ADF to the Microsoft Purview account. The Data Curator role on the root collection of the Microsoft Purview must be assigned to the managed identity of the ADF.

Both this connection and the Purview integration capabilities can be monitored. The default integration capability is the data lineage pipeline. When this pipeline is executed, the lineage information is transmitted to the Purview account. The search bar at the top center of the Data Factoring authoring UI can be used to search for data and perform actions. This is very helpful to understand the data based on metadata, lineage, and annotations. Many organizations heavily rely on tagging and metadata and even to the point of specifying paths and dedicating storage containers for such information.

 With the data searched by Microsoft Purview, it is possible to create Linked Service, Dataset, or dataflow over the data.

All the activity runs from ADF have status, copy duration, throughput, data read, files read, data written, files written, peak connections for both read and write, the parallel copies used, the data integration units, the queue and the transfer durations to provide complete information on the activities performed for monitoring or troubleshooting.

 

 

This is a continuation of the articles on Azure Data Platform and discusses data ingestion for Data Lakes.

ADLS Gen2 or data lake for short can hold petabytes of data and store it as files and folders with file-level security and scale. ADF can load data into the data lake with high throughput and massive parallelization.  A point to point copy activity can take several minutes for gigabytes of data over the internet and the scale out provided by ADF makes it all the more appealing to use. This article discusses some of the considerations made towards data ingestion that inevitably occurs for loading data into a data lake.

First, the prerequisites for data ingestion must be called out. An Azure subscription and a storage account with Azure data lake gen2 enabled are required at the minimum. The source of the data can be another cloud storage or on-premises. ADF can work with many types of data sources but we will focus on the case of ingesting files and folders of varying size and number and to the tune of petabytes in size.

Creating an ADF from the Azure portal is easy to follow with the steps outlined in the user interface. It has an Ingest tile to launch the Copy Data tool. In the properties window, the built-in copy task is available which can be run on-demand or periodic basis.  The data source must be specified and this can point to an existing S3 compatible storage on-premises that provides the file-system used for the files and folders to be copied. This will require an integration runtime, an access key and secret. There are three types of integration runtime: Azure, self-hosted and Azure-SSIS.  A self-hosted integration runtime is capable of running copy activity between a cloud data store and a data store in private network.  The transform activities are dispatched against the compute resources on-premises. The Azure integration runtime is used for data flows while the Azure-SSIS is for relational data. The self-hosted integration runtime makes outbound HTTP connections to the internet so it can sit behind the firewall and not require a direct link to the cloud. It runs on Windows Operating System and a single logical instance can be associated with multiple physical on-premises machines in active-active mode.

The transfer of data must be in binary mode, and it must recursively traverse the files and folders. The destination must be configured as the data lake by pointing to the Azure subscription and the storage account with the ADLS Gen2 option specified. The destination folder structure can preserve that of the origin and a preview option ensures that the data can be copied correctly. ADF can extract zip files prior to sending them to the data lake. The copy operation is tracked with a task. The pipeline comprises of the copy task and the monitor task. When the pipeline run completes successfully, its activity details can be viewed to rerun it if necessary.

All the activity runs have status, copy duration, throughput, data read, files read, data written, files written, peak connections for both read and write, the parallel copies used, the data integration units, the queue and the transfer durations to provide complete information on the activities performed for monitoring or troubleshooting.

 

This is a continuation of the articles on Azure Data Platform and discusses Data Lakes:

Access Control Model:

This section covers the access control model in Azure Data Lake Storage Gen2 which supports the following mechanisms: shared key authorization, shared access signature authorization aka SAS, role-based access control aka RBAC, attribute-based access control aka ABAC and access control lists aka ACL. Out of these the RBAC and ACLs on one hand and shared-key and SAS on the other hand are complimentary. The former has no effect on the latter. Shared-key and shared access both grant access to a user without the need for an identity. RBAC grants coarse-grained access to users such as for read-write of all data.  ABAC refines RBAC role assignments by adding conditions. ACLs grant fine-grained access such as a write access to a specific directory or file. The security principals recognized by the Azure are a user, group, service principal or managed identity  that is defined in the Azure Active Directory. A permission set grants coarse level access such as a read or write access to all the data in a storage account or all the data in a container. Permission sets are granted based on the roles and some well-known roles are Owner, Contributor, Reader, and Storage Account Contributor. The first three can access the data and all four can manage the storage account. They cannot grant access to other security principals but they can provide shared-key and shared access signatures except for the reader role. The order of resolving access grant or denial is RBAC first, ABAC next and ACLs last and applies to all operations such as list, get, create, update, and delete. Security groups are particularly useful to add ACLs. For example, if Azure Data Factory aka ADF ingests data into a folder named /LogData and specific service engineering team upload data in a container and various users analyze the data, then we create a LogsWriter group and LogsReader group to enable these activities. The POSIX permissions assigned to the directory will show rwx permissions assigned the LogWriter group and r-x permissions assigned to the LogReader group. The service principal or the managed service identity aka MSI for ADF will be part of the LogWriter group but the service principal or MSI for Databricks will be part of the LogsReader group. Groups facilitate addition or removal of members without disturbing the assignments. They also help to avoid exceeding the maximum number of role assignments per subscription and the maximum number of ACL entries per file or directory. The limits are 4000 Azure role assignments in a subscription and 32 ACL entries per file or directory.

Premium tier:

ADLS Gen2 supports premium tier and premium block blob storage accounts that are ideal for Big Data analytics and workloads that require low latency and support a high number of operations such as in the case of machine learning. The premium tier supports hierarchical namespace that accelerates big data analytics workloads and enables file-level access control lists. It also supports Azure Blob file system drive for Hadoop.

 The topic discussed here are some key considerations for the use of Azure Data Lake Storage Gen2 which is a highly scalable and cost-effective data lake solution for big data analytics.

The first and foremost consideration is that a data lake is not appropriate where a data warehouse is suitable.  They are complimentary solutions that work together to help us derive key insights from our data. A data lake is a store for all types of data from various sources besides the unstructured versus structured data differentiation.  A retail company can store the past 5 years’ worth of sales data in a data lake, and in addition they can process, and they can process data from social media to extract the new trends in consumption and intelligence from retail analytics solutions and then generate a highly structured data set that is suitable to store in the warehouse. ADLS Gen2 can offer faster performance and Hadoop compatible access with the hierarchical namespace, lower cost, and security with fine grained access controls and native AAD integration. It can become a hyperscale repository that can store up to petabytes of data and hundreds of Gbps in throughput.

Other considerations include what will the data lake store and for how long, what portion of the data is used in analytical workloads, who needs access to what parts of the data lakes, what are the various analytical workloads and what am I going to run on the data lake, what are the types of workloads that will use the data, what are the transaction patterns and the analytics patterns and what is the budget I’m working with.

Organization and managing of data in the data lake is also a key concern for this single data store. Some users of this store  can claim end-to-end ownership of the pipeline and others have a central team that manages, operates and governs the data lake. This calls for different topologies of the data lake such as centralized or federated data lake strategies, implementing with single or multiple storage accounts, and with globally shared or regionally specific footprints.

When the customers of the data lake are both internal and external, the scenarios may be subject to different requirements from security and compliance perspectives, query patterns, access requirements and cost and billing. Storage accounts can be created in different subscriptions for development and production environments and subject to different SLAs. The account boundaries determine the management of logical sets of data in a unified or isolated manner. Subscription limits and quotas apply to resources used with the Azure data lake such as the VM cores and ADF instances. Managed identities and SPNs must have different privileges than those who are merely reading the data.

Backup and archive is done with the help of scripts that create action and filter objects to apply blob tiering to block blobs matching a certain criteria and storage accounts must be provisioned in a brand-new resource group and storage account based upon input variables.

 

 

Data Lake is often used with Azure Data Factory to support Copy operations. It allows for downloading of files and folders directly from the Lake and to the order of thousands of files from folders, but the upload is best done with the help of Azure Data Factory which supports even upload of zip file and the in-transit extraction of files before they are stored in the Data Lake. ADF happens to support a variety of file formats such as Avro, binary and text to name a few and uses all available throughput by performing as many reads and writes in parallel as possible.

Data Lakes work best for partition pruning of time-series data which improves performance by reading only a subset of data.  The pipelines that ingest time-series data often place their files with a structured naming such as /DataSet/YYYY/MM/DD/HH/mm/datafile_YYYY_MM_DD_HH_mm.tsv.

The Hitchhiker’s Guide to Data Lake from Azure recommends that monitoring be set up for effective operations management of this cloud resource. This will help to make sure that it is available for use by any workloads which consume data contained within it. Key considerations include auditing the data lake in terms of frequent operations, having visibility into key performance indicators such as operations with high latency and understanding common errors.  All of the telemetry will be available via Azure Storage Logs which is easy to query with Kusto query language.

Common queries are also candidates for reporting via dashboards for Azure Monitor resource. For example,

1.       Frequent operations can be queried with:

StorageBlobLogs

| where TimeGenerated > ago(3d)

| summarize count() by OperationName

| sort by count_ desc

| render piechart

 

2. High latency operations can be queried with:

StorageBlobLogs

| where TimeGenerated > ago(3d)

| top 10 by DurationMs desc

| project TimeGenerated, OperationName, DurationMs, ServerLatencyMs, ClientLatencyMs = DurationMs – ServerLatencyMs

 

3. Operations causing the most error are caused by:

  StorageBlobLogs

| where TimeGenerated > ago(3d) and StatusText !contains "Success"

| summarize count() by OperationName

| top 10 by count_ desc

 

and these can be reported on the dashboard.

The differentiation between a Healthcare Insurance Industry Cloud and a general purpose cloud solution for health-care data.

 Introduction: This article discusses whether Healthcare and InsuranceTech require different industry clouds and whether they can be directly hosted as cloud solutions. Indeed, the cloud has all the resources that allows any infrastructure to be created for any company and purpose regardless of size. This begs the question of what’s special about Healthcare & InsuranceTech.

First, there is Fast Healthcare Interoperability Resources (FHIR) which is a healthcare data standard and an information network that lets us link data across systems and a communication network that lets us exchange data between systems. Healthcare IT Systems often don’t share the same data models. In fact, as more data becomes digitized, incompatibilities and resolving them becomes more expensive and time-consuming.

Second, there is the Health Insurance Portability and Accountability Act of 1996 and associated laws that establish requirements for the use, disclosure, and safeguarding of protected health information (PHI). HIPAA applies to doctor’s offices, hospitals, health insurers, and other healthcare companies – that create, receive, maintain, transmit or access PHI. HIPAA and Health Information Technology for Economic and Clinical Health Acts include rules for 1. Privacy for safeguards to protect PHI without authorization and 2. Security for administrative, technical, and physical safeguards and 3. a Breach Notification for whenever there is a breach of unsecured PHI occurs.

Third, there is the consent management that involves managing data related to consent and privacy across configuration management for the consent store, data related to the permissions granted by the users and managed resources that include user data mappings and data related to the resources in the form of attributes.

Lastly, there must be an overt display of security and compliance controls that span both observability and security controls. As an example of compliance, the Cybersecurity Maturity Model Certification attempts to prevent the theft of intellectual property and sensitive information from all industrial sectors due to malicious cyber activity. FedRamp High and FedRamp moderate both pertain to account management, monitoring, and role-based access controls and have different impact levels. HIPAA HITRUST 9.2 targets both privilege management and role-based access control. 

The difference between Healthcare and InsuranceTech can be compared to technologies that are customer-facing and backend processing.  On one hand, the customer facing data must be aggregated from various sources and a service or cloud greatly abstracts and simplifies this handling and on the other hand, insurance companies increase the competition by introducing various innovations, analysis and ultimately new options when processing these data. Many industry clouds from different cloud providers have agreed on the need for a dedicated API that can be consumed from various providers, clients and their devices. Insurance IT companies seek to develop new business capabilities with rapid application development and new machine learning models that mandate the need for dedicated pipelines. Even the cloud resources these two use, differ in their purpose. Healthcare benefits from FHIR API and the InsuranceTech prefers data lakes and pipelines.

 Some essentials for an integration pipeline include: 

1. Data Ingestion: data passed to the blob storage and the metadata queued in a job database for downstream processing. 

2. 
   

2. Central components:  

1. messaging queue: all components interact with these which allows scalability.  

2. config store: enables stable pipeline with configuration driven variability 

3. job store: keeps track of various job executions 

4. 
   

3. Job poller/scheduler: this picks up and drops a message in the message queue, for example, Kafka. 

4. 
   

4. Consumers: these must be horizontally scalable where the data is staged in tenant specific database. 

5. 
   

5. Databases: these must be horizontally scalable where the data is transformed and stored in suitable multi-tenant databases. 

6. 
   

6. Data warehouses/ data lake: the data must be denormalized with dimensions along say tenants to support multitenant data sources. 

7. 
   

7. Analytics stack: The Data Lake/warehouse is the source for the analysis stack and preferably U-SQL based for leveraging existing skills 

8. 
   

8. ML stack: also leverages the data lake/warehouse but with emphasis on separation of training and test datasets and the execution and feedback loop for a model. 

9. 
   

9. Monitoring, performance, and security: these include telemetry, auditing, security and compliance, aging and lifecycle management.