Perspectives on Data organization following takeaways from Microsoft Ignite 2023.

Based on the takeaways shared in an article earlier, this section describes how AI will fundamentally change data organizations within enterprises that want to embrace the semantic search facilitated by models such as Llama2 and Phi2. These models bring smartness to workflows and interactions, and it is inevitable that enterprises will strive to leverage this form of understanding of data that was not possible until now. AI needs data and the rich history of data organizations, platforms and their evolutions have many lessons for the forthcoming wave of AI pervasiveness throughout the data usages that will manifest as turnkey creation of custom AI Applications.

When the data warehouses emerged in 1980, structured business data and syntax for querying grew in popularity to the point that they continue to be ubiquitous even after the accumulation of unstructured data by 2010. To cover both forms of data, data lakes became popular and by 2015 both data warehouses and data lakes were required. This dual platform increased the challenges around governance, security, reliability, and management. While file formats supported both schema-based parquet file and delta lake, a unified form of security, governance and cataloguing was required. Now as generative AI helps to understand the semantics of data, a new layer is required to democratize access to data, automating manual administration and seamlessly bridging categories of data platforms.

The challenges data industry has been plagued with will not disappear and the major ones will remain:

-          Technical skills in SQL , python, or Business Intelligence that pose a steep learning curve

-          Enterprises struggling to curate data or scrub it to maintain accuracy.

-          Complexity of managing distinct systems increasing costs and requiring dedicated staff

-          Privacy, governance and compliance will require maintaining lineage, and visibility

And to add to this, emerging data analytics models will require developing and tuning LLMs in platforms that are separate from the data and connect with them manually. Neither the data platforms understand enterprise data organization, nor the data scientists understand why they must be bothered with connecting data

Building on a foundation that unifies querying and managing all forms of data across the enterprise is a great start for overcoming the syntax associated with contents, metadata, queries, reports, lineages and others but the deeper understanding facilitated by custom AI solutions will demand the following features that will add yet another albeit necessary layer for facilitating semantic organization and querying capabilities. These demands include:

-          Natural language processing adaptability and interfaces for the data

-          Semantic cataloguing and discovery because the models can articulate discrepancies in data

-          Automated management as models can optimize data layout, partitioning, and indexing

-          Enhanced governance and privacy as platforms detect, classify, and prevent misuse of sensitive data while simplifying management using natural language.

-          First class support for AI workloads because they deliver accurate results

There is some precedent for this by way of Business Intelligence layer but that was focused on a narrow slice of workloads while this is broader and more sweeping. Ultimately, enterprises will seek to find such a layer either from the products and cloud services they use or will build one themselves.