Gen AI created a new set of applications that require a different data architecture than traditional systems. Traditional databases cannot help innovating in this space and now there are existing applications that are enhanced with AI. The demands on the data architecture that allow people to build applications quickly and efficiently at scale is the most important need of the hour. Even the data structures expected to store records are changing. A search analytics database that stores vector embeddings and indexes vector embeddings so that we can extract value from both your structured and unstructured data. You also need observability along with databases. There is a need to have multiple aspects from the data stores that power these AI-era applications. There are structure and data governance requirements surrounding the storing and use of this data especially with renewed emphasis on building trust by leveraging privacy and data protection capabilities. There is also a need to unify data whether they are from event streams to bring in real-time data into the system or whether they are transactions or change data captures. Performance considerations have also changed from benchmarks that look obsolete in the face of what is required to train the models. From 2015 to today, the main emphasis in data architectures has been the separation of compute from storage at cloud scale that is evident in the way models are trained, tested, and released today as well as the release of successful products like Snowflake and Databricks. This is going to change in the case of AI application data architectures with primary use case of Uber-like applications because it is real-time like people, places and things as well as unifying all the different data sources. There are two really important sides where one side involves data training or tuning with proprietary data sets that comes with infrastructure that allows us to aggregate all this data and build really efficient models and then the other side is the inference side where we take these models and extract embeddings and this comes with serving tier with which to build AI enhanced applications. Both of these need to be enhanced so that there are very fast iterative cycles. And then one more aspect of building both of these subsystems is the enablement of real-time data collection and analysis. Temporal and spatial capabilities also matter as aspects to this data architecture. Also, vectors are important for identifying context, but a new kind of data set needs to be behavioral which comes from metadata filtering where the search space is reduced. Applications that empower drones include Retrieval Augmented Generation, pattern matching, anomaly detection, and recommendation systems just like many other AI applications. Contextual, behavioral, accuracy and personalization of data and search characterizes this architecture. 

Reference: DroneData.docx

#codingexercise: CodingExercise-06-26-2024.docx