Drone Data Architecture:

One of the advantages of a cloud platform for real-time drone data capture and analysis is that the businesses who sign up for it do not have to reinvent it for themselves. In fact, the cloud data architecture, deployment and data driven applications can be stood up with full IaC and data seeding without even involving any drones. When this proof-of-concept has succeeded, scaled, performed and optimized for lowest cost, it is a veritable proprietary patentable asset and one that can serve the world for various use cases. As of today, just a little bit over 10% of enterprises, enact data strategy that works. Drone data a niche but any investments in its planning and architecture will win over mindshare that can only grow. Data is regarded as a byproduct of operations but it can become a driver of business value. With that comes many choices for data infrastructure and tools in a vast ecosystem. Cloud continues to dominate this space with ever increasing storage and computing power to crunch the data. With drone data, the options for being tied down with legacy or on-premises silos just do not exist, so there is an opportunity to start right. Some consider data architecture to be an oxymoron because databases, data warehouses and data lake do not eliminate one another and evolve their own architectures, so that is again an opportunity to start right for drone data without complexity and even come with full-service in the foreseeable increase in regulatory requirements on drone data. Data democratization will favor data literacy which in turn will foster data culture. Also, this builds a single source of truth with purview and that matters.

The previous articles regarding storage of Drone Data enumerated the following components or lines of data organization for workloads such as business analytics, data engineering, streaming and Machine Learning:

• Traditional databases for inventory including progressive states and timestamps along with drone capabilities such as degrees of freedom which is inherently relational 

• Vector database for training and inferences for both self-organizing maps and CNNs

• Performance databases leveraging embedded, unstructured, QoS and cloud databases

• Graph databases to serve graph analytics between drones 

• Cache infrastructure to mitigate the load on the data tier 

• Streaming Data services such as Apache Flink Live and others.

Leaving out the microservices, apis, UI, scriptability and analytics stacks out of the data access discussion and in this section, we describe the workloads in terms of updates and search. This data architecture drawn out from the previous articles strives to drive the most relevant, real-time application experience supporting data acquisition, metadata filtering and the highest F-score search results. A single query must be responded to using vector search, text search, and metadata filtering and consequently span multiple data sources which may not have been virtualized or made amenable to a single SQL-like query interface. While vector embeddings use large language models are becoming popular on the web and cloud computing, model for drones is subject to more specific domain data and filters and draws from the interdisciplinary science from traffic engineering, computer networks, pattern recognition and database systems. This translates to the following requirements:

Fast complex search – spanning vectors, txt, geo and custom json data

Data and index changes – spanning large number of vectors

Rankings – from various ranking algorithms

Real-time and historical updates – with ability to update and delete any data including vectors in milliseconds without incurring reindexing costs

Hyperconverged indexing that includes different types of indexes such as vector index, range index, column store and documents will be sought-after for this kind of platform to manage drone fleet.

#codingexercise: CodingExercise-07-15-2024.docx