Drone Data Architecture:

One of the advantages of a cloud platform for real-time drone data capture and analysis is that the businesses who sign up for it do not have to reinvent it for themselves. In fact, the cloud data architecture, deployment and data driven applications can be stood up with full IaC and data seeding without even involving any drones. When this proof-of-concept has succeeded, scaled, performed and optimized for lowest cost, it is a veritable proprietary patentable asset and one that can serve the world for various use cases. As of today, just a little bit over 10% of enterprises, enact data strategy that works. Drone data a niche but any investments in its planning and architecture will win over mindshare that can only grow. Data is regarded as a byproduct of operations but it can become a driver of business value. With that comes many choices for data infrastructure and tools in a vast ecosystem. Cloud continues to dominate this space with ever increasing storage and computing power to crunch the data. With drone data, the options for being tied down with legacy or on-premises silos just do not exist, so there is an opportunity to start right. Some consider data architecture to be an oxymoron because databases, data warehouses and data lake do not eliminate one another and evolve their own architectures, so that is again an opportunity to start right for drone data without complexity and even come with full-service in the foreseeable increase in regulatory requirements on drone data. Data democratization will favor data literacy which in turn will foster data culture. Also, this builds a single source of truth with purview and that matters.

The previous articles regarding storage of Drone Data enumerated the following components or lines of data organization for workloads such as business analytics, data engineering, streaming and Machine Learning:

• Traditional databases for inventory including progressive states and timestamps along with drone capabilities such as degrees of freedom which is inherently relational 

• Vector database for training and inferences for both self-organizing maps and CNNs

• Performance databases leveraging embedded, unstructured, QoS and cloud databases

• Graph databases to serve graph analytics between drones 

• Cache infrastructure to mitigate the load on the data tier 

• Streaming Data services such as Apache Flink Live and others.

Leaving out the microservices, apis, UI, scriptability and analytics stacks out of the data access discussion and in this section, we describe the workloads in terms of updates and search. This data architecture drawn out from the previous articles strives to drive the most relevant, real-time application experience supporting data acquisition, metadata filtering and the highest F-score search results. A single query must be responded to using vector search, text search, and metadata filtering and consequently span multiple data sources which may not have been virtualized or made amenable to a single SQL-like query interface. While vector embeddings use large language models are becoming popular on the web and cloud computing, model for drones is subject to more specific domain data and filters and draws from the interdisciplinary science from traffic engineering, computer networks, pattern recognition and database systems. This translates to the following requirements:

Fast complex search – spanning vectors, txt, geo and custom json data

Data and index changes – spanning large number of vectors

Rankings – from various ranking algorithms

Real-time and historical updates – with ability to update and delete any data including vectors in milliseconds without incurring reindexing costs

Hyperconverged indexing that includes different types of indexes such as vector index, range index, column store and documents will be sought-after for this kind of platform to manage drone fleet.

#codingexercise: CodingExercise-07-15-2024.docx

 Drone:

- SysId

- ID

- Name

- Status

- Created

- Modified

- CreatedBy

- ModifiedBy

- Degree-of-freedom

- Max speed

- FuelType

- BatteryLife

- NumberOfFans

- FanStatus1-8

- Location

- Weight

- PayloadWeight

- Mileage

- Color

- LightSensor

- HeatSensor

- IRSensor

- Camera

- Zoom

- Pitch

- Yaw

- Roll

- Display0-255

Fleet:

- ID

- Name

- Drones

- Created

- Modified

- CreatedBy

- ModifiedBy

Location

- Geo coordinates

- GPS Tracking

- LastUpdated

-

 These are more use cases targeted for a commercial drone fleet management software that scales elastically and helps drones manage their flight path in real-time.

Case 6: So far the use cases have elaborated conveniences for drone makers, fleet operators, businesses leveraging drone for delivery, but the following use case focuses on end-users. A drone unit and a small personal fleet of drones can come in helpful for individuals such as farmers or for home security where the units fly to take aerial photo/video around a specific landmark such as a home or barn. Today a single drone is sold along with its remote-operated controller for manual flying but a small fleet of drones can also be sold if they are registered with and controlled by cloud software for specific limited activities. The end-user may download an application and scan the qrcodes off the back of drones so that the application registers them with the clouds and is able to relay commands individually to them. Then a set of predetermined algorithms can help navigate the  drones for aerial surveillance and footage. They can be launched on ad hoc basis or scheduled periodically and matched with image recognition for durations when active monitoring might help. With the potential to build different models of drones for different range and speed and the possibility to scale up the number of drones in the fleet seamlessly, this software could meet a variety of usages across sectors such as fire and rescue, pet surveillance, pest control, improved security, and many more that do not fall under the small-and-medium business category and enterprise category of use cases mentioned earlier.  With topologies that allow dedicated controller for a fleet of lightweight drones and the controller receiving input from the cloud software and relaying to the drones while sending back the sensor data from the drones, the entire compute, storage and network available from the cloud can be leveraged to overcome the limitations on what is shipped out-of-box today or enhance existing fleet of popular appliances like vacuum robots. A high volume of sensor data traffic over the internet might be considered slow for response times in near real-time to drones but they are more for guidance or changes to schedule so-to-speak for mundane activities in these limited scopes for drones. The cloud is uniquely positioned to scale up to the demands of real-time traffic.

Case 7: Federal Regulatory Compliance and Governance of drone usages and activities across tenants is a specific use case dedicated to this platform and one that can be alleviated from the responsibilities of the businesses signing up as tenants. The platform is uniquely positioned to provide a full data purview including genealogy, history, archival and auditing of every operation associated with the drones registered with the platform. Information as requested and required to be disclosed based on sub-poena sent by law enforcement agencies are easy to be drafted and responded to by the platform. For the businesses themselves, change data capture of the drone data as well as real-time information from logs and metrics stores are something that they can view for themselves and thus enforce necessary role-based access controls and auditing for their requirements on governance.

Previous use cases: DFCSUseCasesList.docx 

 These are more use cases targeted for a commercial drone fleet management software that scales elastically and helps drones manage their flight path in real-time.

Case 5: Fleet operators looking to serve various businesses want to leverage a common platform for management and operations of their fleet.  Earlier, controllers were assigned to partitioned fleets and the logistics would be dependent on each with isolation provided between controllers. As a multitenant platform, both isolation and scalability are no longer restricted for the customers of this Drone Formation Commercial software. The fleet operators can add as many tenants as they can accommodate, and it is easy to roll inventory from one tenant to another. Decisions taken for the operation of the fleet are not only captured in the same single pane of glass for management but also the outcomes are easy to roll up in consolidated reports from the platform while maintaining drill down on authorized queries. 

Case 6: So far the use cases have elaborated conveniences for drone makers, fleet operators, businesses leveraging drone for delivery, but the following use case focuses on end-users. A drone unit and a small personal fleet of drones can come in helpful for individuals such as farmers or for home security where the units fly to take aerial photo/video around a specific landmark such as a home or barn. Today a single drone is sold along with its remote-operated controller for manual flying but a small fleet of drones can also be sold if they are registered with and controlled by cloud software for specific limited activities. The end-user may download an application and scan the qrcodes off the back of drones so that the application registers them with the clouds and is able to relay commands individually to them. Then a set of predetermined algorithms can help navigate the  drones for aerial surveillance and footage. They can be launched on ad hoc basis or scheduled periodically and matched with image recognition for durations when active monitoring might help. With the potential to build different models of drones for different range and speed and the possibility to scale up the number of drones in the fleet seamlessly, this software could meet a variety of usages across sectors such as fire and rescue, pet surveillance, pest control, improved security, and many more that do not fall under the small-and-medium business category and enterprise category of use cases mentioned earlier.  With topologies that allow dedicated controller for a fleet of lightweight drones and the controller receiving input from the cloud software and relaying to the drones while sending back the sensor data from the drones, the entire compute, storage and network available from the cloud can be leveraged to overcome the limitations on what is shipped out-of-box today or enhance existing fleet of popular appliances like vacuum robots. A high volume of sensor data traffic over the internet might be considered slow for response times in near real-time to drones but they are more for guidance or changes to schedule so-to-speak for mundane activities in these limited scopes for drones. The cloud is uniquely positioned to scale up to the demands of real-time traffic.

 These are more use cases targeted for a commercial drone fleet management software that scales elastically and helps drones manage their flight path in real-time.

Case 4: As drone manufacturers make incredible progress in the scope and purpose of drone activities, they would like to outsource software targeting repeated drills across models and versions of their drones including abilities to plugin customizations for enhanced intelligence. These drone manufacturers are experimenting with a variety of shapes, compositions and degrees of freedom both individually and collectively for formations and their ability to overcome obstacles. A cloud software that can be reached ubiquitously by drones themselves or relayers will be cost-effective in maximizing the output of these drones. Among the features expected by the manufacturers are ability to repeat tests, maintain test history,  and generate test reports. A single transparent pane of glass for observability and manageability of the drones along with these reports and dashboards will be very helpful to them for updating their drones.  Initially the size of the fleets for most of these manufactures might be small in research and development but their deployments on their customer premises could be large and require dial-home capabilities from the units or their controllers and relayers. Different kinds of topology involving different types of units participating in overall fleet management and formations must be supported by this software.

Case 5: Fleet operators looking to serve various businesses want to leverage a common platform for management and operations of their fleet.  Earlier, controllers were assigned to partitioned fleets and the logistics would be dependent on each with isolation provided between controllers. As a multitenant platform, both isolation and scalability are no longer restricted for the customers of this Drone Formation Commercial software. The fleet operators can add as many tenants as they can accommodate and it is easy to roll inventory from one tenant to another. Decisions taken for the operation of the fleet are not only captured in the same single pane of glass for management but also the outcomes are easy to roll up in consolidated reports from the platform while maintaining drill down on authorized queries. 

 These are more use cases targeted for a commercial drone fleet management software that scales elastically and helps drones manage their flight path in real-time.

Case 3. Many businesses outsource their fleet with a fleet provider that maintains a large inventory of hybrid models that are continually updated in their degrees of motion, capabilities and inputs. These businesses expect to have one interface with the fleet provider and another with the fleet management service such that they can choose to organize their fleet dynamically and pass the fleet details to their management service for commands that they can relay to their fleet members. They expect a detailed and comprehensive management dashboard or portal where they can monitor the key performance indicators aka KPI  for the fleet and drill down. They expect these programmability interfaces to be compatible in a way that requires little or no processing other than relaying the commands or updating the goals on the management dashboard.

Case 4: As drone manufacturers make incredible progress in the scope and purpose of drone activities, they would like to outsource software targeting repeated drills across models and versions of their drones including abilities to plugin customizations for enhanced intelligence. These drone manufacturers are experimenting with a variety of shapes, compositions and degrees of freedom both individually and collectively for formations and their ability to overcome obstacles. A cloud software that can be reached ubiquitously by drones themselves or relayers will be cost-effective in maximizing the output of these drones. Among the features expected by the manufacturers are ability to repeat tests, maintain test history,  and generate test reports. A single transparent pane of glass for observability and manageability of the drones along with these reports and dashboards will be very helpful to them for updating their drones.  Initially the size of the fleets for most of these manufactures might be small in research and development but their deployments on their customer premises could be large and require dial-home capabilities from the units or their controllers and relayers. Different kinds of topology involving different types of units participating in overall fleet management and formations must be supported by this software.

 These are the use cases targeted for a commercial drone fleet management software that scales elastically and helps drones manage their flight path in real-time.

Case 1. A retail company owns and operates a fleet of drones with proprietary software to deliver merchandise purchased by its customers to their residences. The software tracks each drone individually providing route and schedule information independent of others. The number of drones exceeds hundreds of thousands. The software is tasked with the dual goal of assigning non-overlapping,  contention free and flow maximizing  flight manifests for each drone as well as keeping the costs low for the operation of a single drone deployed to serve a customer. Controllers for the drones are placed in multiple cities worldwide and operate their own fleet independent of others and cover a specific non-overlapping geographic region. The software differentiates from the one used to mange the fleet of robots in its warehouse in that the space to operate is not a grid but a hub-and-spoke model and the destinations are not internally managed but received from order processing service bearing address, volume, weight and delivery date information in a streaming manner. The software does not differentiate between fleet members and requires the ability to form the fleet dynamically by replacing , adding or removing members. When this proprietary software becomes general purpose use for fleets of varying sizes, operating loads, and quality of service, it becomes promising as a multitenant platform so that different companies do not need to reinvent the wheel. This platform must keep the use of cloud resources as efficient as possible to pass on the savings to businesses and streamline consumption of additional resources when any single parameter is stretched. Additionally, it must provide offloading of any interface of its components for customization by these businesses. For example, they can determine their own scheduling plug-in and are not bound to supplying values to predetermined scheduling templates. Volume and rate of drone command updates must cover the six sigma of normalizes  probability distribution of current and foreseeable future fleet sizes.

Case 2. A small and medium sized business is looking to improve the efficiency of  scheduling for its fleet and wants to download models that it can operate on its own premises or invoke remotely from APIs. This controller-offload-to-the cloud scenario demands world class performance and availability at affordable price. This business does not expect to grow its fleet or experience a lot of churns in its fleet but appreciates full service outsourcing of its fleet management so that it merely controls the inventory and the commands to the fleet members. It also expects to include a clause for liability of drone damages resulting from faulty instructions. It might want to change scheduling algorithms depending on changing business priorities that are not fully covered by the scheduling parameters.

Case 3. Many businesses outsource their fleet with a fleet provider that maintains a large inventory of hybrid models that are continually updated in their degrees of motion, capabilities and inputs. These businesses expect to have one interface with the fleet provider and another with the fleet management service such that they can choose to organize their fleet dynamically and pass the fleet details to their management service for commands that they can relay to their fleet members. They expect a detailed and comprehensive management dashboard or portal where they can monitor the key performance indicators aka KPI  for the fleet and drill down. They expect these programmability interfaces to be compatible in a way that requires little or no processing other than relaying the commands or updating the goals on the management dashboard.

References: https://1drv.ms/w/s!Ashlm-Nw-wnWhPA9saJLYQGA7q2Wiw?e=AONTxo for data architecture 

https://1drv.ms/w/s!Ashlm-Nw-wnWhO4OGADjCj0GVLyFTA?e=UGMEpB for software description.