Object Tracking

Object tracking is done with video and image analysis with and without depending on cloud. Using embedded processors or lightweight edge AI chips, drones can continuously monitor and track objects in real time, maintaining lock even as the scene changes.

If the target is known beforehand (for example, a specific vehicle, person, or structure), onboard tracking algorithms—such as Kernelized Correlation Filters (KCF), Kalman Filters, or deep learning-based trackers pre-trained with the object's signature—can initialize detection using a reference image or video frame. Once the initial detection is established, the drone processes each incoming video frame locally, updating the object's spatial position, velocity, and trajectory. This supports persistent tracking—even when the object moves across the frame or through complex environments—without incurring the delays of transmitting every frame to the cloud.

Crucially, real-time onboard processing enables immediate response behaviors, such as adjusting flight path, camera gimbal, or surveillance pattern to keep the object centered in view. If multiple images are captured in burst mode, image-based keypoint matching (e.g., SIFT, ORB) or template matching can further refine object identity. Embedded systems may use bounding box prediction and re-identification modules to handle occlusion or temporary loss of visibility—all within the drone, keeping autonomy and performance high.

Onboard tracking is especially effective when the UAV must operate in bandwidth-limited or communication-constrained environments, delivering low-latency control and adaptive behavior for predetermined targets while limiting data offloading to only critical events or summary information.

When extended to the cloud, object tracking has leveraged significant compute capabilities, limitless storage and <10ms inter-services latency to demonstrate better performance and efficiency in the following examples:

1. CloudTrack: Semantic Object Tracking with Foundation Models

• CloudTrack introduces a two-part framework: real-time trackers run on the UAV, with cloud back-end using foundation models (e.g., large vision-language models) to deliver advanced semantic understanding and object disambiguation not feasible onboard. Experiments show cloud-enabled semantic object tracking outperforms onboard-only methods in accuracy, scalability, and multi-object scenarios, especially for open-vocabulary or rarely seen object types.3

• Extensive evaluation demonstrates improvement over state-of-the-art onboard approaches for semantic tracking, although it incurs extra runtime in the cloud (i.e., slightly more latency). The cloud empowers missions like search-and-rescue or tracking multiple distinct objects simultaneously that onboard systems struggle with.

2. DeepBrain: Energy and Throughput Benefits

• The DeepBrain project demonstrates cloud vision analytics deployed for drone video streams, with CNN models run in the cloud. Processing speed on cloud GPUs achieves much greater throughput (up to 12 frames/sec) compared to the ~1 fps limit of onboard GPU-less edge devices.

• Cloud offloading reduces the drone's energy consumption, enabling complex model execution (deep CNNs for car detection) that would otherwise overwhelm onboard resources. Thus, real-time cloud object tracking extends flight time and enables richer analytics.

3. AI-Powered Video Analysis for Scale and Accuracy

• Cloud video analytics allows for the aggregation of data across many cameras (or drones), using deep learning for retrospective tracking and behavioral analysis across large regions and time periods.

• AI-powered backend analysis often detects behavioral patterns, anomalies, or tracking targets that onboard models—limited by resource constraints and single-scene context—cannot match.

• Advanced cloud video analysis yields higher tracking accuracy (up to 99% in some deep-learning safety applications), supports forensic tracking, and adaptive queries, outperforming in-camera or edge-only solutions for large-scale applications.

4. Scalable Multi-Camera/Multi-Drone Tracking

• Multi-camera tracking research shows cloud analytics can correlate objects across different drones’ feeds, resolving ambiguities and re-identifying targets across wider areas than onboard systems typically support.

• The cloud backend processes and fuses metadata, supporting cross-drone object association, long-term monitoring, and efficient resource allocation.

Summary Table

Advantage	Onboard Only	Cloud Video Analytics	Reference
Tracking accuracy	Limited by resources	High (deep learning, semantic)	3
Throughput (fps)	~1–5 fps	Up to 12 fps (GPU)	4
Multi-object/vocabulary	Limited	Flexible, open vocabulary	3
Energy consumption (drone)	High	Low (offloaded)	4
Large-scale, post-hoc analytics	Not feasible	Aggregated, region-wide	5

Cloud video analytics has enabled more complex and accurate object tracking in drone applications—especially when advanced model architectures, semantic context, cross-feed correlation, and high throughput are crucial.