Cutting Tokens in Drone Video Analytics
The inference cost of aerial drone images and video even with sampling often runs into thousands of tokens and almost always repeated per query given the long tail of such analytics. GeoVARNet and our proposal asked: "Can representation-guided enrichment reduce computation without materially reducing event recall?" [1]. Players like NVIDIA are also asking similar questions but from a different perspective.
The Video Search and Summarization (VSS) blueprint is NVIDIA's reference architecture for video agents that perform:
• video search,
• visual Q&A,
• summarization,
• alerting,
• report generation,
using VLMs, LLMs, retrieval systems, and video analytics pipelines. They introduced Adaptive Efficient Video Sampling (EVS) that dynamically prunes unchanged visual patches per frame and batches VLM work around moments of activity.[3]
Their reported outcomes include:
• ~80% fewer VLM input tokens for a 60 minute video,
• ~50% faster summarization,
• 17% lower alert-contextualization latency,
• 46% higher concurrent real-time VLM stream capacity [3]
Although the benchmarks for drone video analytics such as UAVBench and EZBenchmark are diverse and require reprocessing, the EVS matters over traditional VLM pipelines that often progress through ingestion, frame sampling, VLM inference, Caption captures, and image embeddings and typically repeat that for every input even though almost nothing changed. EVS attempts to suppress that waste by detecting regions that are largely unchanged and avoiding feeding redundant visual information into the VLM. With their introduction of change detector, they reprocess only changed regions/chunks. This is referred to as token-pruning. GeoVARNet, on the other hand, engages in representation-guided enrichment. So, it can answer questions on an “evidence” basis rather than at “pixel” basis with a much broader definition of evidence and decision variables such as persistence, covisibility, revisits, place identity, entity history, observation coverage and rare-event likelihood. These are all first-class architectural concepts in GeoVARNet.
For example, if a parking lot is observed six times, EVS sees parking lot looks mostly unchanged and reduces token usage while GeoVARNet could additionally infer same place, same appearance, same persistent state, already enriched, and high confidence. This would help avoid VLM calls, embedding calls, graph expansion, retrieval enrichment, and storage growth. That is a stronger form of reduction than token pruning alone.
There are others pursuing similar ideas though not with the same objectives. For example, Keyframe selection systems such as ORB-SLAM3, PKS (photogrammetric key-frame selection) and OpenVSLAM decide to only process frames that contribute new geometry. While GeoVARNet suggests using PKS, those techniques are for geometry redundancy, not semantic redundancy.
Modern video-RAG systems have a workflow that uses frame to embedding conversion and then determine distance from prior embedding to decide whether to retain or discard the frame and while this does not reduce the embedding generation cost for those frame and is used by many commercial video indexing pipelines, it leans on semantic novelty rather than geometry. This fails on tiny moving objects, vehicle swaps, transient events and drone revisits which GeoVARNet tries to preserve.
Another angle to reduce costs is detecting scene boundaries as found in video search, summarization and media indexing. If the scene did not change, why incur cost? But drone videos often have the same scene for different event which scene segmentation misses.
A more recent approach involves memory systems for Video LLMs that aim to improve with
• StreamingVLM-style approaches
• memory-token approaches
• recurrent video transformers
These also obviate repetitive processing but by means of summary memory and sparse visual retention with a targeted transformer context length. While this reduces tokens, it does not retain as GeoVARNet does by compressing observations, places, revisits, and relationships at a higher-level abstraction.
The notable direction with NVIDIA VSS is the push towards Graph-RAG video systems where the architecture includes Graph-RAG, video search, agent workflows and video understanding thus generalizing for all video and not just urban drone videos. It completely avoids reprocessing the video to query the graph.
This direction is a direct threat to GeoVARNet because it combines Embedding Novelty Sampling with Graph-RAG and Keyframe Selection and together it can already provide dramatic savings without introducing additional geometry decomposition and revisitation processing but GeoVARNet could demonstrate that it outperforms token-pruning, keyframe-selection, and embedding-novelty baselines at equal event-recall targets in terms of enrichment workload, VLM calls, storage growth, and event recall, When it does, it has a much stronger claim than reducing computation
