Tuesday, November 17, 2026

 From Drone Video Archives to Governed Autonomy Evidence

Is video search already a competitive market, and where could DVSA still differentiate? Yes, it is already crowded. Generic video search is not the moat. The stronger position is a governed evidence layer. It connects video to missions, telemetry, autonomy decisions, operator interventions, and software lineage. One correction comes first. Camera-based perception does not mean an operator continuously records, streams, or retains video. Amazon states that its delivery-drone cameras are not viewed live by people. Wing describes low-resolution navigation imagery, not a continuous FPV archive. These distinctions shrink the addressable market. Vendor capabilities below are advertised functionality, not independently verified accuracy, completeness, or forensic admissibility. Architectural proposals and strategic inferences are marked as such.

DVSA faces four competitive forces. Public clouds make video understanding and retrieval easy to assemble. Fleet operators already own perception, mission-management, and data pipelines. Independent platforms turn enterprise video libraries into searchable assets. Insider-risk products pair desktop recordings with activity metadata and investigation workflows. AWS publishes architectures for frame-based analysis, shot-based understanding, and multimodal embedding retrieval. Microsoft provides timestamped video insights and search. TwelveLabs separates retrieval models from video-language models. Proofpoint and Teramind link visual recordings to activity logs, risk events, and forensic investigation. The opportunity is therefore not to make video searchable. It is to make operational evidence discoverable, interpretable, policy-controlled, and reproducible across heterogeneous autonomy systems.

Much of the generic pipeline already exists. AWS Bedrock Data Automation extracts chapters, text, and prompt-defined objects, and AWS documents frame, shot, and embedding designs for video search. Azure AI Video Indexer produces time-ranged insights, exact-moment search, and multimodal summaries, in the cloud and on Azure Arc-enabled edge. Google documents Gemini video understanding, timestamped chapters, and multimodal embeddings. Extraction, embeddings, and retrieval are therefore replaceable infrastructure. Mission-aware evidence management must be distinguished from general-purpose indexing. None of these is necessarily a turnkey product that ingests live fleet feeds, synchronizes telemetry, interprets safety events, and exports investigation packages. They do cut the effort to build the generic layer. AWS's own reference architecture combines orchestration, inference, vector search, and timestamp metadata storage.

A new pattern separates retrieval from interpretation. Retrieval models locate candidate video intervals. Video-language models then describe or answer questions about selected intervals. TwelveLabs exposes this through Marengo for embeddings and Pegasus for video-to-text, and AWS offers both through Bedrock. Independent model providers can thus be partners and cloud-enabled competitors at once. For DVSA the proposed flow runs from video and telemetry ingestion to time-aligned segments and event records, then to geospatial, temporal, structured, and vector indexes. It continues to candidate retrieval and targeted video-language analysis, and ends in an evidence-linked answer and investigation package. A generated description is not a complete representation of its source. A description index and a multimodal embedding index preserve different information, and Microsoft's migration documentation distinguishes the two. Product lifecycles are also an architectural risk. Azure AI Vision Video Retrieval was retired on June 30, 2025, with users directed to Content Understanding and Azure AI Search. DVSA should keep a provider-neutral media catalog, segment identifiers, timestamps, and provenance records. Embeddings and vendor insights should be versioned derivatives, never the only description of the archive.

Fleet operators have sophisticated stacks and incompletely disclosed archives. Public evidence shows heavy investment in autonomy and orchestration. It does not show that major platforms keep comprehensive, searchable FPV archives. Retailers and marketplaces must be separated from the operators who supply the vehicles. Walmart works with Wing and Zipline. Uber Eats works with robot partners. DoorDash combines partners with its own Dot platform. Their access to raw sensor data should not be assumed equal. Amazon Prime Air uses onboard cameras and radar for obstacle detection and delivery decisions. AWS describes Step Functions and Batch processing of lidar, aerial imagery, and geospatial inputs, plus ECS and Fargate mission-planning workflows. What is not public is any general-purpose retained FPV archive, its search interface, retention schedule, or third-party access model. Wing supplies Walmart's drone operations and describes low-resolution overhead navigation imagery, flight plans, drone locations, and flight histories. It is unknown whether Walmart receives raw imagery, whether collection is continuous, how long images are kept, or whether internal semantic search exists. Walmart documents its Zipline expansion, but a partnership alone does not show video storage, indexing, evidence export, or retailer access. DoorDash's Dot uses lidar, radar, and cameras, and its Autonomous Delivery Platform orchestrates humans, robots, and drones. Its fleet-wide video search, raw-video availability, retention, and evidentiary lineage are not public. Uber lists Avride, Cartken, Coco, and Serve as partners. A unified Uber-owned sensor archive or common forensic schema is not established.

Amazon shows the strength of the in-house alternative. Its published architecture already spans geospatial processing, delivery-point scoring, trajectory planning, contingency planning, and preflight orchestration. This is far more than a camera stream attached to an order. The strategic inference is that mature operators may build evidence capabilities into existing systems rather than buy a replacement. DVSA would have to offer something complementary: cross-system investigation, portable evidence packages, consistent policy enforcement, or cheaper management of video and its derivatives. Marketplaces add an integration problem. DoorDash picks delivery modes by speed, cost, location, distance, weight, and order traits. Uber integrates services from several autonomy companies. The marketplace holds order and customer-service context. The operator holds mission and sensor context. A remote-assistance system may hold intervention records. Other systems may hold model releases and maintenance history. This is a proposed integration model, not a claim about any company's internal schema. DVSA could target that boundary with standardized incident requests and evidence manifests. A retailer needs a bounded answer to what happened during a delivery, not open access to an operator's camera archive.

Independent platforms already serve enterprise customers. They show both competition and reusable patterns. TwelveLabs offers multimodal retrieval and video-language analysis beyond transcripts. Its Qencode case study adds searchable video as an output of an existing encoding workflow, and searches return timestamped clips. The reusable pattern is enrichment inside an existing media pipeline: encode, generate semantic representations, and publish searchable time ranges. DVSA need not make operators replace their media systems. It can attach evidence generation and governance to ingest, encoding, and mission-closeout workflows. TwelveLabs also publishes a retail-CCTV tutorial that combines library search, structured analysis, and playable timestamped evidence. It overlaps with DVSA's proposed investigation experience, though a tutorial is not a production deployment. Veritone's Digital Media Hub combines media management, AI metadata, search, permissions, sharing, and distribution. Its customers include the San Francisco Giants and Vital Voices, and its PROGRESS announcement describes automated cataloging of a large film archive. These are vendor-reported deployments, not independent evaluations. Buyers purchase workflows around video, not an embedding model. Veritone annotates video segments and exports time-bounded metadata as JSON and XML. DVSA should therefore invest in the full investigator workflow. Find the event. Review the source interval. Inspect the metadata. Record annotations and disagreements. Apply access and disclosure policies. Export a reproducible evidence package. Snowflake adds another pressure. Its TwelveLabs guide stores timestamped embeddings, retrieves them with Cortex Search, and plays results at matching timestamps, with analysis in optional private preview. Video intelligence is becoming a workload inside enterprise data platforms. DVSA should interoperate with them while owning the autonomy-specific evidence model.

Desktop monitoring is a relevant adjacent market. Products already record or reconstruct screen activity, index it, and tie it to security investigations. Video there needs careful reading. Some products use screenshot sequences, activity-triggered capture, and changed-region storage rather than full-motion video, and Proofpoint documents these methods. Teramind advertises live desktop viewing, historical playback, OCR search of recorded screens, indexed activity timelines, policy alerts, and configurable recording and privacy controls. Its newer insider-risk materials describe optional recording around risk events. An investigator searches for sensitive on-screen content and then reviews what happened before and after it. Proofpoint Insider Threat Management, formerly ObserveIT, documents metadata-indexed recording, policy-controlled capture, and investigation workflows. It can switch from metadata-only monitoring to screenshot recording when an alert fires. It manages screenshot data separately, with hot and warm storage tiers. It also audits the people who use the monitoring system, the watch-the-watchers principle. That matters for any platform giving investigators sensitive operational footage. The precedents map directly. Metadata-only monitoring that escalates to visual capture becomes routine telemetry that escalates to protected video when risk triggers fire. A searchable activity timeline linked to replay becomes a mission timeline linking perception, planning, intervention, and video. Application and user recording policies become camera, mission, location, and event policies. Separate visual and metadata lifecycles become separate lifecycles for raw video, tracks, embeddings, summaries, and evidence manifests. Auditing monitoring access becomes auditing who searched, viewed, exported, or unmasked evidence. The analogy has limits. Desktop interfaces expose application names, URLs, commands, and file events. Drone video must infer meaning from moving-camera imagery and imperfect sensor alignment. DVSA should borrow the workflow without assuming comparable accuracy.

The following software patterns are architecture proposals informed by the market. They are not claims that operators already use them. First, separate sensing, recording, transmission, and retention. A sensor operating is not a frame recorded. A frame recorded is not a frame transmitted off the vehicle. A frame transmitted is not a frame retained long term. A frame retained is not a frame available for authorized investigation. This avoids overstating archive size or surveillance capability, as Amazon's onboard processing and Wing's imagery policy show. A mission manifest should state what was captured, retained, discarded, and unavailable. Second, use event-oriented segments, not only video shots. AWS uses shot segmentation to preserve temporal context. Autonomy should extend it to delivery approach and clearance checks, obstacle encounters, localization uncertainty, remote-assistance requests, human interventions, aborted deliveries, and communications interruptions. The key unit is the approach and intervention episode, not a fixed clip. Third, combine structured filtering with semantic retrieval. Use exact filters for mission, time, location, vehicle, software version, and event type, and multimodal retrieval for visual concepts. An illustrative query asks for aborted delivery approaches near pedestrian crossings where a person entered the delivery area after the initial clearance check. The plan should narrow candidates by structured and geospatial constraints, retrieve visual candidates, inspect surrounding intervals, and join operational logs. An answer must separate the video showing a pedestrian, the planner logging an obstacle, and an analyst concluding the pedestrian caused the abort. Fourth, treat semantic outputs as versioned claims. Keep separate records for original sensor evidence, model observations, operator actions and statements, human-reviewed findings, and generated explanations. A VLM narrative should cite its evidence, its model and configuration, and any uncertainty or reviewer correction. Timestamped grounding already exists in commercial tools, and DVSA should extend it into explicit evidence lineage. Fifth, use a durable catalog with rebuildable indexes. The authoritative layer preserves mission and camera identities, source time ranges and synchronization, media integrity records, sensor and software provenance, retention and legal-hold decisions, and derivation relationships. Vector indexes, caches, and summaries stay replaceable, which reduces dependence on one embedding model or cloud product. Sixth, make evidence-preserving retention a first-class policy. The earlier progression from full video to key frames to embeddings to long-lived digests remains an example, not a universal policy. A safer policy distinguishes routine missions, safety events, human interventions, disputes and complaints, legally preserved evidence, and restricted research or training datasets. Deleting raw video while keeping its summary limits what investigators can later verify. A digest can show the integrity of an available object. It cannot rebuild a deleted recording. Semantic compression is a task-dependent information-loss trade-off, not a guaranteed substitute for source evidence.

Where could DVSA differentiate? This is a strategic assessment, not a measured forecast. Generic descriptions, OCR, and summaries face strong cloud and model-provider capabilities, so DVSA should integrate rather than recreate them. Natural-language clip retrieval faces strong specialist and cloud platforms, so DVSA should add mission context and evaluation. Media cataloging and sharing face established enterprise platforms, so DVSA should interoperate with existing archives. Video, telemetry, and decision alignment faces incomplete public operator disclosure, so DVSA should build autonomy-specific connectors and schemas. Cross-operator incident exchange is not established by the reviewed public materials, so DVSA should investigate standardized evidence requests and manifests. Policy propagation across video and derivatives needs application-specific design, so DVSA should make governance consistent across every evidence representation. Reproducible incident reconstruction needs more than semantic search, so DVSA should preserve sources, versions, uncertainty, and analyst actions. The defensible claim is not that competitors lack governance. Cloud, media, and insider-risk vendors already advertise substantial security and control. The narrower hypothesis is that heterogeneous autonomy evidence needs a mission-centered integration and governance layer that general-purpose video products do not automatically supply.

The recommended positioning is this. DVSA is a governed evidence platform for autonomous operations. It connects video, telemetry, perception outputs, interventions, and software lineage into searchable, policy-controlled incident records. It integrates with existing cloud, media, and fleet systems. It should be validated around one bounded workflow before expanding into a horizontal physical-AI platform. Given a disputed delivery or safety event, retrieve the evidence across participating systems, reconstruct the timeline, expose uncertainty and missing data, and produce an auditable review package. The next research step is customer and operator validation of what public documentation leaves open. What camera data actually leaves the vehicle? What remains after routine retention expires? Which organization can authorize access? Can investigators align video with planning and intervention logs? How long does an investigation take today? Can operators export a complete incident package without bespoke engineering? Which parts of the existing workflow would buyers replace, and which must stay?

Public clouds and independent platforms already make video searchable, summarize it, and return relevant moments. Enterprise security products go further by linking visual recordings to activity timelines, risk events, and controlled investigations. DVSA should build on these patterns, not present them as capabilities that do not yet exist. Autonomous delivery adds a more specific evidence problem. A camera image alone may not explain what a vehicle perceived, which route it planned, whether communications degraded, or why a human intervened. The opportunity is to connect these records without confusing original evidence, machine inference, and retrospective explanation. DVSA's next phase should be defined less by the type of camera and more by the integrity of the operational record. Its value lies in preserving the relationships among what was sensed, what was inferred, what was commanded, what occurred, and what can still be verified.


Monday, November 16, 2026

 Delivery Robot Video Forensics and the Opportunity for DVSA Governance

Do four-wheeled food delivery robots in American cities continuously capture video? And could that video become a target for governance, auditability, provenance, compliance, and forensics under a DVSA (Drone Video Security, Safety, and Auditability) initiative? Yes. Every commercially viable sidewalk robot depends on camera-driven perception. Most also add remote supervision, teleoperation, or event recording. Retention policies are rarely disclosed. Even so, substantial video is generated. Sidewalk robots are effectively ground-based drones. They share persistent cameras, edge AI, remote fleet supervision, human intervention, fleet analytics, model-improvement pipelines, and regulatory accountability. They also sit between drones and autonomous vehicles. They are unmanned yet constantly among the public. Coco Robotics operates in Los Angeles, Chicago, Miami, and Helsinki. Its robots navigate busy sidewalks and use low-latency remote operation. This implies that video is streamed to operators and becomes an operational record. Each run yields navigation, pedestrian, crossing, and delivery imagery. It also yields teleoperation video, route and localization traces and intervention records. These matter when questions arise about yielding, collisions, wheelchair access, misclassification, or human override. Serve Robotics grew out of Uber. It publicly advertises live low-latency video, monitoring, and remote control. It offers the richest opportunity. Each trip contains layers of sensor reality, machine perception, navigation reasoning, human oversight, human intervention and communications evidence such as latency, bitrate, and packet loss. Investigators can use these layers to tell algorithm failure from communications failure. The result resembles flight telemetry combined with drone video archives. Starship Technologies has made millions of deliveries. Its robots carry twelve cameras plus radar, ultrasonics, and time-of-flight sensing. It holds a vast pedestrian-height corpus covering congestion, accessibility, weather, and night operation. At that scale raw-video archives give way to event-triggered retention, semantic indexing, object trajectories, scene graphs, VLM summaries and provenance records. These support incident reconstruction, behavioral audits, algorithm audits, safety certification, and synthetic-data validation. Nuro builds road-going vehicles with cameras, lidar, radar and 360-degree perception. It is the closest to advanced drone autonomy. It calls for sensor, fusion, localization, planning, decision, human-review, and software provenance. This converges with AI observability. Smaller operators include Kiwibot, Cartken and campus, hospital, and corporate fleets. They work in semi-private settings with minors, visitors, and sensitive sites. Their concerns shift toward privacy, retention, access, chain of custody and compliance. Standardized DVSA controls would help here. Adoption may come readily. The realistic forensic hierarchy starts with raw video, which has the highest value and the highest cost. It then runs through key frames, sensor metadata, object tracks, perception outputs, and scene graphs. It continues with intervention and teleoperation logs and VLM narratives. It ends with software lineage. DVSA could unify all of these into one evidence graph. Ground robots may prove a stronger governance market than drones. They operate all day among the public. They repeat fixed routes. They produce structured navigation data. They also face growing regulations. This creates recurring risk and recurring demand for evidence and compliance. DVSA is therefore autonomy-video governance. It is a horizontal governance plane for physical AI. This covers delivery, security, warehouse, industrial, inspection and teleoperated ground robots. Each run becomes a governed and cryptographically verifiable record of video, sensors, perception, decisions, operator actions, and model lineage. Drone governance is DVSA 1.0. Delivery robots are a natural route to DVSA 2.0. Enterprise tools have made text and structured data cheap to retain. Video remains hard. Codecs shrink files but do not make them governable. Robot fleets will produce petabytes. The real asset is the meaning in the footage and not the container. This points to semantic compression and deduplication. It also points to event-aware sampling and frame pruning such as NVIDIA's Video Search and Summarization and PruneVid. Related tools include Video RAG, embedding-based vector search, scene graphs, object tracking and AI-written narratives. Retention can then age by meaning rather than time. Full video is kept on day zero. Key frames follow at one month. Embeddings and summaries follow at one year. Governance digests remain for five years. Incidents, safety events, and interventions are kept long-term. Routine runs are kept briefly. Cloud tiering from S3, Azure and Google handles storage but not semantics. The opportunity lies in a software layer above it. Video will become manageable once computers index meaning. Documents became manageable once computers indexed words. This is the trend DVSA most needs to watch.


Sunday, November 15, 2026

 Improving precision and recall in aerial drone images

Implementations of “standard” analytics pipeline on any video fail more often than not in raising the bar for precision and recall due to factors such as quality of data, embeddings, algorithms for retrieval and application of reranking, let alone narrowing the data set down to aerial drone images where the objects don’t even appear in the same façade form as in image search sites. Then, how do we improve the F-score in such analytics.

This comes down primarily to the following relation:

RetrievalQuality≈EmbeddingQuality x IndexRecall x RerankingQuality

Where embedding quality improves semantic understanding, index recall determines true neighbors survive candidate generation, and re-ranking determines final precision. For aerial drone images, the order of priority is domain-specific embeddings, multi-stage retrievals and re-ranking, vector index selection, and hardware scaling.

If the precision and recall is for retrieval quality only, nothing beats Exhaustive-KNN (Exact nearest neighbor search) for its brute force approach but most if not all production systems replace it with an ANN (Approximate nearest neighbor) search. Most researchers establish Ground Truth Recall on datasets such as VisDrone, UAVDT, DOTA, xView and SkyScenes as Exhaustive KNN and then compare ANN against it. And the most favored commercial ANN is the HNSW (Hierarchical Navigable Small World) for its best trade-off between recall, latency, simplicity and maturity with some vector store implementations claiming nearly 95-99% of Exhaustive KNN regardless for 1536 dimension vectors. Applications often tune their retreivals by keeping M between 32-64, efConstruction to be between 400-800 and efSearch to be between 200-1000. These ranges cover products such as Azure AI Search index, Vespa, Weaviate, Qdrant, Milvus and Pinecone.

But as pointed embedding models determine the disambiguity between vectors. Fine-tuned CLIP models such as GeoCLIP, RemoteCLIP, SkyCLIP, SatCLIP, and variations of OpenCLIP out perform regular plain CLIP model and work well as Tier 1 models for aerial drone images. They succeed in detecting drone world objects such as aircraft, runway, warehouses, substation, container yard, and solar farm. But taking it a notch higher is DinoV2 embeddings because they can match buildings, roofs, parking lots and other infrastructures. DinoV2 outperforms both Amazon and Azure for their S3 Vectors and AI Search respectively on similarity search.

If the retrieval is exclusively multimodal, Google SigLIP performs better than other CLIPs on the basis of its zero-shot retrieval, improved ranking, and stronger semantic descriptions. SigLIP + HNSW + Cross-encoder re-ranker significantly raises the game. Similar stages is suitable for image search purposes across domains but the model might need to change from CLIP cross attention to BLIP to Florence to Vision-Language-Models. These improve precision.

GeoVARNet style drone retrieval benefits from adding more stages to the pipeline such as an evidence encoder, higher dimension embedding, filtering the HNSW results and taking the top few even from the re-ranking. This must be combined with metadata retrieval such as GPS, Altitude, Heading, Time, Region, and object detections.

The main takeaway from this article is that focus on embedding and reranking, not so much on the substitute for HNSW. An earlier article explained retrieval escalation techniques for standardizing your pipeline. With these, switching out HNSW rarely moves the needle as much.


Saturday, November 14, 2026

 What percentage of aerial drone videos are top-down nadir videos? Most drone experts would hesitate to give that answer but when pressed they would lean towards a 40-60 split between nadir and oblique/FPV videos. In this article, we explore how accurate that might be. First, there are two di¯erent drone worlds: 1. Photogrammetry/mapping drones 2. Inspection/media/consumer drones The first of these two, benefits from a near-nadir perspective over other angles because it is helpful to create orthomosaics, surface models and measurements. Oblique coverage is helpful to façade or height estimation but not ground/roof coverage. Most commercial shot planning uses a mix, but the percentage varies on a case-by-case basis. Historically photogrammetry and mapping datasets such as OpenDroneMap/ESRI accrued nadir datasets with a dominance of top-down collections but the situation reverses with the ubiquity of hobbyist/consumer drones that generate YouTube drone footage, real-estate marketing, inspections, construction progress, emergency or rescue response, racing and cinematic content. Computer Vision has also switched to the study of these videos as evident from VisDrone/UAV tracking datasets that are forward looking. So, the market determines the ration of these videos. Photogrammetry and Mapping also benefit from still photographs rather than videos. Mapping is easier with one acquisition setup whereas oblique imagery requires using five cameras. Datasets for mapping often have varying ratios depending on use cases but they all benefit from derived orthomosaics and related products. Geospatial analytics and mapping ecosystem among these use cases, have the most skewed ratio and in favor of nadir perspective. Even when the perspectives mix in the same video between takeo¯ and landing, a credible estimate based on duration alone is elusive because of revisits. A single nadir tour of a survey area has high overlaps with those from extended rounds and revisits that usually build up a drone tour flight path. However, nadir videos help with a survey map more than any oblique videos. Any study of such a ratio amongst existing datasets is also subject to the same limitations as with analytics over a single drone video. It is inherently multimodal in nature and the search terms and retrieval frameworks no matter whether hierarchical navigable small world, exhaustive KNN, agentic or tool-based su¯er from significant variations in precision and recall with none coming close to being true to actual. The most credible reply therefore is that the answer might be unknown, but the market determines the choice.

Thursday, November 12, 2026

 PruneVid for aerial drone videos:

The previous article discussed using one of the three supported VLMs for PruneVid to improve token efficiency with the smallest model: PLLaVA. The other two are LLaVA-OneVision and ST-LLM (Liu et al., 2024b).  

ST-LLM feeds all raw, unpooled spatial-temporal tokens directly into the LLM backbone. For an egomotion camera where changes between consecutive frames are incremental, preserving the raw spatial-temporal progression allows the LLM to model subtle frame-to-frame displacements, motion cues, and panning trajectories. To prevent the massive context length from slowing down the system, it uses a dynamic masking strategy during training. This means it can drop redundant tokens where frames overlap significantly while forcing the model to retain a highly granular understanding of the continuous stream

The following implementation uses LLaVA-OneVision instead of the PLLaVA and ST-LLM from the previous article:

"""Standalone LLaVA-OneVision aerial-video runner. No PruneVid/ST-LLM imports."""

import argparse

import faulthandler

import json

import logging

import math

from pathlib import Path

import platform

import time


LOG = logging.getLogger('onevision')

DEFAULT_MODEL = 'llava-hf/llava-onevision-qwen2-0.5b-ov-hf'



def sample_indices(total, fps, start, seconds, frames):

    if total < 1 or not math.isfinite(fps) or fps <= 0:

        raise ValueError('Invalid video frame count or FPS. Transcode to constant-frame-rate MP4.')

    if not math.isfinite(start) or not math.isfinite(seconds) or start < 0 or seconds <= 0 or frames < 1:

        raise ValueError('Require start >= 0, seconds > 0 and frames >= 1.')

    first = int(start * fps)

    last = min(total, int((start + seconds) * fps)) - 1

    if last - first + 1 < frames:

        raise ValueError('Clip is out of range or contains fewer source frames than --frames.')

    return [first] if frames == 1 else [round(first + i * (last-first)/(frames-1)) for i in range(frames)]



def save_json(path, value):

    temporary = path.with_suffix(path.suffix + '.tmp')

    temporary.write_text(json.dumps(value, indent=2), encoding='utf-8')

    temporary.replace(path)



def parse_args():

    p = argparse.ArgumentParser(description=__doc__)

    p.add_argument('--video', required=True)

    p.add_argument('--model-dir', '--model-id', dest='model_dir', default=DEFAULT_MODEL,

                   help='Complete Hugging Face OneVision checkpoint directory or repository ID.')

    p.add_argument('--start', type=float, default=0)

    p.add_argument('--seconds', type=float, default=20)

    p.add_argument('--frames', type=int, default=16)

    p.add_argument('--question', default='Describe the visible terrain, structures, roads, and moving objects. Distinguish camera motion from object motion. State uncertainty; do not invent GPS coordinates or metric distances.')

    p.add_argument('--max-new-tokens', type=int, default=160)

    p.add_argument('--output', default='runs/drone_onevision')

    p.add_argument('--prepare-only', action='store_true')

    p.add_argument('--device', choices=['cuda', 'cpu'], default='cuda')

    p.add_argument('--dtype', choices=['auto', 'float16', 'bfloat16', 'float32'], default='auto')

    p.add_argument('--load-in-4bit', action='store_true')

    p.add_argument('--local-files-only', action='store_true')

    p.add_argument('--revision', default='main', help='Hub revision; use a commit hash for reproducibility.')

    a = p.parse_args()

    if a.max_new_tokens < 1 or a.frames < 1 or not a.question.strip():

        p.error('Require positive frames/max-new-tokens and a nonempty question.')

    if a.device == 'cpu' and a.load_in_4bit:

        p.error('This adapter supports bitsandbytes 4-bit loading on CUDA only.')

    return a



def prepare(a, target):

    import cv2

    from PIL import Image, ImageDraw

    source = Path(a.video).resolve()

    if not source.is_file():

        raise FileNotFoundError(source)

    cap = cv2.VideoCapture(str(source))

    images, samples = [], []

    try:

        if not cap.isOpened():

            raise RuntimeError('OpenCV cannot open the video; check codec and path.')

        fps = float(cap.get(cv2.CAP_PROP_FPS))

        total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

        indices = sample_indices(total, fps, a.start, a.seconds, a.frames)

        for i, index in enumerate(indices):

            cap.set(cv2.CAP_PROP_POS_FRAMES, index)

            ok, bgr = cap.read()

            if not ok:

                raise RuntimeError(f'Failed to decode frame {index}.')

            image = Image.fromarray(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))

            images.append(image)

            image.save(target / f'frame_{i:03d}.jpg')

            samples.append({'frame_index': index, 'nominal_seconds': index / fps})

    finally:

        cap.release()

    sheet = Image.new('RGB', (1280, math.ceil(a.frames / 4) * 205), 'white')

    draw = ImageDraw.Draw(sheet)

    for i, image in enumerate(images):

        thumb = image.copy()

        thumb.thumbnail((320, 180))

        x, y = (i % 4)*320, (i // 4)*205

        sheet.paste(thumb, (x, y))

        draw.text((x+4, y+182), f'{samples[i]["nominal_seconds"]:.3f}s', fill='black')

    sheet.save(target / 'contact_sheet.jpg')

    meta = {'arguments': vars(a), 'video': str(source), 'fps': fps,

            'source_frame_count': total, 'timestamps_are_nominal': True, 'samples': samples}

    save_json(target / 'sampling.json', meta)

    return images, meta



def infer(a, images, metadata):

    import numpy as np

    import torch

    import transformers

    from transformers import AutoConfig, AutoProcessor, LlavaOnevisionForConditionalGeneration

    device = 'cuda:0' if a.device == 'cuda' else 'cpu'

    if a.device == 'cuda':

        if not torch.cuda.is_available():

            raise RuntimeError('CUDA unavailable. Install a CUDA-enabled PyTorch wheel; no implicit CPU fallback.')

        LOG.info('GPU: %s; capability: %s; torch: %s; runtime: %s',

                 torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0), torch.__version__, torch.version.cuda)

        if torch.cuda.get_device_capability(0) == (12, 0) and 'sm_120' not in torch.cuda.get_arch_list():

            LOG.warning('sm_120 not listed in this wheel. The following real GPU computation must pass.')

        x = torch.randn(128, 128, device=device)

        _ = x @ x

        torch.cuda.synchronize()

        del x, _

    dtype_name = a.dtype if a.dtype != 'auto' else ('float32' if a.device == 'cpu' else 'float16')

    if a.device == 'cpu' and dtype_name != 'float32':

        raise ValueError('Use float32 on CPU in this adapter.')

    if dtype_name == 'bfloat16' and a.device == 'cuda' and not torch.cuda.is_bf16_supported():

        raise RuntimeError('This GPU does not report BF16 support.')

    dtype = getattr(torch, dtype_name)

    shared = {'revision': a.revision, 'local_files_only': a.local_files_only, 'trust_remote_code': False}

    config = AutoConfig.from_pretrained(a.model_dir, **shared)

    if config.model_type != 'llava_onevision':

        raise ValueError(f'Expected llava_onevision, got {config.model_type!r}; PLLaVA/ST-LLM weights cannot be reused.')

    processor = AutoProcessor.from_pretrained(a.model_dir, use_fast=False, **shared)

    processor.tokenizer.padding_side = 'left'

    options = dict(config=config, torch_dtype=dtype, low_cpu_mem_usage=True,

                   device_map={'': device}, attn_implementation='sdpa', **shared)

    if a.load_in_4bit:

        from transformers import BitsAndBytesConfig

        options['quantization_config'] = BitsAndBytesConfig(

            load_in_4bit=True, bnb_4bit_quant_type='nf4',

            bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=dtype)

    LOG.info('Loading %s onto %s; no automatic RAM/disk offload.', a.model_dir, device)

    started = time.perf_counter()

    model = LlavaOnevisionForConditionalGeneration.from_pretrained(a.model_dir, **options).eval()

    load_seconds = time.perf_counter()-started

    times = ', '.join(f'{row["nominal_seconds"]:.3f}' for row in metadata['samples'])

    question = (f'The supplied video consists of {a.frames} sampled frames, in chronological order, '

                f'at nominal source-video times [{times}] seconds. Events between samples may be unobserved.\n' + a.question)

    messages = [{'role': 'user', 'content': [{'type': 'video'}, {'type': 'text', 'text': question}]}]

    prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

    clip = np.stack([np.asarray(image) for image in images])

    inputs = processor(text=prompt, videos=[clip], padding=True, return_tensors='pt')

    inputs = inputs.to(device=device, dtype=dtype)

    input_length = inputs['input_ids'].shape[1]

    limit = getattr(config.text_config, 'max_position_embeddings', None)

    if limit and input_length + a.max_new_tokens > limit:

        raise ValueError(f'Context overflow: {input_length}+{a.max_new_tokens}>{limit}; reduce --frames.')

    video_token_count = int((inputs['input_ids'] == config.video_token_index).sum().item())

    if a.device == 'cuda':

        torch.cuda.synchronize()

        torch.cuda.reset_peak_memory_stats()

    started = time.perf_counter()

    with torch.inference_mode():

        generated = model.generate(**inputs, max_new_tokens=a.max_new_tokens,

                                   do_sample=False, num_beams=1, use_cache=True)

    if a.device == 'cuda':

        torch.cuda.synchronize()

    elapsed = time.perf_counter()-started

    answer = processor.batch_decode(generated[:, input_length:], skip_special_tokens=True,

                                    clean_up_tokenization_spaces=False)[0].strip()

    return {**metadata, 'answer': answer, 'model_type': config.model_type,

            'resolved_model_commit': getattr(config, '_commit_hash', None),

            'model_load_seconds': load_seconds, 'generation_seconds': elapsed,

            'input_tokens': input_length, 'video_placeholder_tokens': video_token_count,

            'generated_tokens': generated.shape[1]-input_length,

            'input_shapes': {k:list(v.shape) for k,v in inputs.items() if hasattr(v, 'shape')},

            'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated()/2**30 if a.device == 'cuda' else None,

            'gpu': torch.cuda.get_device_name(0) if a.device == 'cuda' else None,

            'torch_version': torch.__version__, 'transformers_version': transformers.__version__,

            'python_version': platform.python_version(), 'resolved_dtype': dtype_name,

            'note': 'Standard OneVision video inference, not PruneVid pruning or GeoVARNet evidence reasoning. Timestamps are frame-index/FPS approximations.'}



def main():

    faulthandler.enable()

    logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')

    a = parse_args()

    target = Path(a.output)

    target.mkdir(parents=True, exist_ok=True)

    try:

        started = time.perf_counter()

        images, metadata = prepare(a, target)

        preparation_seconds = time.perf_counter()-started

        if a.prepare_only:

            LOG.info('Prepared %d frames, sampling.json and contact sheet; no model loaded.', len(images))

            return

        result = infer(a, images, metadata)

        result['preparation_seconds'] = preparation_seconds

        result['total_seconds'] = time.perf_counter()-started

        save_json(target / 'result.json', result)

        print(result['answer'])

    except Exception as error:

        save_json(target / 'error.json', {'error_type': type(error).__name__, 'message': str(error), 'arguments': vars(a)})

        LOG.exception('Run failed. No fallback answer was generated.')

        raise



if __name__ == '__main__':

    main()



Some caveat remain as before:

1. Libraries I used:

# Approximate environment for run_onevision_drone.py

# Use in a clean environment. GPU inference has not been validated here.

# --extra-index-url https://download.pytorch.org/whl/cu128

 torch==2.7.1+cu128

 torchvision==0.22.1+cu128

 transformers==4.51.3

 accelerate>=1.6,<2

 huggingface-hub>=0.30,<1

 numpy>=1.26,<2

 opencv-python==4.9.0.80

 pillow>=10,<12

 sentencepiece>=0.2,<0.3

 protobuf>=4.25,<6

 safetensors>=0.4.3,<1


 # Optional: uncomment for --load-in-4bit (not needed for the default 0.5B run).

 bitsandbytes>=0.47,<1


2. 

There is a checkpoint/code compatibility issue.


prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint. 

and there are three checkpoints involved which is suspicious:

LlaVa-OneVision

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.


1. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash LlaVA-OneVision compatibility issue.


Wednesday, November 11, 2026

 Cutting Tokens in Drone Video Analytics

The inference cost of aerial drone images and video even with sampling often runs into thousands of tokens and almost always repeated per query given the long tail of such analytics. GeoVARNet and our proposal asked: "Can representation-guided enrichment reduce computation without materially reducing event recall?" [1]. Players like NVIDIA are also asking similar questions but from a different perspective.

The Video Search and Summarization (VSS) blueprint is NVIDIA's reference architecture for video agents that perform:

• video search,

• visual Q&A,

• summarization,

• alerting,

• report generation,

using VLMs, LLMs, retrieval systems, and video analytics pipelines. They introduced Adaptive Efficient Video Sampling (EVS) that dynamically prunes unchanged visual patches per frame and batches VLM work around moments of activity.[3]

Their reported outcomes include:

• ~80% fewer VLM input tokens for a 60 minute video,

• ~50% faster summarization,

• 17% lower alert-contextualization latency,

• 46% higher concurrent real-time VLM stream capacity [3]


Although the benchmarks for drone video analytics such as UAVBench and EZBenchmark are diverse and require reprocessing, the EVS matters over traditional VLM pipelines that often progress through ingestion, frame sampling, VLM inference, Caption captures, and image embeddings and typically repeat that for every input even though almost nothing changed. EVS attempts to suppress that waste by detecting regions that are largely unchanged and avoiding feeding redundant visual information into the VLM. With their introduction of change detector, they reprocess only changed regions/chunks. This is referred to as token-pruning. GeoVARNet, on the other hand, engages in representation-guided enrichment. So, it can answer questions on an “evidence” basis rather than at “pixel” basis with a much broader definition of evidence and decision variables such as persistence, covisibility, revisits, place identity, entity history, observation coverage and rare-event likelihood. These are all first-class architectural concepts in GeoVARNet. 


For example, if a parking lot is observed six times, EVS sees parking lot looks mostly unchanged and reduces token usage while GeoVARNet could additionally infer same place, same appearance, same persistent state, already enriched, and high confidence. This would help avoid VLM calls, embedding calls, graph expansion, retrieval enrichment, and storage growth. That is a stronger form of reduction than token pruning alone.


There are others pursuing similar ideas though not with the same objectives. For example, Keyframe selection systems such as ORB-SLAM3, PKS (photogrammetric key-frame selection) and OpenVSLAM decide to only process frames that contribute new geometry. While GeoVARNet suggests using PKS, those techniques are for geometry redundancy, not semantic redundancy.

Modern video-RAG systems have a workflow that uses frame to embedding conversion and then determine distance from prior embedding to decide whether to retain or discard the frame and while this does not reduce the embedding generation cost for those frame and is used by many commercial video indexing pipelines, it leans on semantic novelty rather than geometry. This fails on tiny moving objects, vehicle swaps, transient events and drone revisits which GeoVARNet tries to preserve.

Another angle to reduce costs is detecting scene boundaries as found in video search, summarization and media indexing. If the scene did not change, why incur cost? But drone videos often have the same scene for different event which scene segmentation misses.


A more recent approach involves memory systems for Video LLMs that aim to improve with 

• StreamingVLM-style approaches

• memory-token approaches

• recurrent video transformers

These also obviate repetitive processing but by means of summary memory and sparse visual retention with a targeted transformer context length. While this reduces tokens, it does not retain as GeoVARNet does by compressing observations, places, revisits, and relationships at a higher-level abstraction.

The notable direction with NVIDIA VSS is the push towards Graph-RAG video systems where the architecture includes Graph-RAG, video search, agent workflows and video understanding thus generalizing for all video and not just urban drone videos. It completely avoids reprocessing the video to query the graph. 

This direction is a direct threat to GeoVARNet because it combines Embedding Novelty Sampling with Graph-RAG and Keyframe Selection and together it can already provide dramatic savings without introducing additional geometry decomposition and revisitation processing but GeoVARNet could demonstrate that it outperforms token-pruning, keyframe-selection, and embedding-novelty baselines at equal event-recall targets in terms of enrichment workload, VLM calls, storage growth, and event recall, When it does, it has a much stronger claim than reducing computation 


Tuesday, November 10, 2026

 PruneVid for aerial drone videos:

The previous article discussed using one of the three supported VLMs for PruneVid to improve token efficiency with the smallest model: PLLaVA. The other two are LLaVA-OneVision and ST-LLM (Liu et al., 2024b).  

ST-LLM feeds all raw, unpooled spatial-temporal tokens directly into the LLM backbone. For an egomotion camera where changes between consecutive frames are incremental, preserving the raw spatial-temporal progression allows the LLM to model subtle frame-to-frame displacements, motion cues, and panning trajectories. To prevent the massive context length from slowing down the system, it uses a dynamic masking strategy during training. This means it can drop redundant tokens where frames overlap significantly while forcing the model to retain a highly granular understanding of the continuous stream

The following implementation uses ST-LLM instead of the PLLaVA from the previous article:

"""Single-clip adapter for Visual-AI/PruneVid. Run from its repository root."""

"""ST-LLM official-API adapter; run from the ST-LLM repository root."""

import argparse

import json

from pathlib import Path

import time



def main():

    parser = argparse.ArgumentParser(description=__doc__)

    parser.add_argument('--video', required=True, help='Already-trimmed video clip')

    parser.add_argument('--ckpt-path', required=True)

    parser.add_argument('--qformer-path', required=True)

    parser.add_argument('--cfg-path', default='config/instructblipbase_stllm_conversation.yaml')

    parser.add_argument('--frames', type=int, default=16)

    parser.add_argument('--video-input', choices=['all', 'residual'], default='all')

    parser.add_argument('--max-new-tokens', type=int, default=160)

    parser.add_argument('--question', default='Describe terrain, structures, roads and moving objects. Distinguish camera motion from object motion. Do not invent GPS coordinates or distances.')

    parser.add_argument('--output', default='runs/drone_stllm.json')

    a = parser.parse_args()

    if a.frames < 16 or a.max_new_tokens < 1:

        parser.error('Use frames >=16 and max-new-tokens >=1.')

    for path in [a.video, a.qformer_path, a.cfg_path]:

        if not Path(path).is_file():

            parser.error('Missing file: ' + path)

    if not (Path(a.ckpt_path) / 'config.json').is_file():

        parser.error('ckpt-path must be the conversation_weight directory containing config.json.')

    import torch

    if not torch.cuda.is_available():

        raise RuntimeError('CUDA unavailable; fix PyTorch before loading ST-LLM.')

    if torch.cuda.get_device_capability(0) == (12, 0) and 'sm_120' not in torch.cuda.get_arch_list():

        raise RuntimeError('Install a Blackwell-compatible PyTorch wheel with sm_120 support.')

    x = torch.randn(64, 64, device='cuda')

    _ = x @ x

    torch.cuda.synchronize()

    from stllm.common.config import Config

    from stllm.common.registry import registry

    import stllm.models

    from stllm.conversation.conversation import Chat, CONV_instructblip_Vicuna0

    config_args = argparse.Namespace(cfg_path=a.cfg_path, options=None)

    cfg = Config(config_args)

    mc = cfg.model_cfg

    mc.ckpt = str(Path(a.ckpt_path).resolve())

    mc.llama_model = mc.ckpt

    mc.q_former_model = str(Path(a.qformer_path).resolve())

    mc.device_8bit = 0

    mc.use_grad_checkpoint = False

    # Keep residual modules when loading this residual-trained checkpoint.

    # Switch only its inference aggregation after loading.

    model_cls = registry.get_model_class(mc.arch)

    print('Loading official ST-LLM checkpoint; full FP16, no quantization/offload.', flush=True)

    model = model_cls.from_config(mc).to(device='cuda:0', dtype=torch.float16).eval()

    chat = Chat(model, device='cuda:0')

    chat.model.video_input = a.video_input

    conv = CONV_instructblip_Vicuna0.copy()

    images = []

    torch.cuda.reset_peak_memory_stats()

    start = time.perf_counter()

    with torch.inference_mode():

        chat.upload_video(str(Path(a.video).resolve()), conv, images,

                          num_frame=a.frames, text=a.question)

        chat.ask('###Human: ' + a.question + ' ###Assistant: ', conv)

        probe = conv.copy()

        probe.append_message(probe.roles[1], None)

        embeddings, _ = chat.get_context_emb_sim(probe, images, system=True)

        input_tokens = int(embeddings.shape[1])

        visual_tokens = int(images[0].shape[1])

        del embeddings

        context_limit = int(model.config.max_position_embeddings)

        required = input_tokens + a.max_new_tokens

        if required > context_limit:

            raise RuntimeError(f'Context overflow: {input_tokens} input + {a.max_new_tokens} output > {context_limit}. Reduce --frames; no silent truncation allowed.')

        answer, _ = chat.answer(conv=conv, img_list=images, do_sample=False,

                                num_beams=1, max_new_tokens=a.max_new_tokens,

                                max_length=context_limit)

    torch.cuda.synchronize()

    result = {'arguments': vars(a), 'answer': answer, 'visual_tokens': visual_tokens,

              'input_tokens': input_tokens, 'context_limit': context_limit,

              'encode_and_generate_seconds': time.perf_counter()-start,

              'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated()/2**30,

              'torch_version': torch.__version__, 'gpu': torch.cuda.get_device_name(0),

              'note': 'all retains all per-frame encoder/Q-Former output tokens, not raw pixels or all ViT patches; residual uses checkpoint global-local path.'}

    dest = Path(a.output)

    dest.parent.mkdir(parents=True, exist_ok=True)

    dest.write_text(json.dumps(result, indent=2), encoding='utf-8')

    print(answer)



if __name__ == '__main__':

    main()


Some caveat remain as before:

1. Libraries I used:

Python 3.10.22

torch 2.2.1+cu118

transformers 4.40.2

peft 0.10.0

accelerate 0.26.1

safetensors 0.4.2

mmcv 1.7.2

decord 0.6.0

moviepy 1.0.3

imageio 2.27.0

matplotlib 3.8.3

protobuf 4.23.4

sentencepiece 0.2.0

einops 0.6.1

2. 

There is a checkpoint/code compatibility issue.


prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint. 

and there are three checkpoints involved which is suspicious:

stllm-blackwell

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.


3. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash / ST-LLM compatibility issue.




#codingexercise: CodingExercise-11-10-2026.docx