The previous article described an implementation. The following are from test runs:

1. Libraries I used:
   

Python 3.10.22

torch 2.2.1+cu118

transformers 4.40.2

peft 0.10.0

accelerate 0.26.1

safetensors 0.4.2

mmcv 1.7.2

decord 0.6.0

moviepy 1.0.3

imageio 2.27.0

matplotlib 3.8.3

protobuf 4.23.4

sentencepiece 0.2.0

einops 0.6.1

   2. There is a checkpoint/code compatibility issue.

prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint.

and there are three checkpoints involved which is suspicious:
pllava-7b

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.

 

3.  3. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash / pllava compatibility issue.

 PruneVid for aerial drone videos:

"""Single-clip adapter for Visual-AI/PruneVid. Run from its repository root."""
import argparse
import json
import math
import sys
import time
from pathlib import Path


def main():
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument('--video', required=True)
    p.add_argument('--output', default='drone_result')
    p.add_argument('--start', type=float, default=0)
    p.add_argument('--seconds', type=float, default=20)
    p.add_argument('--frames', type=int, default=16)
    p.add_argument('--prepare-only', action='store_true')
    p.add_argument('--model-dir', default='MODELS/pllava-7b')
    p.add_argument('--weight-dir')
    p.add_argument('--lora-alpha', type=int, default=14)
    p.add_argument('--question', default='Describe the visible terrain, structures, roads and moving objects. Distinguish camera motion from object motion. State uncertainty; do not invent GPS coordinates or metric distances.')
    p.add_argument('--alpha', type=float, default=0.4)
    p.add_argument('--tau', type=float, default=0.8)
    p.add_argument('--cluster-ratio', type=float, default=0.5)
    p.add_argument('--temporal-segment-ratio', type=float, default=0.25)
    p.add_argument('--selected-layer', type=int, default=10)
    p.add_argument('--max-new-tokens', type=int, default=160)
    a = p.parse_args()
    if a.frames < 4 or a.seconds <= 0 or a.start < 0:
        p.error('Require frames >= 4, seconds > 0 and start >= 0.')
    for name in ('alpha', 'cluster_ratio', 'temporal_segment_ratio'):
        if not 0 < getattr(a, name) <= 1:
            p.error(name + ' must be in (0, 1].')
    if not -1 <= a.tau <= 1:
        p.error('tau must be in [-1, 1].')
    import cv2
    import numpy as np
    from PIL import Image, ImageDraw
    video = Path(a.video).resolve()
    if not video.is_file():
        raise FileNotFoundError(video)
    target = Path(a.output)
    target.mkdir(parents=True, exist_ok=True)
    cap = cv2.VideoCapture(str(video))
    try:
        if not cap.isOpened():
            raise RuntimeError('Cannot open video; check path and codec.')
        fps = cap.get(cv2.CAP_PROP_FPS)
        total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
        if not math.isfinite(fps) or fps <= 0 or total <= 0:
            raise RuntimeError('Invalid FPS/frame count; transcode video before sampling.')
        first = int(a.start * fps)
        last = min(total, int((a.start + a.seconds) * fps)) - 1
        if last - first + 1 < a.frames:
            raise RuntimeError('Requested clip has fewer source frames than --frames.')
        indices = np.linspace(first, last, a.frames).round().astype(int)
        images, samples = [], []
        for i, idx in enumerate(indices):
            cap.set(cv2.CAP_PROP_POS_FRAMES, int(idx))
            ok, bgr = cap.read()
            if not ok:
                raise RuntimeError(f'Cannot decode source frame {idx}.')
            image = Image.fromarray(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))
            images.append(image)
            image.save(target / f'frame_{i:03d}.jpg')
            samples.append({'frame_index': int(idx), 'nominal_seconds': float(idx / fps)})
    finally:
        cap.release()
    sheet = Image.new('RGB', (4 * 320, math.ceil(a.frames / 4) * 205), 'white')
    draw = ImageDraw.Draw(sheet)
    for i, image in enumerate(images):
        thumb = image.copy()
        thumb.thumbnail((320, 180))
        x, y = (i % 4) * 320, (i // 4) * 205
        sheet.paste(thumb, (x, y))
        draw.text((x + 4, y + 182), f'{samples[i]["nominal_seconds"]:.2f}s', fill='black')
    sheet.save(target / 'contact_sheet.jpg')
    metadata = {'arguments': vars(a), 'video': str(video), 'fps': fps,
                'timestamps_are_nominal': True, 'samples': samples}
    (target / 'sampling.json').write_text(json.dumps(metadata, indent=2), encoding='utf-8')
    if a.prepare_only:
        print('Prepared sampled frames and contact sheet; no model loaded.')
        return
    import torch
    if not torch.cuda.is_available():
        raise RuntimeError('Official-model path requires working NVIDIA CUDA in this adapter. Use --prepare-only for CPU preprocessing.')
    if not torch.cuda.is_bf16_supported():
        raise RuntimeError('Official loader uses bfloat16; this CUDA device does not report BF16 support.')
    root = Path.cwd()
    if not (root / 'tasks/eval/model_utils.py').is_file():
        raise RuntimeError('Run from the cloned PruneVid root, with this script copied there.')
    sys.path.insert(0, str(root))
    from tasks.eval.model_utils import load_pllava
    from tasks.eval.eval_utils import conv_templates
    model_dir = str(Path(a.model_dir).resolve())
    weight_dir = str(Path(a.weight_dir or a.model_dir).resolve())
    if 'llava' not in model_dir:
        raise RuntimeError('Official loader selects PLLaVA by lowercase "llava" in model path; name the directory pllava-7b.')
    for directory in (model_dir, weight_dir):
        if not (Path(directory) / 'config.json').is_file():
            raise FileNotFoundError(f'Missing config.json in {directory}')
    torch.manual_seed(0)
    model, processor = load_pllava(
        model_dir, num_frames=a.frames, use_lora=True, weight_dir=weight_dir,
        lora_alpha=a.lora_alpha, pooling_shape=(a.frames, 12, 12),
        selected_layer=a.selected_layer, alpha=a.alpha, tau=a.tau,
        cluster_ratio=a.cluster_ratio, temporal_segment_ratio=a.temporal_segment_ratio)
    model = model.to('cuda').eval()
    conv = conv_templates['plain'].copy()
    conv.user_query(a.question, is_mm=True)
    inputs = processor(text=conv.get_prompt(), images=images, return_tensors='pt').to(model.device)
    if 'pixel_values' in inputs and inputs['pixel_values'] is not None:
        inputs['pixel_values'] = inputs['pixel_values'].to(dtype=next(model.parameters()).dtype)
    torch.cuda.synchronize()
    torch.cuda.reset_peak_memory_stats()
    started = time.perf_counter()
    with torch.inference_mode():
        tokens = model.generate(**inputs, media_type='video', do_sample=False,
                                max_new_tokens=a.max_new_tokens, use_cache=True)
    torch.cuda.synchronize()
    decoded = processor.batch_decode(tokens, skip_special_tokens=True,
                                     clean_up_tokenization_spaces=False)[0]
    answer = decoded.split(conv.roles[-1])[-1].removesuffix('</s>').strip()
    result = {**metadata, 'answer': answer, 'raw_decoded_text': decoded,
              'generation_seconds': time.perf_counter() - started,
              'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated() / 2**30,
              'gpu': torch.cuda.get_device_name(0), 'torch_version': torch.__version__,
              'note': 'No measured retained-token count; no baseline or accuracy claim.'}
    (target / 'result.json').write_text(json.dumps(result, indent=2), encoding='utf-8')
    print(answer)


if __name__ == '__main__':
    main()

Continued from previous post

 Just as AI observability evolved from model monitoring into decision accountability, perception observability is likely to evolve from object detection into world-model accountability. The systems that succeed will not simply identify what is visible. They will explain how every conclusion emerged, how it changed over time, and how an investigator can independently verify it. Even with an LLM-as-a-judge paradigm, the validation must remain independent.

The ability to defend a perception-derived belief through a complete chain of evidence becomes the defining characteristic of trustworthy aerial intelligence.

Continued from previous post

 

A related concept is spatial auditability.

Drone imagery is inherently geometric. Objects occupy locations within a coordinate system. Observations occur from different viewpoints, altitudes, sensor poses, and environmental conditions. Spatial auditability therefore requires the ability to justify not only that something exists, but where it exists. When a perception system reports that a vehicle is located at a particular coordinate, the system should retain the geometric chain that produced that estimate including camera parameters, pose estimates, registration transformations, geo-referencing models, local uncertainty estimates, and alignment quality metrics. This information helps location change from an assertion to evidence-supported knowledge.

Perception consistency monitoring extends perception observation and helps preserve observation lineage. Application observability tracks CPU utilization, latency, and service health; AI observability tracks hallucinations, latency, cost, and accuracy. Perception observability adds health signals specific to the perception process: observation drift, spatial inconsistency, temporal instability, identity fragmentation, registration degradation, evidence conflicts, and world-model uncertainty.

Imagine a drone corridor inspection system. For months, it consistently identifies pole locations within a few centimeters. Suddenly localization quality deteriorates because GPS quality worsens, illumination changes, or a visual registration model begins drifting.

A perception-observability layer would detect the degradation before analysts begin consuming corrupted outputs. Instead of waiting for incorrect conclusions, the system continuously validates the coherence of its own world model.

Another essential capability is revisiting awareness. Most real-world environments are observed repeatedly. Buildings are revisited. Roads are revisited. Assets are revisited. Infrastructure is revisited. Traditional computer vision often treats each revisit as a new problem. Perception observability treats revisits as opportunities to strengthen belief.

When a drone returns to a location, the system should compare current observations against historical knowledge. It should record whether observations reinforce, modify, or contradict prior beliefs. Each revisit becomes an audit event. The resulting system develops something resembling memory. Instead of a sequence of disconnected detections, it maintains a continuously evolving understanding of the environment. This becomes even more powerful when multiple sensing modalities are introduced. For example, modern aerial systems increasingly combine RGB cameras, thermal sensors, multispectral imagery, SAR, LiDAR, inertial measurements, and external geospatial databases.

Perception observability must also answer: Which sensor contributed to what evidence?

When a thermal anomaly triggers an alert, the system should indicate whether the conclusion relied solely on thermal data, was corroborated by visual imagery, was strengthened by contextual information, or faced contradictory evidence from other sources. Cross-modal evidence tracking makes sensor fusion transparent and inspectable. As perception systems become autonomous, auditability becomes a requirement rather than merely a useful capability. Future systems will increasingly trigger operational decisions automatically. They may dispatch inspectors, prioritize maintenance activities, initiate planning actions, direct robotic assets, or support public-safety decisions.

When decisions carry financial, operational, or safety consequences, stakeholders inevitably ask:

What did the system know?

When did the system know it?

Why did the system believe it?

What evidruly audit questions. The future of aerial intelligence may be defined by the ability to preserve and communicate evidence. In that future, the perception pipeline becomes a continuously auditable record of how observations become knowledge.

Continued from previous

Perception observability replaces confidence-centric thinking with evidence-centric thinking.

Every observation becomes traceable to the imagery, geometry, trajectory, sensor conditions, and prior observations that contributed to it. Instead of merely stating that a truck exists at a location, the system can demonstrate that the object was observed in fourteen separate passes, viewed from multiple headings, tracked across three days, localized through consistent geometric registration, and repeatedly associated with a particular operational area. The belief becomes explainable because its provenance is visible.

At the heart of perception observability lies the concept of observation lineage.

Observation lineage is analogous to data lineage in analytics systems. Just as data lineage traces a dashboard metric back to databases and transformations, observation lineage traces a world-state assertion back to the sensor evidence that created it.

Consider a simple statement:

"A new storage building appeared in Sector C."

A perception-observable system should be capable of revealing:

Which missions contributed evidence?

Which frames suggested change?

Which detections participated in?

Which registration models aligned imagery across time?

Which change detection algorithms trigger alerts?

Which observations contradicted the conclusion?

How confidence evolved as additional evidence accumulated.

The conclusion should emerge as the last link in an auditable chain of evidence.

This naturally leads to the second major pillar: temporal auditability.

Most vision systems evaluate frames independently. Humans, however, reason temporally. If a building suddenly appears in one image and disappears in the next ten observations, we become skeptical. We seek continuity.

Temporal auditability captures that intuition computationally.

Every world-state belief should maintain a history describing how it evolved. An analyst should be able to inspect the lifecycle of any object and answer questions such as:

When was this object first observed?

When did confidence increase?

When did evidence weaken?

When did the classification of an object change?

When was the object last confirmed?

What observations support its continued existence?

In a mature perception-observability framework, every entity becomes a living record rather than a static detection.

 Perception observability is making an autonomous sensing system explain itself. Just as application observability helps engineers understand what software is doing and AI observability helps engineers understand what models are doing, perception observability seeks to reveal what a sensing system believes about the world, why it believes it, how that belief evolved over time, and what evidence supports each conclusion.

That distinction is relevant to the world of drones. Most current AI observability platforms focus on the internals of a model. They monitor prompts, token usage, latency, hallucinations, outputs, and reasoning traces. They answer questions such as: Why did the model produce this answer? Why did the agent call this tool? Why did performance degrade after deployment?

Aerial drone systems face a fundamentally different challenge. As they struggle to be accurate in precision and recall, they must be trusted to perceive reality. A drone observing a city, a construction site, a power corridor, a disaster zone, or a battlefield is continuously transforming photons into beliefs about the physical world. In this context, an auditor could ask why does the system believes that this object exists at this location, with this identity, at this moment in time?

Perception observability attempts to answer that question.

Imagine a drone flying over an industrial facility every day for six months. During that time, it detects vehicles, buildings, equipment, containers, roads, vegetation changes, construction progress, temporary structures, and human activity. Traditional analytics pipelines often produce outputs as isolated events. A vehicle is detected. A building is classified. A region is segmented. An alert is issued.

Yet when an analyst asks where that conclusion came from, the system often has no meaningful answer beyond confidence scores. Confidence, however, is not evidence. A score of 0.97 does not state why the conclusion is true.

 Cutting Tokens in Drone Video Analytics

The inference cost of aerial drone images and video even with sampling often runs into thousands of tokens and almost always repeated per query given the long tail of such analytics. GeoVARNet and our proposal asked: "Can representation-guided enrichment reduce computation without materially reducing event recall?" [1]. Players like NVIDIA are also asking similar questions but from a different perspective.

The Video Search and Summarization (VSS) blueprint is NVIDIA's reference architecture for video agents that perform:

• video search,

• visual Q&A,

• summarization,

• alerting,

• report generation,

using VLMs, LLMs, retrieval systems, and video analytics pipelines. They introduced Adaptive Efficient Video Sampling (EVS) that dynamically prunes unchanged visual patches per frame and batches VLM work around moments of activity.[3]

Their reported outcomes include:

• ~80% fewer VLM input tokens for a 60 minute video,

• ~50% faster summarization,

• 17% lower alert-contextualization latency,

• 46% higher concurrent real-time VLM stream capacity [3]

Although the benchmarks for drone video analytics such as UAVBench and EZBenchmark are diverse and require reprocessing, the EVS matters over traditional VLM pipelines that often progress through ingestion, frame sampling, VLM inference, Caption captures, and image embeddings and typically repeat that for every input even though almost nothing changed. EVS attempts to suppress that waste by detecting regions that are largely unchanged and avoiding feeding redundant visual information into the VLM. With their introduction of change detector, they reprocess only changed regions/chunks. This is referred to as token-pruning. GeoVARNet, on the other hand, engages in representation-guided enrichment. So, it can answer questions on an “evidence” basis rather than at “pixel” basis with a much broader definition of evidence and decision variables such as persistence, covisibility, revisits, place identity, entity history, observation coverage and rare-event likelihood. These are all first-class architectural concepts in GeoVARNet. 

For example, if a parking lot is observed six times, EVS sees parking lot looks mostly unchanged and reduces token usage while GeoVARNet could additionally infer same place, same appearance, same persistent state, already enriched, and high confidence. This would help avoid VLM calls, embedding calls, graph expansion, retrieval enrichment, and storage growth. That is a stronger form of reduction than token pruning alone.

There are others pursuing similar ideas though not with the same objectives. For example, Keyframe selection systems such as ORB-SLAM3, PKS (photogrammetric key-frame selection) and OpenVSLAM decide to only process frames that contribute new geometry. While GeoVARNet suggests using PKS, those techniques are for geometry redundancy, not semantic redundancy.

Modern video-RAG systems have a workflow that uses frame to embedding conversion and then determine distance from prior embedding to decide whether to retain or discard the frame and while this does not reduce the embedding generation cost for those frame and is used by many commercial video indexing pipelines, it leans on semantic novelty rather than geometry. This fails on tiny moving objects, vehicle swaps, transient events and drone revisits which GeoVARNet tries to preserve.

Another angle to reduce costs is detecting scene boundaries as found in video search, summarization and media indexing. If the scene did not change, why incur cost? But drone videos often have the same scene for different event which scene segmentation misses.

A more recent approach involves memory systems for Video LLMs that aim to improve with 

• StreamingVLM-style approaches

• memory-token approaches

• recurrent video transformers

These also obviate repetitive processing but by means of summary memory and sparse visual retention with a targeted transformer context length. While this reduces tokens, it does not retain as GeoVARNet does by compressing observations, places, revisits, and relationships at a higher-level abstraction.

The notable direction with NVIDIA VSS is the push towards Graph-RAG video systems where the architecture includes Graph-RAG, video search, agent workflows and video understanding thus generalizing for all video and not just urban drone videos. It completely avoids reprocessing the video to query the graph. 

This direction is a direct threat to GeoVARNet because it combines Embedding Novelty Sampling with Graph-RAG and Keyframe Selection and together it can already provide dramatic savings without introducing additional geometry decomposition and revisitation processing but GeoVARNet could demonstrate that it outperforms token-pruning, keyframe-selection, and embedding-novelty baselines at equal event-recall targets in terms of enrichment workload, VLM calls, storage growth, and event recall, When it does, it has a much stronger claim than reducing computation