PruneVid for aerial drone videos:

The previous article discussed using one of the three supported VLMs for PruneVid to improve token efficiency with the smallest model: PLLaVA. The other two are LLaVA-OneVision and ST-LLM (Liu et al., 2024b).  

ST-LLM feeds all raw, unpooled spatial-temporal tokens directly into the LLM backbone. For an egomotion camera where changes between consecutive frames are incremental, preserving the raw spatial-temporal progression allows the LLM to model subtle frame-to-frame displacements, motion cues, and panning trajectories. To prevent the massive context length from slowing down the system, it uses a dynamic masking strategy during training. This means it can drop redundant tokens where frames overlap significantly while forcing the model to retain a highly granular understanding of the continuous stream

The following implementation uses ST-LLM instead of the PLLaVA from the previous article:

"""Single-clip adapter for Visual-AI/PruneVid. Run from its repository root."""

"""ST-LLM official-API adapter; run from the ST-LLM repository root."""

import argparse

import json

from pathlib import Path

import time

def main():

    parser = argparse.ArgumentParser(description=__doc__)

    parser.add_argument('--video', required=True, help='Already-trimmed video clip')

    parser.add_argument('--ckpt-path', required=True)

    parser.add_argument('--qformer-path', required=True)

    parser.add_argument('--cfg-path', default='config/instructblipbase_stllm_conversation.yaml')

    parser.add_argument('--frames', type=int, default=16)

    parser.add_argument('--video-input', choices=['all', 'residual'], default='all')

    parser.add_argument('--max-new-tokens', type=int, default=160)

    parser.add_argument('--question', default='Describe terrain, structures, roads and moving objects. Distinguish camera motion from object motion. Do not invent GPS coordinates or distances.')

    parser.add_argument('--output', default='runs/drone_stllm.json')

    a = parser.parse_args()

    if a.frames < 16 or a.max_new_tokens < 1:

        parser.error('Use frames >=16 and max-new-tokens >=1.')

    for path in [a.video, a.qformer_path, a.cfg_path]:

        if not Path(path).is_file():

            parser.error('Missing file: ' + path)

    if not (Path(a.ckpt_path) / 'config.json').is_file():

        parser.error('ckpt-path must be the conversation_weight directory containing config.json.')

    import torch

    if not torch.cuda.is_available():

        raise RuntimeError('CUDA unavailable; fix PyTorch before loading ST-LLM.')

    if torch.cuda.get_device_capability(0) == (12, 0) and 'sm_120' not in torch.cuda.get_arch_list():

        raise RuntimeError('Install a Blackwell-compatible PyTorch wheel with sm_120 support.')

    x = torch.randn(64, 64, device='cuda')

    _ = x @ x

    torch.cuda.synchronize()

    from stllm.common.config import Config

    from stllm.common.registry import registry

    import stllm.models

    from stllm.conversation.conversation import Chat, CONV_instructblip_Vicuna0

    config_args = argparse.Namespace(cfg_path=a.cfg_path, options=None)

    cfg = Config(config_args)

    mc = cfg.model_cfg

    mc.ckpt = str(Path(a.ckpt_path).resolve())

    mc.llama_model = mc.ckpt

    mc.q_former_model = str(Path(a.qformer_path).resolve())

    mc.device_8bit = 0

    mc.use_grad_checkpoint = False

    # Keep residual modules when loading this residual-trained checkpoint.

    # Switch only its inference aggregation after loading.

    model_cls = registry.get_model_class(mc.arch)

    print('Loading official ST-LLM checkpoint; full FP16, no quantization/offload.', flush=True)

    model = model_cls.from_config(mc).to(device='cuda:0', dtype=torch.float16).eval()

    chat = Chat(model, device='cuda:0')

    chat.model.video_input = a.video_input

    conv = CONV_instructblip_Vicuna0.copy()

    images = []

    torch.cuda.reset_peak_memory_stats()

    start = time.perf_counter()

    with torch.inference_mode():

        chat.upload_video(str(Path(a.video).resolve()), conv, images,

                          num_frame=a.frames, text=a.question)

        chat.ask('###Human: ' + a.question + ' ###Assistant: ', conv)

        probe = conv.copy()

        probe.append_message(probe.roles[1], None)

        embeddings, _ = chat.get_context_emb_sim(probe, images, system=True)

        input_tokens = int(embeddings.shape[1])

        visual_tokens = int(images[0].shape[1])

        del embeddings

        context_limit = int(model.config.max_position_embeddings)

        required = input_tokens + a.max_new_tokens

        if required > context_limit:

            raise RuntimeError(f'Context overflow: {input_tokens} input + {a.max_new_tokens} output > {context_limit}. Reduce --frames; no silent truncation allowed.')

        answer, _ = chat.answer(conv=conv, img_list=images, do_sample=False,

                                num_beams=1, max_new_tokens=a.max_new_tokens,

                                max_length=context_limit)

    torch.cuda.synchronize()

    result = {'arguments': vars(a), 'answer': answer, 'visual_tokens': visual_tokens,

              'input_tokens': input_tokens, 'context_limit': context_limit,

              'encode_and_generate_seconds': time.perf_counter()-start,

              'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated()/2**30,

              'torch_version': torch.__version__, 'gpu': torch.cuda.get_device_name(0),

              'note': 'all retains all per-frame encoder/Q-Former output tokens, not raw pixels or all ViT patches; residual uses checkpoint global-local path.'}

    dest = Path(a.output)

    dest.parent.mkdir(parents=True, exist_ok=True)

    dest.write_text(json.dumps(result, indent=2), encoding='utf-8')

    print(answer)

if __name__ == '__main__':

    main()

Some caveat remain as before:

1. Libraries I used:

Python 3.10.22

torch 2.2.1+cu118

transformers 4.40.2

peft 0.10.0

accelerate 0.26.1

safetensors 0.4.2

mmcv 1.7.2

decord 0.6.0

moviepy 1.0.3

imageio 2.27.0

matplotlib 3.8.3

protobuf 4.23.4

sentencepiece 0.2.0

einops 0.6.1

2. 

There is a checkpoint/code compatibility issue.

prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint. 

and there are three checkpoints involved which is suspicious:

stllm-blackwell

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.

3. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash / ST-LLM compatibility issue.

 The previous article described an implementation. The following are from test runs:

1. Libraries I used:
   

Python 3.10.22

torch 2.2.1+cu118

transformers 4.40.2

peft 0.10.0

accelerate 0.26.1

safetensors 0.4.2

mmcv 1.7.2

decord 0.6.0

moviepy 1.0.3

imageio 2.27.0

matplotlib 3.8.3

protobuf 4.23.4

sentencepiece 0.2.0

einops 0.6.1

   2. There is a checkpoint/code compatibility issue.

prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint.

and there are three checkpoints involved which is suspicious:
pllava-7b

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.

 

3.  3. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash / pllava compatibility issue.

 PruneVid for aerial drone videos:

"""Single-clip adapter for Visual-AI/PruneVid. Run from its repository root."""
import argparse
import json
import math
import sys
import time
from pathlib import Path


def main():
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument('--video', required=True)
    p.add_argument('--output', default='drone_result')
    p.add_argument('--start', type=float, default=0)
    p.add_argument('--seconds', type=float, default=20)
    p.add_argument('--frames', type=int, default=16)
    p.add_argument('--prepare-only', action='store_true')
    p.add_argument('--model-dir', default='MODELS/pllava-7b')
    p.add_argument('--weight-dir')
    p.add_argument('--lora-alpha', type=int, default=14)
    p.add_argument('--question', default='Describe the visible terrain, structures, roads and moving objects. Distinguish camera motion from object motion. State uncertainty; do not invent GPS coordinates or metric distances.')
    p.add_argument('--alpha', type=float, default=0.4)
    p.add_argument('--tau', type=float, default=0.8)
    p.add_argument('--cluster-ratio', type=float, default=0.5)
    p.add_argument('--temporal-segment-ratio', type=float, default=0.25)
    p.add_argument('--selected-layer', type=int, default=10)
    p.add_argument('--max-new-tokens', type=int, default=160)
    a = p.parse_args()
    if a.frames < 4 or a.seconds <= 0 or a.start < 0:
        p.error('Require frames >= 4, seconds > 0 and start >= 0.')
    for name in ('alpha', 'cluster_ratio', 'temporal_segment_ratio'):
        if not 0 < getattr(a, name) <= 1:
            p.error(name + ' must be in (0, 1].')
    if not -1 <= a.tau <= 1:
        p.error('tau must be in [-1, 1].')
    import cv2
    import numpy as np
    from PIL import Image, ImageDraw
    video = Path(a.video).resolve()
    if not video.is_file():
        raise FileNotFoundError(video)
    target = Path(a.output)
    target.mkdir(parents=True, exist_ok=True)
    cap = cv2.VideoCapture(str(video))
    try:
        if not cap.isOpened():
            raise RuntimeError('Cannot open video; check path and codec.')
        fps = cap.get(cv2.CAP_PROP_FPS)
        total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
        if not math.isfinite(fps) or fps <= 0 or total <= 0:
            raise RuntimeError('Invalid FPS/frame count; transcode video before sampling.')
        first = int(a.start * fps)
        last = min(total, int((a.start + a.seconds) * fps)) - 1
        if last - first + 1 < a.frames:
            raise RuntimeError('Requested clip has fewer source frames than --frames.')
        indices = np.linspace(first, last, a.frames).round().astype(int)
        images, samples = [], []
        for i, idx in enumerate(indices):
            cap.set(cv2.CAP_PROP_POS_FRAMES, int(idx))
            ok, bgr = cap.read()
            if not ok:
                raise RuntimeError(f'Cannot decode source frame {idx}.')
            image = Image.fromarray(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))
            images.append(image)
            image.save(target / f'frame_{i:03d}.jpg')
            samples.append({'frame_index': int(idx), 'nominal_seconds': float(idx / fps)})
    finally:
        cap.release()
    sheet = Image.new('RGB', (4 * 320, math.ceil(a.frames / 4) * 205), 'white')
    draw = ImageDraw.Draw(sheet)
    for i, image in enumerate(images):
        thumb = image.copy()
        thumb.thumbnail((320, 180))
        x, y = (i % 4) * 320, (i // 4) * 205
        sheet.paste(thumb, (x, y))
        draw.text((x + 4, y + 182), f'{samples[i]["nominal_seconds"]:.2f}s', fill='black')
    sheet.save(target / 'contact_sheet.jpg')
    metadata = {'arguments': vars(a), 'video': str(video), 'fps': fps,
                'timestamps_are_nominal': True, 'samples': samples}
    (target / 'sampling.json').write_text(json.dumps(metadata, indent=2), encoding='utf-8')
    if a.prepare_only:
        print('Prepared sampled frames and contact sheet; no model loaded.')
        return
    import torch
    if not torch.cuda.is_available():
        raise RuntimeError('Official-model path requires working NVIDIA CUDA in this adapter. Use --prepare-only for CPU preprocessing.')
    if not torch.cuda.is_bf16_supported():
        raise RuntimeError('Official loader uses bfloat16; this CUDA device does not report BF16 support.')
    root = Path.cwd()
    if not (root / 'tasks/eval/model_utils.py').is_file():
        raise RuntimeError('Run from the cloned PruneVid root, with this script copied there.')
    sys.path.insert(0, str(root))
    from tasks.eval.model_utils import load_pllava
    from tasks.eval.eval_utils import conv_templates
    model_dir = str(Path(a.model_dir).resolve())
    weight_dir = str(Path(a.weight_dir or a.model_dir).resolve())
    if 'llava' not in model_dir:
        raise RuntimeError('Official loader selects PLLaVA by lowercase "llava" in model path; name the directory pllava-7b.')
    for directory in (model_dir, weight_dir):
        if not (Path(directory) / 'config.json').is_file():
            raise FileNotFoundError(f'Missing config.json in {directory}')
    torch.manual_seed(0)
    model, processor = load_pllava(
        model_dir, num_frames=a.frames, use_lora=True, weight_dir=weight_dir,
        lora_alpha=a.lora_alpha, pooling_shape=(a.frames, 12, 12),
        selected_layer=a.selected_layer, alpha=a.alpha, tau=a.tau,
        cluster_ratio=a.cluster_ratio, temporal_segment_ratio=a.temporal_segment_ratio)
    model = model.to('cuda').eval()
    conv = conv_templates['plain'].copy()
    conv.user_query(a.question, is_mm=True)
    inputs = processor(text=conv.get_prompt(), images=images, return_tensors='pt').to(model.device)
    if 'pixel_values' in inputs and inputs['pixel_values'] is not None:
        inputs['pixel_values'] = inputs['pixel_values'].to(dtype=next(model.parameters()).dtype)
    torch.cuda.synchronize()
    torch.cuda.reset_peak_memory_stats()
    started = time.perf_counter()
    with torch.inference_mode():
        tokens = model.generate(**inputs, media_type='video', do_sample=False,
                                max_new_tokens=a.max_new_tokens, use_cache=True)
    torch.cuda.synchronize()
    decoded = processor.batch_decode(tokens, skip_special_tokens=True,
                                     clean_up_tokenization_spaces=False)[0]
    answer = decoded.split(conv.roles[-1])[-1].removesuffix('</s>').strip()
    result = {**metadata, 'answer': answer, 'raw_decoded_text': decoded,
              'generation_seconds': time.perf_counter() - started,
              'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated() / 2**30,
              'gpu': torch.cuda.get_device_name(0), 'torch_version': torch.__version__,
              'note': 'No measured retained-token count; no baseline or accuracy claim.'}
    (target / 'result.json').write_text(json.dumps(result, indent=2), encoding='utf-8')
    print(answer)


if __name__ == '__main__':
    main()

Continued from previous post

 Just as AI observability evolved from model monitoring into decision accountability, perception observability is likely to evolve from object detection into world-model accountability. The systems that succeed will not simply identify what is visible. They will explain how every conclusion emerged, how it changed over time, and how an investigator can independently verify it. Even with an LLM-as-a-judge paradigm, the validation must remain independent.

The ability to defend a perception-derived belief through a complete chain of evidence becomes the defining characteristic of trustworthy aerial intelligence.

Continued from previous post

 

A related concept is spatial auditability.

Drone imagery is inherently geometric. Objects occupy locations within a coordinate system. Observations occur from different viewpoints, altitudes, sensor poses, and environmental conditions. Spatial auditability therefore requires the ability to justify not only that something exists, but where it exists. When a perception system reports that a vehicle is located at a particular coordinate, the system should retain the geometric chain that produced that estimate including camera parameters, pose estimates, registration transformations, geo-referencing models, local uncertainty estimates, and alignment quality metrics. This information helps location change from an assertion to evidence-supported knowledge.

Perception consistency monitoring extends perception observation and helps preserve observation lineage. Application observability tracks CPU utilization, latency, and service health; AI observability tracks hallucinations, latency, cost, and accuracy. Perception observability adds health signals specific to the perception process: observation drift, spatial inconsistency, temporal instability, identity fragmentation, registration degradation, evidence conflicts, and world-model uncertainty.

Imagine a drone corridor inspection system. For months, it consistently identifies pole locations within a few centimeters. Suddenly localization quality deteriorates because GPS quality worsens, illumination changes, or a visual registration model begins drifting.

A perception-observability layer would detect the degradation before analysts begin consuming corrupted outputs. Instead of waiting for incorrect conclusions, the system continuously validates the coherence of its own world model.

Another essential capability is revisiting awareness. Most real-world environments are observed repeatedly. Buildings are revisited. Roads are revisited. Assets are revisited. Infrastructure is revisited. Traditional computer vision often treats each revisit as a new problem. Perception observability treats revisits as opportunities to strengthen belief.

When a drone returns to a location, the system should compare current observations against historical knowledge. It should record whether observations reinforce, modify, or contradict prior beliefs. Each revisit becomes an audit event. The resulting system develops something resembling memory. Instead of a sequence of disconnected detections, it maintains a continuously evolving understanding of the environment. This becomes even more powerful when multiple sensing modalities are introduced. For example, modern aerial systems increasingly combine RGB cameras, thermal sensors, multispectral imagery, SAR, LiDAR, inertial measurements, and external geospatial databases.

Perception observability must also answer: Which sensor contributed to what evidence?

When a thermal anomaly triggers an alert, the system should indicate whether the conclusion relied solely on thermal data, was corroborated by visual imagery, was strengthened by contextual information, or faced contradictory evidence from other sources. Cross-modal evidence tracking makes sensor fusion transparent and inspectable. As perception systems become autonomous, auditability becomes a requirement rather than merely a useful capability. Future systems will increasingly trigger operational decisions automatically. They may dispatch inspectors, prioritize maintenance activities, initiate planning actions, direct robotic assets, or support public-safety decisions.

When decisions carry financial, operational, or safety consequences, stakeholders inevitably ask:

What did the system know?

When did the system know it?

Why did the system believe it?

What evidruly audit questions. The future of aerial intelligence may be defined by the ability to preserve and communicate evidence. In that future, the perception pipeline becomes a continuously auditable record of how observations become knowledge.

Continued from previous

Perception observability replaces confidence-centric thinking with evidence-centric thinking.

Every observation becomes traceable to the imagery, geometry, trajectory, sensor conditions, and prior observations that contributed to it. Instead of merely stating that a truck exists at a location, the system can demonstrate that the object was observed in fourteen separate passes, viewed from multiple headings, tracked across three days, localized through consistent geometric registration, and repeatedly associated with a particular operational area. The belief becomes explainable because its provenance is visible.

At the heart of perception observability lies the concept of observation lineage.

Observation lineage is analogous to data lineage in analytics systems. Just as data lineage traces a dashboard metric back to databases and transformations, observation lineage traces a world-state assertion back to the sensor evidence that created it.

Consider a simple statement:

"A new storage building appeared in Sector C."

A perception-observable system should be capable of revealing:

Which missions contributed evidence?

Which frames suggested change?

Which detections participated in?

Which registration models aligned imagery across time?

Which change detection algorithms trigger alerts?

Which observations contradicted the conclusion?

How confidence evolved as additional evidence accumulated.

The conclusion should emerge as the last link in an auditable chain of evidence.

This naturally leads to the second major pillar: temporal auditability.

Most vision systems evaluate frames independently. Humans, however, reason temporally. If a building suddenly appears in one image and disappears in the next ten observations, we become skeptical. We seek continuity.

Temporal auditability captures that intuition computationally.

Every world-state belief should maintain a history describing how it evolved. An analyst should be able to inspect the lifecycle of any object and answer questions such as:

When was this object first observed?

When did confidence increase?

When did evidence weaken?

When did the classification of an object change?

When was the object last confirmed?

What observations support its continued existence?

In a mature perception-observability framework, every entity becomes a living record rather than a static detection.

 Perception observability is making an autonomous sensing system explain itself. Just as application observability helps engineers understand what software is doing and AI observability helps engineers understand what models are doing, perception observability seeks to reveal what a sensing system believes about the world, why it believes it, how that belief evolved over time, and what evidence supports each conclusion.

That distinction is relevant to the world of drones. Most current AI observability platforms focus on the internals of a model. They monitor prompts, token usage, latency, hallucinations, outputs, and reasoning traces. They answer questions such as: Why did the model produce this answer? Why did the agent call this tool? Why did performance degrade after deployment?

Aerial drone systems face a fundamentally different challenge. As they struggle to be accurate in precision and recall, they must be trusted to perceive reality. A drone observing a city, a construction site, a power corridor, a disaster zone, or a battlefield is continuously transforming photons into beliefs about the physical world. In this context, an auditor could ask why does the system believes that this object exists at this location, with this identity, at this moment in time?

Perception observability attempts to answer that question.

Imagine a drone flying over an industrial facility every day for six months. During that time, it detects vehicles, buildings, equipment, containers, roads, vegetation changes, construction progress, temporary structures, and human activity. Traditional analytics pipelines often produce outputs as isolated events. A vehicle is detected. A building is classified. A region is segmented. An alert is issued.

Yet when an analyst asks where that conclusion came from, the system often has no meaningful answer beyond confidence scores. Confidence, however, is not evidence. A score of 0.97 does not state why the conclusion is true.