What percentage of aerial drone videos are top-down nadir videos? Most drone experts would hesitate to give that answer but when pressed they would lean towards a 40-60 split between nadir and oblique/FPV videos. In this article, we explore how accurate that might be. First, there are two di¯erent drone worlds: 1. Photogrammetry/mapping drones 2. Inspection/media/consumer drones The first of these two, benefits from a near-nadir perspective over other angles because it is helpful to create orthomosaics, surface models and measurements. Oblique coverage is helpful to façade or height estimation but not ground/roof coverage. Most commercial shot planning uses a mix, but the percentage varies on a case-by-case basis. Historically photogrammetry and mapping datasets such as OpenDroneMap/ESRI accrued nadir datasets with a dominance of top-down collections but the situation reverses with the ubiquity of hobbyist/consumer drones that generate YouTube drone footage, real-estate marketing, inspections, construction progress, emergency or rescue response, racing and cinematic content. Computer Vision has also switched to the study of these videos as evident from VisDrone/UAV tracking datasets that are forward looking. So, the market determines the ration of these videos. Photogrammetry and Mapping also benefit from still photographs rather than videos. Mapping is easier with one acquisition setup whereas oblique imagery requires using five cameras. Datasets for mapping often have varying ratios depending on use cases but they all benefit from derived orthomosaics and related products. Geospatial analytics and mapping ecosystem among these use cases, have the most skewed ratio and in favor of nadir perspective. Even when the perspectives mix in the same video between takeo¯ and landing, a credible estimate based on duration alone is elusive because of revisits. A single nadir tour of a survey area has high overlaps with those from extended rounds and revisits that usually build up a drone tour flight path. However, nadir videos help with a survey map more than any oblique videos. Any study of such a ratio amongst existing datasets is also subject to the same limitations as with analytics over a single drone video. It is inherently multimodal in nature and the search terms and retrieval frameworks no matter whether hierarchical navigable small world, exhaustive KNN, agentic or tool-based su¯er from significant variations in precision and recall with none coming close to being true to actual. The most credible reply therefore is that the answer might be unknown, but the market determines the choice.

 PruneVid for aerial drone videos:

The previous article discussed using one of the three supported VLMs for PruneVid to improve token efficiency with the smallest model: PLLaVA. The other two are LLaVA-OneVision and ST-LLM (Liu et al., 2024b).  

ST-LLM feeds all raw, unpooled spatial-temporal tokens directly into the LLM backbone. For an egomotion camera where changes between consecutive frames are incremental, preserving the raw spatial-temporal progression allows the LLM to model subtle frame-to-frame displacements, motion cues, and panning trajectories. To prevent the massive context length from slowing down the system, it uses a dynamic masking strategy during training. This means it can drop redundant tokens where frames overlap significantly while forcing the model to retain a highly granular understanding of the continuous stream

The following implementation uses LLaVA-OneVision instead of the PLLaVA and ST-LLM from the previous article:

"""Standalone LLaVA-OneVision aerial-video runner. No PruneVid/ST-LLM imports."""

import argparse

import faulthandler

import json

import logging

import math

from pathlib import Path

import platform

import time

LOG = logging.getLogger('onevision')

DEFAULT_MODEL = 'llava-hf/llava-onevision-qwen2-0.5b-ov-hf'

def sample_indices(total, fps, start, seconds, frames):

    if total < 1 or not math.isfinite(fps) or fps <= 0:

        raise ValueError('Invalid video frame count or FPS. Transcode to constant-frame-rate MP4.')

    if not math.isfinite(start) or not math.isfinite(seconds) or start < 0 or seconds <= 0 or frames < 1:

        raise ValueError('Require start >= 0, seconds > 0 and frames >= 1.')

    first = int(start * fps)

    last = min(total, int((start + seconds) * fps)) - 1

    if last - first + 1 < frames:

        raise ValueError('Clip is out of range or contains fewer source frames than --frames.')

    return [first] if frames == 1 else [round(first + i * (last-first)/(frames-1)) for i in range(frames)]

def save_json(path, value):

    temporary = path.with_suffix(path.suffix + '.tmp')

    temporary.write_text(json.dumps(value, indent=2), encoding='utf-8')

    temporary.replace(path)

def parse_args():

    p = argparse.ArgumentParser(description=__doc__)

    p.add_argument('--video', required=True)

    p.add_argument('--model-dir', '--model-id', dest='model_dir', default=DEFAULT_MODEL,

                   help='Complete Hugging Face OneVision checkpoint directory or repository ID.')

    p.add_argument('--start', type=float, default=0)

    p.add_argument('--seconds', type=float, default=20)

    p.add_argument('--frames', type=int, default=16)

    p.add_argument('--question', default='Describe the visible terrain, structures, roads, and moving objects. Distinguish camera motion from object motion. State uncertainty; do not invent GPS coordinates or metric distances.')

    p.add_argument('--max-new-tokens', type=int, default=160)

    p.add_argument('--output', default='runs/drone_onevision')

    p.add_argument('--prepare-only', action='store_true')

    p.add_argument('--device', choices=['cuda', 'cpu'], default='cuda')

    p.add_argument('--dtype', choices=['auto', 'float16', 'bfloat16', 'float32'], default='auto')

    p.add_argument('--load-in-4bit', action='store_true')

    p.add_argument('--local-files-only', action='store_true')

    p.add_argument('--revision', default='main', help='Hub revision; use a commit hash for reproducibility.')

    a = p.parse_args()

    if a.max_new_tokens < 1 or a.frames < 1 or not a.question.strip():

        p.error('Require positive frames/max-new-tokens and a nonempty question.')

    if a.device == 'cpu' and a.load_in_4bit:

        p.error('This adapter supports bitsandbytes 4-bit loading on CUDA only.')

    return a

def prepare(a, target):

    import cv2

    from PIL import Image, ImageDraw

    source = Path(a.video).resolve()

    if not source.is_file():

        raise FileNotFoundError(source)

    cap = cv2.VideoCapture(str(source))

    images, samples = [], []

    try:

        if not cap.isOpened():

            raise RuntimeError('OpenCV cannot open the video; check codec and path.')

        fps = float(cap.get(cv2.CAP_PROP_FPS))

        total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

        indices = sample_indices(total, fps, a.start, a.seconds, a.frames)

        for i, index in enumerate(indices):

            cap.set(cv2.CAP_PROP_POS_FRAMES, index)

            ok, bgr = cap.read()

            if not ok:

                raise RuntimeError(f'Failed to decode frame {index}.')

            image = Image.fromarray(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))

            images.append(image)

            image.save(target / f'frame_{i:03d}.jpg')

            samples.append({'frame_index': index, 'nominal_seconds': index / fps})

    finally:

        cap.release()

    sheet = Image.new('RGB', (1280, math.ceil(a.frames / 4) * 205), 'white')

    draw = ImageDraw.Draw(sheet)

    for i, image in enumerate(images):

        thumb = image.copy()

        thumb.thumbnail((320, 180))

        x, y = (i % 4)*320, (i // 4)*205

        sheet.paste(thumb, (x, y))

        draw.text((x+4, y+182), f'{samples[i]["nominal_seconds"]:.3f}s', fill='black')

    sheet.save(target / 'contact_sheet.jpg')

    meta = {'arguments': vars(a), 'video': str(source), 'fps': fps,

            'source_frame_count': total, 'timestamps_are_nominal': True, 'samples': samples}

    save_json(target / 'sampling.json', meta)

    return images, meta

def infer(a, images, metadata):

    import numpy as np

    import torch

    import transformers

    from transformers import AutoConfig, AutoProcessor, LlavaOnevisionForConditionalGeneration

    device = 'cuda:0' if a.device == 'cuda' else 'cpu'

    if a.device == 'cuda':

        if not torch.cuda.is_available():

            raise RuntimeError('CUDA unavailable. Install a CUDA-enabled PyTorch wheel; no implicit CPU fallback.')

        LOG.info('GPU: %s; capability: %s; torch: %s; runtime: %s',

                 torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0), torch.__version__, torch.version.cuda)

        if torch.cuda.get_device_capability(0) == (12, 0) and 'sm_120' not in torch.cuda.get_arch_list():

            LOG.warning('sm_120 not listed in this wheel. The following real GPU computation must pass.')

        x = torch.randn(128, 128, device=device)

        _ = x @ x

        torch.cuda.synchronize()

        del x, _

    dtype_name = a.dtype if a.dtype != 'auto' else ('float32' if a.device == 'cpu' else 'float16')

    if a.device == 'cpu' and dtype_name != 'float32':

        raise ValueError('Use float32 on CPU in this adapter.')

    if dtype_name == 'bfloat16' and a.device == 'cuda' and not torch.cuda.is_bf16_supported():

        raise RuntimeError('This GPU does not report BF16 support.')

    dtype = getattr(torch, dtype_name)

    shared = {'revision': a.revision, 'local_files_only': a.local_files_only, 'trust_remote_code': False}

    config = AutoConfig.from_pretrained(a.model_dir, **shared)

    if config.model_type != 'llava_onevision':

        raise ValueError(f'Expected llava_onevision, got {config.model_type!r}; PLLaVA/ST-LLM weights cannot be reused.')

    processor = AutoProcessor.from_pretrained(a.model_dir, use_fast=False, **shared)

    processor.tokenizer.padding_side = 'left'

    options = dict(config=config, torch_dtype=dtype, low_cpu_mem_usage=True,

                   device_map={'': device}, attn_implementation='sdpa', **shared)

    if a.load_in_4bit:

        from transformers import BitsAndBytesConfig

        options['quantization_config'] = BitsAndBytesConfig(

            load_in_4bit=True, bnb_4bit_quant_type='nf4',

            bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=dtype)

    LOG.info('Loading %s onto %s; no automatic RAM/disk offload.', a.model_dir, device)

    started = time.perf_counter()

    model = LlavaOnevisionForConditionalGeneration.from_pretrained(a.model_dir, **options).eval()

    load_seconds = time.perf_counter()-started

    times = ', '.join(f'{row["nominal_seconds"]:.3f}' for row in metadata['samples'])

    question = (f'The supplied video consists of {a.frames} sampled frames, in chronological order, '

                f'at nominal source-video times [{times}] seconds. Events between samples may be unobserved.\n' + a.question)

    messages = [{'role': 'user', 'content': [{'type': 'video'}, {'type': 'text', 'text': question}]}]

    prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

    clip = np.stack([np.asarray(image) for image in images])

    inputs = processor(text=prompt, videos=[clip], padding=True, return_tensors='pt')

    inputs = inputs.to(device=device, dtype=dtype)

    input_length = inputs['input_ids'].shape[1]

    limit = getattr(config.text_config, 'max_position_embeddings', None)

    if limit and input_length + a.max_new_tokens > limit:

        raise ValueError(f'Context overflow: {input_length}+{a.max_new_tokens}>{limit}; reduce --frames.')

    video_token_count = int((inputs['input_ids'] == config.video_token_index).sum().item())

    if a.device == 'cuda':

        torch.cuda.synchronize()

        torch.cuda.reset_peak_memory_stats()

    started = time.perf_counter()

    with torch.inference_mode():

        generated = model.generate(**inputs, max_new_tokens=a.max_new_tokens,

                                   do_sample=False, num_beams=1, use_cache=True)

    if a.device == 'cuda':

        torch.cuda.synchronize()

    elapsed = time.perf_counter()-started

    answer = processor.batch_decode(generated[:, input_length:], skip_special_tokens=True,

                                    clean_up_tokenization_spaces=False)[0].strip()

    return {**metadata, 'answer': answer, 'model_type': config.model_type,

            'resolved_model_commit': getattr(config, '_commit_hash', None),

            'model_load_seconds': load_seconds, 'generation_seconds': elapsed,

            'input_tokens': input_length, 'video_placeholder_tokens': video_token_count,

            'generated_tokens': generated.shape[1]-input_length,

            'input_shapes': {k:list(v.shape) for k,v in inputs.items() if hasattr(v, 'shape')},

            'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated()/2**30 if a.device == 'cuda' else None,

            'gpu': torch.cuda.get_device_name(0) if a.device == 'cuda' else None,

            'torch_version': torch.__version__, 'transformers_version': transformers.__version__,

            'python_version': platform.python_version(), 'resolved_dtype': dtype_name,

            'note': 'Standard OneVision video inference, not PruneVid pruning or GeoVARNet evidence reasoning. Timestamps are frame-index/FPS approximations.'}

def main():

    faulthandler.enable()

    logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')

    a = parse_args()

    target = Path(a.output)

    target.mkdir(parents=True, exist_ok=True)

    try:

        started = time.perf_counter()

        images, metadata = prepare(a, target)

        preparation_seconds = time.perf_counter()-started

        if a.prepare_only:

            LOG.info('Prepared %d frames, sampling.json and contact sheet; no model loaded.', len(images))

            return

        result = infer(a, images, metadata)

        result['preparation_seconds'] = preparation_seconds

        result['total_seconds'] = time.perf_counter()-started

        save_json(target / 'result.json', result)

        print(result['answer'])

    except Exception as error:

        save_json(target / 'error.json', {'error_type': type(error).__name__, 'message': str(error), 'arguments': vars(a)})

        LOG.exception('Run failed. No fallback answer was generated.')

        raise

if __name__ == '__main__':

    main()

Some caveat remain as before:

1. Libraries I used:

# Approximate environment for run_onevision_drone.py

# Use in a clean environment. GPU inference has not been validated here.

# --extra-index-url https://download.pytorch.org/whl/cu128

 torch==2.7.1+cu128

 torchvision==0.22.1+cu128

 transformers==4.51.3

 accelerate>=1.6,<2

 huggingface-hub>=0.30,<1

 numpy>=1.26,<2

 opencv-python==4.9.0.80

 pillow>=10,<12

 sentencepiece>=0.2,<0.3

 protobuf>=4.25,<6

 safetensors>=0.4.3,<1

 # Optional: uncomment for --load-in-4bit (not needed for the default 0.5B run).

 bitsandbytes>=0.47,<1

2. 

There is a checkpoint/code compatibility issue.

prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint. 

and there are three checkpoints involved which is suspicious:

LlaVa-OneVision

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.

1. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash LlaVA-OneVision compatibility issue.

 Cutting Tokens in Drone Video Analytics

The inference cost of aerial drone images and video even with sampling often runs into thousands of tokens and almost always repeated per query given the long tail of such analytics. GeoVARNet and our proposal asked: "Can representation-guided enrichment reduce computation without materially reducing event recall?" [1]. Players like NVIDIA are also asking similar questions but from a different perspective.

The Video Search and Summarization (VSS) blueprint is NVIDIA's reference architecture for video agents that perform:

• video search,

• visual Q&A,

• summarization,

• alerting,

• report generation,

using VLMs, LLMs, retrieval systems, and video analytics pipelines. They introduced Adaptive Efficient Video Sampling (EVS) that dynamically prunes unchanged visual patches per frame and batches VLM work around moments of activity.[3]

Their reported outcomes include:

• ~80% fewer VLM input tokens for a 60 minute video,

• ~50% faster summarization,

• 17% lower alert-contextualization latency,

• 46% higher concurrent real-time VLM stream capacity [3]

Although the benchmarks for drone video analytics such as UAVBench and EZBenchmark are diverse and require reprocessing, the EVS matters over traditional VLM pipelines that often progress through ingestion, frame sampling, VLM inference, Caption captures, and image embeddings and typically repeat that for every input even though almost nothing changed. EVS attempts to suppress that waste by detecting regions that are largely unchanged and avoiding feeding redundant visual information into the VLM. With their introduction of change detector, they reprocess only changed regions/chunks. This is referred to as token-pruning. GeoVARNet, on the other hand, engages in representation-guided enrichment. So, it can answer questions on an “evidence” basis rather than at “pixel” basis with a much broader definition of evidence and decision variables such as persistence, covisibility, revisits, place identity, entity history, observation coverage and rare-event likelihood. These are all first-class architectural concepts in GeoVARNet. 

For example, if a parking lot is observed six times, EVS sees parking lot looks mostly unchanged and reduces token usage while GeoVARNet could additionally infer same place, same appearance, same persistent state, already enriched, and high confidence. This would help avoid VLM calls, embedding calls, graph expansion, retrieval enrichment, and storage growth. That is a stronger form of reduction than token pruning alone.

There are others pursuing similar ideas though not with the same objectives. For example, Keyframe selection systems such as ORB-SLAM3, PKS (photogrammetric key-frame selection) and OpenVSLAM decide to only process frames that contribute new geometry. While GeoVARNet suggests using PKS, those techniques are for geometry redundancy, not semantic redundancy.

Modern video-RAG systems have a workflow that uses frame to embedding conversion and then determine distance from prior embedding to decide whether to retain or discard the frame and while this does not reduce the embedding generation cost for those frame and is used by many commercial video indexing pipelines, it leans on semantic novelty rather than geometry. This fails on tiny moving objects, vehicle swaps, transient events and drone revisits which GeoVARNet tries to preserve.

Another angle to reduce costs is detecting scene boundaries as found in video search, summarization and media indexing. If the scene did not change, why incur cost? But drone videos often have the same scene for different event which scene segmentation misses.

A more recent approach involves memory systems for Video LLMs that aim to improve with 

• StreamingVLM-style approaches

• memory-token approaches

• recurrent video transformers

These also obviate repetitive processing but by means of summary memory and sparse visual retention with a targeted transformer context length. While this reduces tokens, it does not retain as GeoVARNet does by compressing observations, places, revisits, and relationships at a higher-level abstraction.

The notable direction with NVIDIA VSS is the push towards Graph-RAG video systems where the architecture includes Graph-RAG, video search, agent workflows and video understanding thus generalizing for all video and not just urban drone videos. It completely avoids reprocessing the video to query the graph. 

This direction is a direct threat to GeoVARNet because it combines Embedding Novelty Sampling with Graph-RAG and Keyframe Selection and together it can already provide dramatic savings without introducing additional geometry decomposition and revisitation processing but GeoVARNet could demonstrate that it outperforms token-pruning, keyframe-selection, and embedding-novelty baselines at equal event-recall targets in terms of enrichment workload, VLM calls, storage growth, and event recall, When it does, it has a much stronger claim than reducing computation 

 PruneVid for aerial drone videos:

The previous article discussed using one of the three supported VLMs for PruneVid to improve token efficiency with the smallest model: PLLaVA. The other two are LLaVA-OneVision and ST-LLM (Liu et al., 2024b).  

ST-LLM feeds all raw, unpooled spatial-temporal tokens directly into the LLM backbone. For an egomotion camera where changes between consecutive frames are incremental, preserving the raw spatial-temporal progression allows the LLM to model subtle frame-to-frame displacements, motion cues, and panning trajectories. To prevent the massive context length from slowing down the system, it uses a dynamic masking strategy during training. This means it can drop redundant tokens where frames overlap significantly while forcing the model to retain a highly granular understanding of the continuous stream

The following implementation uses ST-LLM instead of the PLLaVA from the previous article:

"""Single-clip adapter for Visual-AI/PruneVid. Run from its repository root."""

"""ST-LLM official-API adapter; run from the ST-LLM repository root."""

import argparse

import json

from pathlib import Path

import time

def main():

    parser = argparse.ArgumentParser(description=__doc__)

    parser.add_argument('--video', required=True, help='Already-trimmed video clip')

    parser.add_argument('--ckpt-path', required=True)

    parser.add_argument('--qformer-path', required=True)

    parser.add_argument('--cfg-path', default='config/instructblipbase_stllm_conversation.yaml')

    parser.add_argument('--frames', type=int, default=16)

    parser.add_argument('--video-input', choices=['all', 'residual'], default='all')

    parser.add_argument('--max-new-tokens', type=int, default=160)

    parser.add_argument('--question', default='Describe terrain, structures, roads and moving objects. Distinguish camera motion from object motion. Do not invent GPS coordinates or distances.')

    parser.add_argument('--output', default='runs/drone_stllm.json')

    a = parser.parse_args()

    if a.frames < 16 or a.max_new_tokens < 1:

        parser.error('Use frames >=16 and max-new-tokens >=1.')

    for path in [a.video, a.qformer_path, a.cfg_path]:

        if not Path(path).is_file():

            parser.error('Missing file: ' + path)

    if not (Path(a.ckpt_path) / 'config.json').is_file():

        parser.error('ckpt-path must be the conversation_weight directory containing config.json.')

    import torch

    if not torch.cuda.is_available():

        raise RuntimeError('CUDA unavailable; fix PyTorch before loading ST-LLM.')

    if torch.cuda.get_device_capability(0) == (12, 0) and 'sm_120' not in torch.cuda.get_arch_list():

        raise RuntimeError('Install a Blackwell-compatible PyTorch wheel with sm_120 support.')

    x = torch.randn(64, 64, device='cuda')

    _ = x @ x

    torch.cuda.synchronize()

    from stllm.common.config import Config

    from stllm.common.registry import registry

    import stllm.models

    from stllm.conversation.conversation import Chat, CONV_instructblip_Vicuna0

    config_args = argparse.Namespace(cfg_path=a.cfg_path, options=None)

    cfg = Config(config_args)

    mc = cfg.model_cfg

    mc.ckpt = str(Path(a.ckpt_path).resolve())

    mc.llama_model = mc.ckpt

    mc.q_former_model = str(Path(a.qformer_path).resolve())

    mc.device_8bit = 0

    mc.use_grad_checkpoint = False

    # Keep residual modules when loading this residual-trained checkpoint.

    # Switch only its inference aggregation after loading.

    model_cls = registry.get_model_class(mc.arch)

    print('Loading official ST-LLM checkpoint; full FP16, no quantization/offload.', flush=True)

    model = model_cls.from_config(mc).to(device='cuda:0', dtype=torch.float16).eval()

    chat = Chat(model, device='cuda:0')

    chat.model.video_input = a.video_input

    conv = CONV_instructblip_Vicuna0.copy()

    images = []

    torch.cuda.reset_peak_memory_stats()

    start = time.perf_counter()

    with torch.inference_mode():

        chat.upload_video(str(Path(a.video).resolve()), conv, images,

                          num_frame=a.frames, text=a.question)

        chat.ask('###Human: ' + a.question + ' ###Assistant: ', conv)

        probe = conv.copy()

        probe.append_message(probe.roles[1], None)

        embeddings, _ = chat.get_context_emb_sim(probe, images, system=True)

        input_tokens = int(embeddings.shape[1])

        visual_tokens = int(images[0].shape[1])

        del embeddings

        context_limit = int(model.config.max_position_embeddings)

        required = input_tokens + a.max_new_tokens

        if required > context_limit:

            raise RuntimeError(f'Context overflow: {input_tokens} input + {a.max_new_tokens} output > {context_limit}. Reduce --frames; no silent truncation allowed.')

        answer, _ = chat.answer(conv=conv, img_list=images, do_sample=False,

                                num_beams=1, max_new_tokens=a.max_new_tokens,

                                max_length=context_limit)

    torch.cuda.synchronize()

    result = {'arguments': vars(a), 'answer': answer, 'visual_tokens': visual_tokens,

              'input_tokens': input_tokens, 'context_limit': context_limit,

              'encode_and_generate_seconds': time.perf_counter()-start,

              'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated()/2**30,

              'torch_version': torch.__version__, 'gpu': torch.cuda.get_device_name(0),

              'note': 'all retains all per-frame encoder/Q-Former output tokens, not raw pixels or all ViT patches; residual uses checkpoint global-local path.'}

    dest = Path(a.output)

    dest.parent.mkdir(parents=True, exist_ok=True)

    dest.write_text(json.dumps(result, indent=2), encoding='utf-8')

    print(answer)

if __name__ == '__main__':

    main()

Some caveat remain as before:

1. Libraries I used:

Python 3.10.22

torch 2.2.1+cu118

transformers 4.40.2

peft 0.10.0

accelerate 0.26.1

safetensors 0.4.2

mmcv 1.7.2

decord 0.6.0

moviepy 1.0.3

imageio 2.27.0

matplotlib 3.8.3

protobuf 4.23.4

sentencepiece 0.2.0

einops 0.6.1

2. 

There is a checkpoint/code compatibility issue.

prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint. 

and there are three checkpoints involved which is suspicious:

stllm-blackwell

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.

3. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash / ST-LLM compatibility issue.

#codingexercise: CodingExercise-11-10-2026.docx

 The previous article described an implementation. The following are from test runs:

1. Libraries I used:
   

Python 3.10.22

torch 2.2.1+cu118

transformers 4.40.2

peft 0.10.0

accelerate 0.26.1

safetensors 0.4.2

mmcv 1.7.2

decord 0.6.0

moviepy 1.0.3

imageio 2.27.0

matplotlib 3.8.3

protobuf 4.23.4

sentencepiece 0.2.0

einops 0.6.1

   2. There is a checkpoint/code compatibility issue.

prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint.

and there are three checkpoints involved which is suspicious:
pllava-7b

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.

 

3.  3. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash / pllava compatibility issue.

 PruneVid for aerial drone videos:

"""Single-clip adapter for Visual-AI/PruneVid. Run from its repository root."""
import argparse
import json
import math
import sys
import time
from pathlib import Path


def main():
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument('--video', required=True)
    p.add_argument('--output', default='drone_result')
    p.add_argument('--start', type=float, default=0)
    p.add_argument('--seconds', type=float, default=20)
    p.add_argument('--frames', type=int, default=16)
    p.add_argument('--prepare-only', action='store_true')
    p.add_argument('--model-dir', default='MODELS/pllava-7b')
    p.add_argument('--weight-dir')
    p.add_argument('--lora-alpha', type=int, default=14)
    p.add_argument('--question', default='Describe the visible terrain, structures, roads and moving objects. Distinguish camera motion from object motion. State uncertainty; do not invent GPS coordinates or metric distances.')
    p.add_argument('--alpha', type=float, default=0.4)
    p.add_argument('--tau', type=float, default=0.8)
    p.add_argument('--cluster-ratio', type=float, default=0.5)
    p.add_argument('--temporal-segment-ratio', type=float, default=0.25)
    p.add_argument('--selected-layer', type=int, default=10)
    p.add_argument('--max-new-tokens', type=int, default=160)
    a = p.parse_args()
    if a.frames < 4 or a.seconds <= 0 or a.start < 0:
        p.error('Require frames >= 4, seconds > 0 and start >= 0.')
    for name in ('alpha', 'cluster_ratio', 'temporal_segment_ratio'):
        if not 0 < getattr(a, name) <= 1:
            p.error(name + ' must be in (0, 1].')
    if not -1 <= a.tau <= 1:
        p.error('tau must be in [-1, 1].')
    import cv2
    import numpy as np
    from PIL import Image, ImageDraw
    video = Path(a.video).resolve()
    if not video.is_file():
        raise FileNotFoundError(video)
    target = Path(a.output)
    target.mkdir(parents=True, exist_ok=True)
    cap = cv2.VideoCapture(str(video))
    try:
        if not cap.isOpened():
            raise RuntimeError('Cannot open video; check path and codec.')
        fps = cap.get(cv2.CAP_PROP_FPS)
        total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
        if not math.isfinite(fps) or fps <= 0 or total <= 0:
            raise RuntimeError('Invalid FPS/frame count; transcode video before sampling.')
        first = int(a.start * fps)
        last = min(total, int((a.start + a.seconds) * fps)) - 1
        if last - first + 1 < a.frames:
            raise RuntimeError('Requested clip has fewer source frames than --frames.')
        indices = np.linspace(first, last, a.frames).round().astype(int)
        images, samples = [], []
        for i, idx in enumerate(indices):
            cap.set(cv2.CAP_PROP_POS_FRAMES, int(idx))
            ok, bgr = cap.read()
            if not ok:
                raise RuntimeError(f'Cannot decode source frame {idx}.')
            image = Image.fromarray(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))
            images.append(image)
            image.save(target / f'frame_{i:03d}.jpg')
            samples.append({'frame_index': int(idx), 'nominal_seconds': float(idx / fps)})
    finally:
        cap.release()
    sheet = Image.new('RGB', (4 * 320, math.ceil(a.frames / 4) * 205), 'white')
    draw = ImageDraw.Draw(sheet)
    for i, image in enumerate(images):
        thumb = image.copy()
        thumb.thumbnail((320, 180))
        x, y = (i % 4) * 320, (i // 4) * 205
        sheet.paste(thumb, (x, y))
        draw.text((x + 4, y + 182), f'{samples[i]["nominal_seconds"]:.2f}s', fill='black')
    sheet.save(target / 'contact_sheet.jpg')
    metadata = {'arguments': vars(a), 'video': str(video), 'fps': fps,
                'timestamps_are_nominal': True, 'samples': samples}
    (target / 'sampling.json').write_text(json.dumps(metadata, indent=2), encoding='utf-8')
    if a.prepare_only:
        print('Prepared sampled frames and contact sheet; no model loaded.')
        return
    import torch
    if not torch.cuda.is_available():
        raise RuntimeError('Official-model path requires working NVIDIA CUDA in this adapter. Use --prepare-only for CPU preprocessing.')
    if not torch.cuda.is_bf16_supported():
        raise RuntimeError('Official loader uses bfloat16; this CUDA device does not report BF16 support.')
    root = Path.cwd()
    if not (root / 'tasks/eval/model_utils.py').is_file():
        raise RuntimeError('Run from the cloned PruneVid root, with this script copied there.')
    sys.path.insert(0, str(root))
    from tasks.eval.model_utils import load_pllava
    from tasks.eval.eval_utils import conv_templates
    model_dir = str(Path(a.model_dir).resolve())
    weight_dir = str(Path(a.weight_dir or a.model_dir).resolve())
    if 'llava' not in model_dir:
        raise RuntimeError('Official loader selects PLLaVA by lowercase "llava" in model path; name the directory pllava-7b.')
    for directory in (model_dir, weight_dir):
        if not (Path(directory) / 'config.json').is_file():
            raise FileNotFoundError(f'Missing config.json in {directory}')
    torch.manual_seed(0)
    model, processor = load_pllava(
        model_dir, num_frames=a.frames, use_lora=True, weight_dir=weight_dir,
        lora_alpha=a.lora_alpha, pooling_shape=(a.frames, 12, 12),
        selected_layer=a.selected_layer, alpha=a.alpha, tau=a.tau,
        cluster_ratio=a.cluster_ratio, temporal_segment_ratio=a.temporal_segment_ratio)
    model = model.to('cuda').eval()
    conv = conv_templates['plain'].copy()
    conv.user_query(a.question, is_mm=True)
    inputs = processor(text=conv.get_prompt(), images=images, return_tensors='pt').to(model.device)
    if 'pixel_values' in inputs and inputs['pixel_values'] is not None:
        inputs['pixel_values'] = inputs['pixel_values'].to(dtype=next(model.parameters()).dtype)
    torch.cuda.synchronize()
    torch.cuda.reset_peak_memory_stats()
    started = time.perf_counter()
    with torch.inference_mode():
        tokens = model.generate(**inputs, media_type='video', do_sample=False,
                                max_new_tokens=a.max_new_tokens, use_cache=True)
    torch.cuda.synchronize()
    decoded = processor.batch_decode(tokens, skip_special_tokens=True,
                                     clean_up_tokenization_spaces=False)[0]
    answer = decoded.split(conv.roles[-1])[-1].removesuffix('</s>').strip()
    result = {**metadata, 'answer': answer, 'raw_decoded_text': decoded,
              'generation_seconds': time.perf_counter() - started,
              'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated() / 2**30,
              'gpu': torch.cuda.get_device_name(0), 'torch_version': torch.__version__,
              'note': 'No measured retained-token count; no baseline or accuracy claim.'}
    (target / 'result.json').write_text(json.dumps(result, indent=2), encoding='utf-8')
    print(answer)


if __name__ == '__main__':
    main()

Continued from previous post

 Just as AI observability evolved from model monitoring into decision accountability, perception observability is likely to evolve from object detection into world-model accountability. The systems that succeed will not simply identify what is visible. They will explain how every conclusion emerged, how it changed over time, and how an investigator can independently verify it. Even with an LLM-as-a-judge paradigm, the validation must remain independent.

The ability to defend a perception-derived belief through a complete chain of evidence becomes the defining characteristic of trustworthy aerial intelligence.