What percentage of aerial drone videos are top-down nadir videos? Most drone experts would hesitate to give that answer but when pressed they would lean towards a 40-60 split between nadir and oblique/FPV videos. In this article, we explore how accurate that might be. First, there are two di¯erent drone worlds: 1. Photogrammetry/mapping drones 2. Inspection/media/consumer drones The first of these two, benefits from a near-nadir perspective over other angles because it is helpful to create orthomosaics, surface models and measurements. Oblique coverage is helpful to façade or height estimation but not ground/roof coverage. Most commercial shot planning uses a mix, but the percentage varies on a case-by-case basis. Historically photogrammetry and mapping datasets such as OpenDroneMap/ESRI accrued nadir datasets with a dominance of top-down collections but the situation reverses with the ubiquity of hobbyist/consumer drones that generate YouTube drone footage, real-estate marketing, inspections, construction progress, emergency or rescue response, racing and cinematic content. Computer Vision has also switched to the study of these videos as evident from VisDrone/UAV tracking datasets that are forward looking. So, the market determines the ration of these videos. Photogrammetry and Mapping also benefit from still photographs rather than videos. Mapping is easier with one acquisition setup whereas oblique imagery requires using five cameras. Datasets for mapping often have varying ratios depending on use cases but they all benefit from derived orthomosaics and related products. Geospatial analytics and mapping ecosystem among these use cases, have the most skewed ratio and in favor of nadir perspective. Even when the perspectives mix in the same video between takeo¯ and landing, a credible estimate based on duration alone is elusive because of revisits. A single nadir tour of a survey area has high overlaps with those from extended rounds and revisits that usually build up a drone tour flight path. However, nadir videos help with a survey map more than any oblique videos. Any study of such a ratio amongst existing datasets is also subject to the same limitations as with analytics over a single drone video. It is inherently multimodal in nature and the search terms and retrieval frameworks no matter whether hierarchical navigable small world, exhaustive KNN, agentic or tool-based su¯er from significant variations in precision and recall with none coming close to being true to actual. The most credible reply therefore is that the answer might be unknown, but the market determines the choice.

 PruneVid for aerial drone videos:

The previous article discussed using one of the three supported VLMs for PruneVid to improve token efficiency with the smallest model: PLLaVA. The other two are LLaVA-OneVision and ST-LLM (Liu et al., 2024b).  

ST-LLM feeds all raw, unpooled spatial-temporal tokens directly into the LLM backbone. For an egomotion camera where changes between consecutive frames are incremental, preserving the raw spatial-temporal progression allows the LLM to model subtle frame-to-frame displacements, motion cues, and panning trajectories. To prevent the massive context length from slowing down the system, it uses a dynamic masking strategy during training. This means it can drop redundant tokens where frames overlap significantly while forcing the model to retain a highly granular understanding of the continuous stream

The following implementation uses LLaVA-OneVision instead of the PLLaVA and ST-LLM from the previous article:

"""Standalone LLaVA-OneVision aerial-video runner. No PruneVid/ST-LLM imports."""

import argparse

import faulthandler

import json

import logging

import math

from pathlib import Path

import platform

import time

LOG = logging.getLogger('onevision')

DEFAULT_MODEL = 'llava-hf/llava-onevision-qwen2-0.5b-ov-hf'

def sample_indices(total, fps, start, seconds, frames):

    if total < 1 or not math.isfinite(fps) or fps <= 0:

        raise ValueError('Invalid video frame count or FPS. Transcode to constant-frame-rate MP4.')

    if not math.isfinite(start) or not math.isfinite(seconds) or start < 0 or seconds <= 0 or frames < 1:

        raise ValueError('Require start >= 0, seconds > 0 and frames >= 1.')

    first = int(start * fps)

    last = min(total, int((start + seconds) * fps)) - 1

    if last - first + 1 < frames:

        raise ValueError('Clip is out of range or contains fewer source frames than --frames.')

    return [first] if frames == 1 else [round(first + i * (last-first)/(frames-1)) for i in range(frames)]

def save_json(path, value):

    temporary = path.with_suffix(path.suffix + '.tmp')

    temporary.write_text(json.dumps(value, indent=2), encoding='utf-8')

    temporary.replace(path)

def parse_args():

    p = argparse.ArgumentParser(description=__doc__)

    p.add_argument('--video', required=True)

    p.add_argument('--model-dir', '--model-id', dest='model_dir', default=DEFAULT_MODEL,

                   help='Complete Hugging Face OneVision checkpoint directory or repository ID.')

    p.add_argument('--start', type=float, default=0)

    p.add_argument('--seconds', type=float, default=20)

    p.add_argument('--frames', type=int, default=16)

    p.add_argument('--question', default='Describe the visible terrain, structures, roads, and moving objects. Distinguish camera motion from object motion. State uncertainty; do not invent GPS coordinates or metric distances.')

    p.add_argument('--max-new-tokens', type=int, default=160)

    p.add_argument('--output', default='runs/drone_onevision')

    p.add_argument('--prepare-only', action='store_true')

    p.add_argument('--device', choices=['cuda', 'cpu'], default='cuda')

    p.add_argument('--dtype', choices=['auto', 'float16', 'bfloat16', 'float32'], default='auto')

    p.add_argument('--load-in-4bit', action='store_true')

    p.add_argument('--local-files-only', action='store_true')

    p.add_argument('--revision', default='main', help='Hub revision; use a commit hash for reproducibility.')

    a = p.parse_args()

    if a.max_new_tokens < 1 or a.frames < 1 or not a.question.strip():

        p.error('Require positive frames/max-new-tokens and a nonempty question.')

    if a.device == 'cpu' and a.load_in_4bit:

        p.error('This adapter supports bitsandbytes 4-bit loading on CUDA only.')

    return a

def prepare(a, target):

    import cv2

    from PIL import Image, ImageDraw

    source = Path(a.video).resolve()

    if not source.is_file():

        raise FileNotFoundError(source)

    cap = cv2.VideoCapture(str(source))

    images, samples = [], []

    try:

        if not cap.isOpened():

            raise RuntimeError('OpenCV cannot open the video; check codec and path.')

        fps = float(cap.get(cv2.CAP_PROP_FPS))

        total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

        indices = sample_indices(total, fps, a.start, a.seconds, a.frames)

        for i, index in enumerate(indices):

            cap.set(cv2.CAP_PROP_POS_FRAMES, index)

            ok, bgr = cap.read()

            if not ok:

                raise RuntimeError(f'Failed to decode frame {index}.')

            image = Image.fromarray(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))

            images.append(image)

            image.save(target / f'frame_{i:03d}.jpg')

            samples.append({'frame_index': index, 'nominal_seconds': index / fps})

    finally:

        cap.release()

    sheet = Image.new('RGB', (1280, math.ceil(a.frames / 4) * 205), 'white')

    draw = ImageDraw.Draw(sheet)

    for i, image in enumerate(images):

        thumb = image.copy()

        thumb.thumbnail((320, 180))

        x, y = (i % 4)*320, (i // 4)*205

        sheet.paste(thumb, (x, y))

        draw.text((x+4, y+182), f'{samples[i]["nominal_seconds"]:.3f}s', fill='black')

    sheet.save(target / 'contact_sheet.jpg')

    meta = {'arguments': vars(a), 'video': str(source), 'fps': fps,

            'source_frame_count': total, 'timestamps_are_nominal': True, 'samples': samples}

    save_json(target / 'sampling.json', meta)

    return images, meta

def infer(a, images, metadata):

    import numpy as np

    import torch

    import transformers

    from transformers import AutoConfig, AutoProcessor, LlavaOnevisionForConditionalGeneration

    device = 'cuda:0' if a.device == 'cuda' else 'cpu'

    if a.device == 'cuda':

        if not torch.cuda.is_available():

            raise RuntimeError('CUDA unavailable. Install a CUDA-enabled PyTorch wheel; no implicit CPU fallback.')

        LOG.info('GPU: %s; capability: %s; torch: %s; runtime: %s',

                 torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0), torch.__version__, torch.version.cuda)

        if torch.cuda.get_device_capability(0) == (12, 0) and 'sm_120' not in torch.cuda.get_arch_list():

            LOG.warning('sm_120 not listed in this wheel. The following real GPU computation must pass.')

        x = torch.randn(128, 128, device=device)

        _ = x @ x

        torch.cuda.synchronize()

        del x, _

    dtype_name = a.dtype if a.dtype != 'auto' else ('float32' if a.device == 'cpu' else 'float16')

    if a.device == 'cpu' and dtype_name != 'float32':

        raise ValueError('Use float32 on CPU in this adapter.')

    if dtype_name == 'bfloat16' and a.device == 'cuda' and not torch.cuda.is_bf16_supported():

        raise RuntimeError('This GPU does not report BF16 support.')

    dtype = getattr(torch, dtype_name)

    shared = {'revision': a.revision, 'local_files_only': a.local_files_only, 'trust_remote_code': False}

    config = AutoConfig.from_pretrained(a.model_dir, **shared)

    if config.model_type != 'llava_onevision':

        raise ValueError(f'Expected llava_onevision, got {config.model_type!r}; PLLaVA/ST-LLM weights cannot be reused.')

    processor = AutoProcessor.from_pretrained(a.model_dir, use_fast=False, **shared)

    processor.tokenizer.padding_side = 'left'

    options = dict(config=config, torch_dtype=dtype, low_cpu_mem_usage=True,

                   device_map={'': device}, attn_implementation='sdpa', **shared)

    if a.load_in_4bit:

        from transformers import BitsAndBytesConfig

        options['quantization_config'] = BitsAndBytesConfig(

            load_in_4bit=True, bnb_4bit_quant_type='nf4',

            bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=dtype)

    LOG.info('Loading %s onto %s; no automatic RAM/disk offload.', a.model_dir, device)

    started = time.perf_counter()

    model = LlavaOnevisionForConditionalGeneration.from_pretrained(a.model_dir, **options).eval()

    load_seconds = time.perf_counter()-started

    times = ', '.join(f'{row["nominal_seconds"]:.3f}' for row in metadata['samples'])

    question = (f'The supplied video consists of {a.frames} sampled frames, in chronological order, '

                f'at nominal source-video times [{times}] seconds. Events between samples may be unobserved.\n' + a.question)

    messages = [{'role': 'user', 'content': [{'type': 'video'}, {'type': 'text', 'text': question}]}]

    prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

    clip = np.stack([np.asarray(image) for image in images])

    inputs = processor(text=prompt, videos=[clip], padding=True, return_tensors='pt')

    inputs = inputs.to(device=device, dtype=dtype)

    input_length = inputs['input_ids'].shape[1]

    limit = getattr(config.text_config, 'max_position_embeddings', None)

    if limit and input_length + a.max_new_tokens > limit:

        raise ValueError(f'Context overflow: {input_length}+{a.max_new_tokens}>{limit}; reduce --frames.')

    video_token_count = int((inputs['input_ids'] == config.video_token_index).sum().item())

    if a.device == 'cuda':

        torch.cuda.synchronize()

        torch.cuda.reset_peak_memory_stats()

    started = time.perf_counter()

    with torch.inference_mode():

        generated = model.generate(**inputs, max_new_tokens=a.max_new_tokens,

                                   do_sample=False, num_beams=1, use_cache=True)

    if a.device == 'cuda':

        torch.cuda.synchronize()

    elapsed = time.perf_counter()-started

    answer = processor.batch_decode(generated[:, input_length:], skip_special_tokens=True,

                                    clean_up_tokenization_spaces=False)[0].strip()

    return {**metadata, 'answer': answer, 'model_type': config.model_type,

            'resolved_model_commit': getattr(config, '_commit_hash', None),

            'model_load_seconds': load_seconds, 'generation_seconds': elapsed,

            'input_tokens': input_length, 'video_placeholder_tokens': video_token_count,

            'generated_tokens': generated.shape[1]-input_length,

            'input_shapes': {k:list(v.shape) for k,v in inputs.items() if hasattr(v, 'shape')},

            'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated()/2**30 if a.device == 'cuda' else None,

            'gpu': torch.cuda.get_device_name(0) if a.device == 'cuda' else None,

            'torch_version': torch.__version__, 'transformers_version': transformers.__version__,

            'python_version': platform.python_version(), 'resolved_dtype': dtype_name,

            'note': 'Standard OneVision video inference, not PruneVid pruning or GeoVARNet evidence reasoning. Timestamps are frame-index/FPS approximations.'}

def main():

    faulthandler.enable()

    logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')

    a = parse_args()

    target = Path(a.output)

    target.mkdir(parents=True, exist_ok=True)

    try:

        started = time.perf_counter()

        images, metadata = prepare(a, target)

        preparation_seconds = time.perf_counter()-started

        if a.prepare_only:

            LOG.info('Prepared %d frames, sampling.json and contact sheet; no model loaded.', len(images))

            return

        result = infer(a, images, metadata)

        result['preparation_seconds'] = preparation_seconds

        result['total_seconds'] = time.perf_counter()-started

        save_json(target / 'result.json', result)

        print(result['answer'])

    except Exception as error:

        save_json(target / 'error.json', {'error_type': type(error).__name__, 'message': str(error), 'arguments': vars(a)})

        LOG.exception('Run failed. No fallback answer was generated.')

        raise

if __name__ == '__main__':

    main()

Some caveat remain as before:

1. Libraries I used:

# Approximate environment for run_onevision_drone.py

# Use in a clean environment. GPU inference has not been validated here.

# --extra-index-url https://download.pytorch.org/whl/cu128

 torch==2.7.1+cu128

 torchvision==0.22.1+cu128

 transformers==4.51.3

 accelerate>=1.6,<2

 huggingface-hub>=0.30,<1

 numpy>=1.26,<2

 opencv-python==4.9.0.80

 pillow>=10,<12

 sentencepiece>=0.2,<0.3

 protobuf>=4.25,<6

 safetensors>=0.4.3,<1

 # Optional: uncomment for --load-in-4bit (not needed for the default 0.5B run).

 bitsandbytes>=0.47,<1

2. 

There is a checkpoint/code compatibility issue.

prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint. 

and there are three checkpoints involved which is suspicious:

LlaVa-OneVision

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.

1. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash LlaVA-OneVision compatibility issue.

 Cutting Tokens in Drone Video Analytics

The inference cost of aerial drone images and video even with sampling often runs into thousands of tokens and almost always repeated per query given the long tail of such analytics. GeoVARNet and our proposal asked: "Can representation-guided enrichment reduce computation without materially reducing event recall?" [1]. Players like NVIDIA are also asking similar questions but from a different perspective.

The Video Search and Summarization (VSS) blueprint is NVIDIA's reference architecture for video agents that perform:

• video search,

• visual Q&A,

• summarization,

• alerting,

• report generation,

using VLMs, LLMs, retrieval systems, and video analytics pipelines. They introduced Adaptive Efficient Video Sampling (EVS) that dynamically prunes unchanged visual patches per frame and batches VLM work around moments of activity.[3]

Their reported outcomes include:

• ~80% fewer VLM input tokens for a 60 minute video,

• ~50% faster summarization,

• 17% lower alert-contextualization latency,

• 46% higher concurrent real-time VLM stream capacity [3]

Although the benchmarks for drone video analytics such as UAVBench and EZBenchmark are diverse and require reprocessing, the EVS matters over traditional VLM pipelines that often progress through ingestion, frame sampling, VLM inference, Caption captures, and image embeddings and typically repeat that for every input even though almost nothing changed. EVS attempts to suppress that waste by detecting regions that are largely unchanged and avoiding feeding redundant visual information into the VLM. With their introduction of change detector, they reprocess only changed regions/chunks. This is referred to as token-pruning. GeoVARNet, on the other hand, engages in representation-guided enrichment. So, it can answer questions on an “evidence” basis rather than at “pixel” basis with a much broader definition of evidence and decision variables such as persistence, covisibility, revisits, place identity, entity history, observation coverage and rare-event likelihood. These are all first-class architectural concepts in GeoVARNet. 

For example, if a parking lot is observed six times, EVS sees parking lot looks mostly unchanged and reduces token usage while GeoVARNet could additionally infer same place, same appearance, same persistent state, already enriched, and high confidence. This would help avoid VLM calls, embedding calls, graph expansion, retrieval enrichment, and storage growth. That is a stronger form of reduction than token pruning alone.

There are others pursuing similar ideas though not with the same objectives. For example, Keyframe selection systems such as ORB-SLAM3, PKS (photogrammetric key-frame selection) and OpenVSLAM decide to only process frames that contribute new geometry. While GeoVARNet suggests using PKS, those techniques are for geometry redundancy, not semantic redundancy.

Modern video-RAG systems have a workflow that uses frame to embedding conversion and then determine distance from prior embedding to decide whether to retain or discard the frame and while this does not reduce the embedding generation cost for those frame and is used by many commercial video indexing pipelines, it leans on semantic novelty rather than geometry. This fails on tiny moving objects, vehicle swaps, transient events and drone revisits which GeoVARNet tries to preserve.

Another angle to reduce costs is detecting scene boundaries as found in video search, summarization and media indexing. If the scene did not change, why incur cost? But drone videos often have the same scene for different event which scene segmentation misses.

A more recent approach involves memory systems for Video LLMs that aim to improve with 

• StreamingVLM-style approaches

• memory-token approaches

• recurrent video transformers

These also obviate repetitive processing but by means of summary memory and sparse visual retention with a targeted transformer context length. While this reduces tokens, it does not retain as GeoVARNet does by compressing observations, places, revisits, and relationships at a higher-level abstraction.

The notable direction with NVIDIA VSS is the push towards Graph-RAG video systems where the architecture includes Graph-RAG, video search, agent workflows and video understanding thus generalizing for all video and not just urban drone videos. It completely avoids reprocessing the video to query the graph. 

This direction is a direct threat to GeoVARNet because it combines Embedding Novelty Sampling with Graph-RAG and Keyframe Selection and together it can already provide dramatic savings without introducing additional geometry decomposition and revisitation processing but GeoVARNet could demonstrate that it outperforms token-pruning, keyframe-selection, and embedding-novelty baselines at equal event-recall targets in terms of enrichment workload, VLM calls, storage growth, and event recall, When it does, it has a much stronger claim than reducing computation 

 PruneVid for aerial drone videos:

The previous article discussed using one of the three supported VLMs for PruneVid to improve token efficiency with the smallest model: PLLaVA. The other two are LLaVA-OneVision and ST-LLM (Liu et al., 2024b).  

ST-LLM feeds all raw, unpooled spatial-temporal tokens directly into the LLM backbone. For an egomotion camera where changes between consecutive frames are incremental, preserving the raw spatial-temporal progression allows the LLM to model subtle frame-to-frame displacements, motion cues, and panning trajectories. To prevent the massive context length from slowing down the system, it uses a dynamic masking strategy during training. This means it can drop redundant tokens where frames overlap significantly while forcing the model to retain a highly granular understanding of the continuous stream

The following implementation uses ST-LLM instead of the PLLaVA from the previous article:

"""Single-clip adapter for Visual-AI/PruneVid. Run from its repository root."""

"""ST-LLM official-API adapter; run from the ST-LLM repository root."""

import argparse

import json

from pathlib import Path

import time

def main():

    parser = argparse.ArgumentParser(description=__doc__)

    parser.add_argument('--video', required=True, help='Already-trimmed video clip')

    parser.add_argument('--ckpt-path', required=True)

    parser.add_argument('--qformer-path', required=True)

    parser.add_argument('--cfg-path', default='config/instructblipbase_stllm_conversation.yaml')

    parser.add_argument('--frames', type=int, default=16)

    parser.add_argument('--video-input', choices=['all', 'residual'], default='all')

    parser.add_argument('--max-new-tokens', type=int, default=160)

    parser.add_argument('--question', default='Describe terrain, structures, roads and moving objects. Distinguish camera motion from object motion. Do not invent GPS coordinates or distances.')

    parser.add_argument('--output', default='runs/drone_stllm.json')

    a = parser.parse_args()

    if a.frames < 16 or a.max_new_tokens < 1:

        parser.error('Use frames >=16 and max-new-tokens >=1.')

    for path in [a.video, a.qformer_path, a.cfg_path]:

        if not Path(path).is_file():

            parser.error('Missing file: ' + path)

    if not (Path(a.ckpt_path) / 'config.json').is_file():

        parser.error('ckpt-path must be the conversation_weight directory containing config.json.')

    import torch

    if not torch.cuda.is_available():

        raise RuntimeError('CUDA unavailable; fix PyTorch before loading ST-LLM.')

    if torch.cuda.get_device_capability(0) == (12, 0) and 'sm_120' not in torch.cuda.get_arch_list():

        raise RuntimeError('Install a Blackwell-compatible PyTorch wheel with sm_120 support.')

    x = torch.randn(64, 64, device='cuda')

    _ = x @ x

    torch.cuda.synchronize()

    from stllm.common.config import Config

    from stllm.common.registry import registry

    import stllm.models

    from stllm.conversation.conversation import Chat, CONV_instructblip_Vicuna0

    config_args = argparse.Namespace(cfg_path=a.cfg_path, options=None)

    cfg = Config(config_args)

    mc = cfg.model_cfg

    mc.ckpt = str(Path(a.ckpt_path).resolve())

    mc.llama_model = mc.ckpt

    mc.q_former_model = str(Path(a.qformer_path).resolve())

    mc.device_8bit = 0

    mc.use_grad_checkpoint = False

    # Keep residual modules when loading this residual-trained checkpoint.

    # Switch only its inference aggregation after loading.

    model_cls = registry.get_model_class(mc.arch)

    print('Loading official ST-LLM checkpoint; full FP16, no quantization/offload.', flush=True)

    model = model_cls.from_config(mc).to(device='cuda:0', dtype=torch.float16).eval()

    chat = Chat(model, device='cuda:0')

    chat.model.video_input = a.video_input

    conv = CONV_instructblip_Vicuna0.copy()

    images = []

    torch.cuda.reset_peak_memory_stats()

    start = time.perf_counter()

    with torch.inference_mode():

        chat.upload_video(str(Path(a.video).resolve()), conv, images,

                          num_frame=a.frames, text=a.question)

        chat.ask('###Human: ' + a.question + ' ###Assistant: ', conv)

        probe = conv.copy()

        probe.append_message(probe.roles[1], None)

        embeddings, _ = chat.get_context_emb_sim(probe, images, system=True)

        input_tokens = int(embeddings.shape[1])

        visual_tokens = int(images[0].shape[1])

        del embeddings

        context_limit = int(model.config.max_position_embeddings)

        required = input_tokens + a.max_new_tokens

        if required > context_limit:

            raise RuntimeError(f'Context overflow: {input_tokens} input + {a.max_new_tokens} output > {context_limit}. Reduce --frames; no silent truncation allowed.')

        answer, _ = chat.answer(conv=conv, img_list=images, do_sample=False,

                                num_beams=1, max_new_tokens=a.max_new_tokens,

                                max_length=context_limit)

    torch.cuda.synchronize()

    result = {'arguments': vars(a), 'answer': answer, 'visual_tokens': visual_tokens,

              'input_tokens': input_tokens, 'context_limit': context_limit,

              'encode_and_generate_seconds': time.perf_counter()-start,

              'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated()/2**30,

              'torch_version': torch.__version__, 'gpu': torch.cuda.get_device_name(0),

              'note': 'all retains all per-frame encoder/Q-Former output tokens, not raw pixels or all ViT patches; residual uses checkpoint global-local path.'}

    dest = Path(a.output)

    dest.parent.mkdir(parents=True, exist_ok=True)

    dest.write_text(json.dumps(result, indent=2), encoding='utf-8')

    print(answer)

if __name__ == '__main__':

    main()

Some caveat remain as before:

1. Libraries I used:

Python 3.10.22

torch 2.2.1+cu118

transformers 4.40.2

peft 0.10.0

accelerate 0.26.1

safetensors 0.4.2

mmcv 1.7.2

decord 0.6.0

moviepy 1.0.3

imageio 2.27.0

matplotlib 3.8.3

protobuf 4.23.4

sentencepiece 0.2.0

einops 0.6.1

2. 

There is a checkpoint/code compatibility issue.

prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint. 

and there are three checkpoints involved which is suspicious:

stllm-blackwell

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.

3. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash / ST-LLM compatibility issue.

#codingexercise: CodingExercise-11-10-2026.docx

 The previous article described an implementation. The following are from test runs:

1. Libraries I used:
   

Python 3.10.22

torch 2.2.1+cu118

transformers 4.40.2

peft 0.10.0

accelerate 0.26.1

safetensors 0.4.2

mmcv 1.7.2

decord 0.6.0

moviepy 1.0.3

imageio 2.27.0

matplotlib 3.8.3

protobuf 4.23.4

sentencepiece 0.2.0

einops 0.6.1

   2. There is a checkpoint/code compatibility issue.

prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint.

and there are three checkpoints involved which is suspicious:
pllava-7b

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.

 

3.  3. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash / pllava compatibility issue.

 PruneVid for aerial drone videos:

"""Single-clip adapter for Visual-AI/PruneVid. Run from its repository root."""
import argparse
import json
import math
import sys
import time
from pathlib import Path


def main():
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument('--video', required=True)
    p.add_argument('--output', default='drone_result')
    p.add_argument('--start', type=float, default=0)
    p.add_argument('--seconds', type=float, default=20)
    p.add_argument('--frames', type=int, default=16)
    p.add_argument('--prepare-only', action='store_true')
    p.add_argument('--model-dir', default='MODELS/pllava-7b')
    p.add_argument('--weight-dir')
    p.add_argument('--lora-alpha', type=int, default=14)
    p.add_argument('--question', default='Describe the visible terrain, structures, roads and moving objects. Distinguish camera motion from object motion. State uncertainty; do not invent GPS coordinates or metric distances.')
    p.add_argument('--alpha', type=float, default=0.4)
    p.add_argument('--tau', type=float, default=0.8)
    p.add_argument('--cluster-ratio', type=float, default=0.5)
    p.add_argument('--temporal-segment-ratio', type=float, default=0.25)
    p.add_argument('--selected-layer', type=int, default=10)
    p.add_argument('--max-new-tokens', type=int, default=160)
    a = p.parse_args()
    if a.frames < 4 or a.seconds <= 0 or a.start < 0:
        p.error('Require frames >= 4, seconds > 0 and start >= 0.')
    for name in ('alpha', 'cluster_ratio', 'temporal_segment_ratio'):
        if not 0 < getattr(a, name) <= 1:
            p.error(name + ' must be in (0, 1].')
    if not -1 <= a.tau <= 1:
        p.error('tau must be in [-1, 1].')
    import cv2
    import numpy as np
    from PIL import Image, ImageDraw
    video = Path(a.video).resolve()
    if not video.is_file():
        raise FileNotFoundError(video)
    target = Path(a.output)
    target.mkdir(parents=True, exist_ok=True)
    cap = cv2.VideoCapture(str(video))
    try:
        if not cap.isOpened():
            raise RuntimeError('Cannot open video; check path and codec.')
        fps = cap.get(cv2.CAP_PROP_FPS)
        total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
        if not math.isfinite(fps) or fps <= 0 or total <= 0:
            raise RuntimeError('Invalid FPS/frame count; transcode video before sampling.')
        first = int(a.start * fps)
        last = min(total, int((a.start + a.seconds) * fps)) - 1
        if last - first + 1 < a.frames:
            raise RuntimeError('Requested clip has fewer source frames than --frames.')
        indices = np.linspace(first, last, a.frames).round().astype(int)
        images, samples = [], []
        for i, idx in enumerate(indices):
            cap.set(cv2.CAP_PROP_POS_FRAMES, int(idx))
            ok, bgr = cap.read()
            if not ok:
                raise RuntimeError(f'Cannot decode source frame {idx}.')
            image = Image.fromarray(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))
            images.append(image)
            image.save(target / f'frame_{i:03d}.jpg')
            samples.append({'frame_index': int(idx), 'nominal_seconds': float(idx / fps)})
    finally:
        cap.release()
    sheet = Image.new('RGB', (4 * 320, math.ceil(a.frames / 4) * 205), 'white')
    draw = ImageDraw.Draw(sheet)
    for i, image in enumerate(images):
        thumb = image.copy()
        thumb.thumbnail((320, 180))
        x, y = (i % 4) * 320, (i // 4) * 205
        sheet.paste(thumb, (x, y))
        draw.text((x + 4, y + 182), f'{samples[i]["nominal_seconds"]:.2f}s', fill='black')
    sheet.save(target / 'contact_sheet.jpg')
    metadata = {'arguments': vars(a), 'video': str(video), 'fps': fps,
                'timestamps_are_nominal': True, 'samples': samples}
    (target / 'sampling.json').write_text(json.dumps(metadata, indent=2), encoding='utf-8')
    if a.prepare_only:
        print('Prepared sampled frames and contact sheet; no model loaded.')
        return
    import torch
    if not torch.cuda.is_available():
        raise RuntimeError('Official-model path requires working NVIDIA CUDA in this adapter. Use --prepare-only for CPU preprocessing.')
    if not torch.cuda.is_bf16_supported():
        raise RuntimeError('Official loader uses bfloat16; this CUDA device does not report BF16 support.')
    root = Path.cwd()
    if not (root / 'tasks/eval/model_utils.py').is_file():
        raise RuntimeError('Run from the cloned PruneVid root, with this script copied there.')
    sys.path.insert(0, str(root))
    from tasks.eval.model_utils import load_pllava
    from tasks.eval.eval_utils import conv_templates
    model_dir = str(Path(a.model_dir).resolve())
    weight_dir = str(Path(a.weight_dir or a.model_dir).resolve())
    if 'llava' not in model_dir:
        raise RuntimeError('Official loader selects PLLaVA by lowercase "llava" in model path; name the directory pllava-7b.')
    for directory in (model_dir, weight_dir):
        if not (Path(directory) / 'config.json').is_file():
            raise FileNotFoundError(f'Missing config.json in {directory}')
    torch.manual_seed(0)
    model, processor = load_pllava(
        model_dir, num_frames=a.frames, use_lora=True, weight_dir=weight_dir,
        lora_alpha=a.lora_alpha, pooling_shape=(a.frames, 12, 12),
        selected_layer=a.selected_layer, alpha=a.alpha, tau=a.tau,
        cluster_ratio=a.cluster_ratio, temporal_segment_ratio=a.temporal_segment_ratio)
    model = model.to('cuda').eval()
    conv = conv_templates['plain'].copy()
    conv.user_query(a.question, is_mm=True)
    inputs = processor(text=conv.get_prompt(), images=images, return_tensors='pt').to(model.device)
    if 'pixel_values' in inputs and inputs['pixel_values'] is not None:
        inputs['pixel_values'] = inputs['pixel_values'].to(dtype=next(model.parameters()).dtype)
    torch.cuda.synchronize()
    torch.cuda.reset_peak_memory_stats()
    started = time.perf_counter()
    with torch.inference_mode():
        tokens = model.generate(**inputs, media_type='video', do_sample=False,
                                max_new_tokens=a.max_new_tokens, use_cache=True)
    torch.cuda.synchronize()
    decoded = processor.batch_decode(tokens, skip_special_tokens=True,
                                     clean_up_tokenization_spaces=False)[0]
    answer = decoded.split(conv.roles[-1])[-1].removesuffix('</s>').strip()
    result = {**metadata, 'answer': answer, 'raw_decoded_text': decoded,
              'generation_seconds': time.perf_counter() - started,
              'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated() / 2**30,
              'gpu': torch.cuda.get_device_name(0), 'torch_version': torch.__version__,
              'note': 'No measured retained-token count; no baseline or accuracy claim.'}
    (target / 'result.json').write_text(json.dumps(result, indent=2), encoding='utf-8')
    print(answer)


if __name__ == '__main__':
    main()

Continued from previous post

 Just as AI observability evolved from model monitoring into decision accountability, perception observability is likely to evolve from object detection into world-model accountability. The systems that succeed will not simply identify what is visible. They will explain how every conclusion emerged, how it changed over time, and how an investigator can independently verify it. Even with an LLM-as-a-judge paradigm, the validation must remain independent.

The ability to defend a perception-derived belief through a complete chain of evidence becomes the defining characteristic of trustworthy aerial intelligence.

Continued from previous post

 

A related concept is spatial auditability.

Drone imagery is inherently geometric. Objects occupy locations within a coordinate system. Observations occur from different viewpoints, altitudes, sensor poses, and environmental conditions. Spatial auditability therefore requires the ability to justify not only that something exists, but where it exists. When a perception system reports that a vehicle is located at a particular coordinate, the system should retain the geometric chain that produced that estimate including camera parameters, pose estimates, registration transformations, geo-referencing models, local uncertainty estimates, and alignment quality metrics. This information helps location change from an assertion to evidence-supported knowledge.

Perception consistency monitoring extends perception observation and helps preserve observation lineage. Application observability tracks CPU utilization, latency, and service health; AI observability tracks hallucinations, latency, cost, and accuracy. Perception observability adds health signals specific to the perception process: observation drift, spatial inconsistency, temporal instability, identity fragmentation, registration degradation, evidence conflicts, and world-model uncertainty.

Imagine a drone corridor inspection system. For months, it consistently identifies pole locations within a few centimeters. Suddenly localization quality deteriorates because GPS quality worsens, illumination changes, or a visual registration model begins drifting.

A perception-observability layer would detect the degradation before analysts begin consuming corrupted outputs. Instead of waiting for incorrect conclusions, the system continuously validates the coherence of its own world model.

Another essential capability is revisiting awareness. Most real-world environments are observed repeatedly. Buildings are revisited. Roads are revisited. Assets are revisited. Infrastructure is revisited. Traditional computer vision often treats each revisit as a new problem. Perception observability treats revisits as opportunities to strengthen belief.

When a drone returns to a location, the system should compare current observations against historical knowledge. It should record whether observations reinforce, modify, or contradict prior beliefs. Each revisit becomes an audit event. The resulting system develops something resembling memory. Instead of a sequence of disconnected detections, it maintains a continuously evolving understanding of the environment. This becomes even more powerful when multiple sensing modalities are introduced. For example, modern aerial systems increasingly combine RGB cameras, thermal sensors, multispectral imagery, SAR, LiDAR, inertial measurements, and external geospatial databases.

Perception observability must also answer: Which sensor contributed to what evidence?

When a thermal anomaly triggers an alert, the system should indicate whether the conclusion relied solely on thermal data, was corroborated by visual imagery, was strengthened by contextual information, or faced contradictory evidence from other sources. Cross-modal evidence tracking makes sensor fusion transparent and inspectable. As perception systems become autonomous, auditability becomes a requirement rather than merely a useful capability. Future systems will increasingly trigger operational decisions automatically. They may dispatch inspectors, prioritize maintenance activities, initiate planning actions, direct robotic assets, or support public-safety decisions.

When decisions carry financial, operational, or safety consequences, stakeholders inevitably ask:

What did the system know?

When did the system know it?

Why did the system believe it?

What evidruly audit questions. The future of aerial intelligence may be defined by the ability to preserve and communicate evidence. In that future, the perception pipeline becomes a continuously auditable record of how observations become knowledge.

Continued from previous

Perception observability replaces confidence-centric thinking with evidence-centric thinking.

Every observation becomes traceable to the imagery, geometry, trajectory, sensor conditions, and prior observations that contributed to it. Instead of merely stating that a truck exists at a location, the system can demonstrate that the object was observed in fourteen separate passes, viewed from multiple headings, tracked across three days, localized through consistent geometric registration, and repeatedly associated with a particular operational area. The belief becomes explainable because its provenance is visible.

At the heart of perception observability lies the concept of observation lineage.

Observation lineage is analogous to data lineage in analytics systems. Just as data lineage traces a dashboard metric back to databases and transformations, observation lineage traces a world-state assertion back to the sensor evidence that created it.

Consider a simple statement:

"A new storage building appeared in Sector C."

A perception-observable system should be capable of revealing:

Which missions contributed evidence?

Which frames suggested change?

Which detections participated in?

Which registration models aligned imagery across time?

Which change detection algorithms trigger alerts?

Which observations contradicted the conclusion?

How confidence evolved as additional evidence accumulated.

The conclusion should emerge as the last link in an auditable chain of evidence.

This naturally leads to the second major pillar: temporal auditability.

Most vision systems evaluate frames independently. Humans, however, reason temporally. If a building suddenly appears in one image and disappears in the next ten observations, we become skeptical. We seek continuity.

Temporal auditability captures that intuition computationally.

Every world-state belief should maintain a history describing how it evolved. An analyst should be able to inspect the lifecycle of any object and answer questions such as:

When was this object first observed?

When did confidence increase?

When did evidence weaken?

When did the classification of an object change?

When was the object last confirmed?

What observations support its continued existence?

In a mature perception-observability framework, every entity becomes a living record rather than a static detection.

 Perception observability is making an autonomous sensing system explain itself. Just as application observability helps engineers understand what software is doing and AI observability helps engineers understand what models are doing, perception observability seeks to reveal what a sensing system believes about the world, why it believes it, how that belief evolved over time, and what evidence supports each conclusion.

That distinction is relevant to the world of drones. Most current AI observability platforms focus on the internals of a model. They monitor prompts, token usage, latency, hallucinations, outputs, and reasoning traces. They answer questions such as: Why did the model produce this answer? Why did the agent call this tool? Why did performance degrade after deployment?

Aerial drone systems face a fundamentally different challenge. As they struggle to be accurate in precision and recall, they must be trusted to perceive reality. A drone observing a city, a construction site, a power corridor, a disaster zone, or a battlefield is continuously transforming photons into beliefs about the physical world. In this context, an auditor could ask why does the system believes that this object exists at this location, with this identity, at this moment in time?

Perception observability attempts to answer that question.

Imagine a drone flying over an industrial facility every day for six months. During that time, it detects vehicles, buildings, equipment, containers, roads, vegetation changes, construction progress, temporary structures, and human activity. Traditional analytics pipelines often produce outputs as isolated events. A vehicle is detected. A building is classified. A region is segmented. An alert is issued.

Yet when an analyst asks where that conclusion came from, the system often has no meaningful answer beyond confidence scores. Confidence, however, is not evidence. A score of 0.97 does not state why the conclusion is true.

 Cutting Tokens in Drone Video Analytics

The inference cost of aerial drone images and video even with sampling often runs into thousands of tokens and almost always repeated per query given the long tail of such analytics. GeoVARNet and our proposal asked: "Can representation-guided enrichment reduce computation without materially reducing event recall?" [1]. Players like NVIDIA are also asking similar questions but from a different perspective.

The Video Search and Summarization (VSS) blueprint is NVIDIA's reference architecture for video agents that perform:

• video search,

• visual Q&A,

• summarization,

• alerting,

• report generation,

using VLMs, LLMs, retrieval systems, and video analytics pipelines. They introduced Adaptive Efficient Video Sampling (EVS) that dynamically prunes unchanged visual patches per frame and batches VLM work around moments of activity.[3]

Their reported outcomes include:

• ~80% fewer VLM input tokens for a 60 minute video,

• ~50% faster summarization,

• 17% lower alert-contextualization latency,

• 46% higher concurrent real-time VLM stream capacity [3]

Although the benchmarks for drone video analytics such as UAVBench and EZBenchmark are diverse and require reprocessing, the EVS matters over traditional VLM pipelines that often progress through ingestion, frame sampling, VLM inference, Caption captures, and image embeddings and typically repeat that for every input even though almost nothing changed. EVS attempts to suppress that waste by detecting regions that are largely unchanged and avoiding feeding redundant visual information into the VLM. With their introduction of change detector, they reprocess only changed regions/chunks. This is referred to as token-pruning. GeoVARNet, on the other hand, engages in representation-guided enrichment. So, it can answer questions on an “evidence” basis rather than at “pixel” basis with a much broader definition of evidence and decision variables such as persistence, covisibility, revisits, place identity, entity history, observation coverage and rare-event likelihood. These are all first-class architectural concepts in GeoVARNet. 

For example, if a parking lot is observed six times, EVS sees parking lot looks mostly unchanged and reduces token usage while GeoVARNet could additionally infer same place, same appearance, same persistent state, already enriched, and high confidence. This would help avoid VLM calls, embedding calls, graph expansion, retrieval enrichment, and storage growth. That is a stronger form of reduction than token pruning alone.

There are others pursuing similar ideas though not with the same objectives. For example, Keyframe selection systems such as ORB-SLAM3, PKS (photogrammetric key-frame selection) and OpenVSLAM decide to only process frames that contribute new geometry. While GeoVARNet suggests using PKS, those techniques are for geometry redundancy, not semantic redundancy.

Modern video-RAG systems have a workflow that uses frame to embedding conversion and then determine distance from prior embedding to decide whether to retain or discard the frame and while this does not reduce the embedding generation cost for those frame and is used by many commercial video indexing pipelines, it leans on semantic novelty rather than geometry. This fails on tiny moving objects, vehicle swaps, transient events and drone revisits which GeoVARNet tries to preserve.

Another angle to reduce costs is detecting scene boundaries as found in video search, summarization and media indexing. If the scene did not change, why incur cost? But drone videos often have the same scene for different event which scene segmentation misses.

A more recent approach involves memory systems for Video LLMs that aim to improve with 

• StreamingVLM-style approaches

• memory-token approaches

• recurrent video transformers

These also obviate repetitive processing but by means of summary memory and sparse visual retention with a targeted transformer context length. While this reduces tokens, it does not retain as GeoVARNet does by compressing observations, places, revisits, and relationships at a higher-level abstraction.

The notable direction with NVIDIA VSS is the push towards Graph-RAG video systems where the architecture includes Graph-RAG, video search, agent workflows and video understanding thus generalizing for all video and not just urban drone videos. It completely avoids reprocessing the video to query the graph. 

This direction is a direct threat to GeoVARNet because it combines Embedding Novelty Sampling with Graph-RAG and Keyframe Selection and together it can already provide dramatic savings without introducing additional geometry decomposition and revisitation processing but GeoVARNet could demonstrate that it outperforms token-pruning, keyframe-selection, and embedding-novelty baselines at equal event-recall targets in terms of enrichment workload, VLM calls, storage growth, and event recall, When it does, it has a much stronger claim than reducing computation 

 Procore AI and DroneDeploy integration 

This is highly relevant to drone video analytics market survey. Following Procore's formal acquisition of DroneDeploy, the two platforms have tightly integrated their technologies. The workflow combines DroneDeploy’s massive multi-modal visual database with Procore AI's deep project record system to build a "system of intelligence" for construction sites. [1, 2] 

How Procore AI Handles Drone Data 

Procore AI functions as the "reasoning and action" layer on top of DroneDeploy's visual "perception" layer. When a dock-based drone uploads raw spatial and visual data, Procore AI processes and contextualizes it through several automated workflows: [1] 

• Automated Observations & Issue Tagging: Advanced vision-language models (VLMs) scan the 2D orthomosaics and 3D models. If the AI flags a discrepancy (like an safety hazard or structural misalignment), it automatically populates Procore Observations, assigns a risk level, and routes it to the responsible subcontractor. [1, 2, 3] 

• BIM and Drawing Alignment: Procore AI matches the drone’s actual as-built maps directly against the project’s design drawings and 3D BIM models. It highlights variances instantly so project managers can visually confirm work-in-place before approving payments. [1, 2, 3, 4] 

• Autonomous Record Generation: The AI extracts metadata and visual highlights from daily drone flights to automatically fill out Procore Daily Logs and photo albums, removing the manual paperwork burden from site superintendents. [1, 2] 

• Predictive Conversational Queries: Using specialized "digital coworkers," teams can query the platform via text or voice (e.g., asking "Where are drywallers tracking to finish in Area A?"), and Procore AI will analyze the visual timeline to provide a specific, data-backed response. [1, 2] 

Built-In Analytics Currently Available in DroneDeploy 

DroneDeploy operates with powerful, native automated engines—specifically Progress AI and Safety AI—trained on over 20 trillion square feet of visual site data. Its built-in analytics capabilities span several core areas: [1, 2] 

Analytics Category	Core Capabilities & Features	Built-In Tools & Outputs
Progress AI & Scheduling	• Auto-detects trade work across 80+ specific construction categories.  • Compares physical site progress against uploaded Primavera P6 schedules.	• Schedule Deviation Reports: Flags delayed vs. planned start/end dates.  • Actualized XER Exports: Automatically outputs schedule files with real-world dates populated by AI.
Earthwork & Volumetrics	• Measures stockpile volumes, cuts/fills, slopes, and linear distances.  • Tracks daily rate of mass grading and excavation changes.	• Cut/Fill History Timelines: Tracks total earthwork volume changes over time to resolve subcontractor grading disputes.
Safety & Risk Mitigation	• Continuously scans site maps for safety compliance violations.  • Identifies missing guardrails, open trenches, and missing PPE with 95% accuracy.	• Daily Morning Reports: Delivers an automated summary of potential site stalls, hazards, and visual milestones straight to project inboxes.
3D Reality & Ground Analytics	• Processes high-density cloud photogrammetry up to 10,000 images.  • Integrates aerial maps with interior ground data.	• Interior Maps & Floor/Ceiling Scans: Overlays ceiling and floor conditions to sequence trades and review material layouts.  • Gaussian Splat Pre-checking: Validates 3D models before rendering to guarantee successful data generation.

 

Queries: 

The merger establishes a unified visual graph database where DroneDeploy’s voxel grids, 3D meshes, and point clouds are mapped to Procore’s relational project data (subcontractors, cost codes, schedules, and drawing sets). [1, 2, 3] 

Because the integration bridges DroneDeploy's multi-modal computer vision (spatial perception) with Procore AI's LLM-driven reasoning capabilities, users can query the unified system using highly advanced, multi-dimensional prompts. [1, 2, 3] 

1. Object Detection & Multi-Class Tracking 

These queries leverage DroneDeploy’s deep learning classification engines to isolate objects, monitor assets across flights, and correlate them directly with Procore's field logs. [1, 2] 

• Asset / Fleet Query: "Cross-reference today's aerial orthomosaic against our Procore Equipment Log. Detect all excavators on site, identify which ones have been stationary for more than 48 hours, and flag their exact GPS coordinates to the site logistics coordinator." 

• Inventory & Logistics Query: "Scan the designated structural steel laydown yard in the drone map. Identify, bound, and count all uninstalled W-shape beams. Verify if the detected count matches the bill of materials attached to Submittal #402." 

2. Size Estimation, Scale Resolution & Ground Truth 

DroneDeploy uses Ground Control Points (GCPs) and RTK-stabilized photogrammetry to provide 1–2% accuracy relative to ground lasers. These queries evaluate material states by converting pixels into volumetric metrics. [1, 2] 

• Volumetric Deviation Query: "Calculate the current total cut/fill volume of the grading area on the southern perimeter. Cross-examine this value against the earthwork baseline plan in Procore. If the variance exceeds 500 cubic yards, compute the exact financial impact using the unit rate specified in Contract Exhibit B." [1] 

• Structural Clearance Query: "Using yesterday’s high-density 3D mesh processing, isolate the concrete columns in Sector 3. Measure the exact clearance distances between column edges and the edge-of-slab. Flag any column that resolves to a clearance variance greater than \(\pm \)0.5 inches from the structural CAD drawing overlay." [1, 2] 

3. Cross-Scene Comparison & Automated Change Detection 

These queries execute pixel-by-pixel, vertex-by-vertex delta maps between distinct visual realities: the actual jobsite snapshot versus the project's digital design assets. [1] 

• BIM-vs-Reality Discrepancy Query: "Overlay the current DroneDeploy Ground Pro 3D interior ceiling map with the MEP structural model in Procore Models. Detect and display all spatial collisions where installed mechanical ductwork deviates from the original design path before the framing crew closes up the walls." 

• Pre-Pour Verification Query: "Analyze the high-resolution pre-pour map of the Level 3 deck. Check the placement density and spatial spacing of the rebar grid against the concrete reinforcement shop drawings. Generate a Procore Inspection card marking 'Pass' or 'Fail' based on structural code tolerances." [1, 2, 3, 4] 

4. Scene Revisits & Spatio-Temporal Context Querying 

This represents the highest tier of query capability, tracking the physical evolution of specific spatial coordinates over time and assigning trade accountability based on schedule records. [1] 

• Historical Forensics Query: "An active water leak was detected behind the drywall in Room 204. Go back through the localized spatio-temporal visual timeline of Room 204. Retrieve the historical DroneDeploy Ground walk-through video recorded the exact day before the drywall was hung. Isolate the plumbing assembly images from that date and look up which plumbing subcontractor signed off on the daily log." 

• Trade Sequencing & Lookahead Query: "Evaluate the rate of structural steel framing assembly over the last three weekly drone flights. Based on this physical installation velocity, project the realistic completion date for the roof deck. Compare this prediction against the active Primavera P6 schedule in Procore and notify the roofing subcontractor if they need to shift their mobilization window." [1, 2] 

DroneDeploy’s implementation specifics: 

DroneDeploy converts raw unstructured physical environments into mathematically queryable data by processing digital imagery through a cloud-based photogrammetry pipeline. It captures overlapping aerial or ground photos (using 60–80% overlap), aligns them using camera metadata (RTK/PPK coordinates), and constructs spatial datasets across three progressive architectural layers. [1, 2, 3, 4] 

1. 3D Point Clouds: The Structural Core 

The point cloud is the foundational geometric layer constructed immediately after image alignment. DroneDeploy's cloud graphics engines use Structure from Motion (SfM) and Multi-View Stereo (MVS) algorithms to build it: [1, 2] 

• Feature Extraction & Matching: Algorithms parse incoming JPEGs to identify static reference features invariant to lighting and camera positioning (e.g., using SIFT-based variations). These features are matched across hundreds of adjacent images. [1, 2] 

• Triangulation: By calculating the intersecting visual rays from known camera coordinates and focal lengths, the software estimates a sparse point cloud. [1, 2] 

• Densification: The pipeline expands the sparse map into a dense point cloud by interpolating depth values for virtually every full-resolution pixel. The resulting output is an unclassified, georeferenced dataset containing millions of distinct [X, Y, Z] spatial points, each assigned an accompanying [R, G, B] value extracted from the physical imagery. [1, 2, 3, 4] 

2. Voxel Grids: The Volumetric Optimization Layer 

While a raw point cloud accurately records spatial hits, it remains a collection of disjointed floating coordinates. To make this data readable for machine learning model inference, volumetric analysis, and rapid network streaming, DroneDeploy implements Voxelization: [1, 2] 

• Octree Space-Partitioning: Unstructured [X, Y, Z] point data is ingested into an octree data structure—a tree structure where each internal node has exactly eight children. The octree bounds the entire 3D project space and recursively divides it down into discrete cubic volume elements (voxels) based on a target grid minimum resolution (e.g., 2cm or 5cm cubes). [1, 2] 

• Occupancy Mapping & Decimation: The system maps points into discrete grid indices. If points populate a voxel, that cell is flagged as "occupied". This acts as a spatial compression tool, eliminating redundant density while producing a solid-looking, uniform 3D visualization. [1, 2] 

• Voxel Streaming: For robotics and web browsers, DroneDeploy streams these voxels as optimized binary chunks. This network-friendly representation slashes raw data bandwidth by roughly 68% compared to traditional point cloud streaming formats, enabling real-time multi-user 3D interaction without client-side lag. [1] 

3. 3D Meshes: The Textured Reality Canvas 

To generate continuous, realistic surfaces for visual walkthroughs and design checking, DroneDeploy transforms points into continuous geometric surfaces via Meshing: [1, 2] 

• Surface Reconstruction: Surface geometry algorithms (such as Poisson Surface Reconstruction) pass over the dense point cloud data to infer surface topology. It connects neighboring spatial points into a continuous web of millions of tiny interlocking polygons—typically triangles. 

• UV Mapping & Texturing: The raw geometric mesh outlines shape, but lacks color and realistic texture. DroneDeploy’s engine cuts the 3D polygon model flat into 2D coordinates (UV space) and project-stitches the original, high-resolution source photographs directly onto the geometric facets. 

• NeRF Integration: DroneDeploy also exposes specialized Neural Radiance Fields (NeRFs) datasets. By pairing traditional photogrammetry points with neural network scene learning, the system can approximate realistic lighting, transparency, and complex reflections, allowing teams to walk through highly detailed digital twins. [1, 2] 

#codingexercise: CodingExercise-11-02-2026.docx