Thursday, November 19, 2026

 Degla AI and DVSA-API compete most directly over the semantic interpretation of drone imagery: turning an operator’s question or objective into visual evidence, interpreting that evidence in context, and producing a response useful to a mission. Their principal difference is where that response leads. Degla positions its onboard intelligence as part of a continuous mission-decision loop, whereas DVSA-API’s documented query pipeline organizes selected aerial imagery and analytical tools around answering questions. Degla AI brings visual question answering closer to the aircraft’s next observation; DVSA provides an architecture for interrogating the observations already available. This distinction shows there is overlap: both must determine what is visible, what the operator means, which evidence matters, and whether that evidence is sufficient to support a conclusion. Degla relies on the onboard mission interpretation while DVSA-API is a curated-frame grounding and agent-based question-answering workflow

Degla’s software is described as a coordinated reasoning-and-perception stack, not a single model whose internal architecture has been disclosed. That said, it does use an onboard vision-language model to interpret imagery, understand intent, and guide mission planning and adaptation. It’s website separates “LLM reasoning,” which interprets objectives and connects them to decisions and responses, from “VLM perception,” which interprets camera scenes in the mission’s context, and from detection and tracking. These descriptions are compatible with several implementations: a shared multimodal model serving multiple roles, a VLM coupled to a separate language-model planner, or an orchestrated collection of models and tools. 

Consider the instruction, “Find the missing hiker wearing a red sweater.” This is language grounding: connecting a linguistic description to a particular visual candidate. The instruction supplies a conjunction of conditions—human subject, relevant clothing attribute, and a mission-specific identity hypothesis. A useful response must separate what the image supports from what the mission presumes. “A person wearing red is visible beside the trail” is an observational claim; “the missing hiker has been found” adds an identity conclusion that red clothing alone cannot establish. This makes no claim on precision or recall.

The query-response process could be a conditional multimodal inference: the response depends on the instruction, the visual observations, and whatever mission state or history the application supplies. Degla’s website specifically describes separate reasoning, perception, and tracking functions; a model-generated description need not itself perform localization or maintain identity across frames.

The connection to flight planning indicators imply possibility for a match. The next observation can then confirm, weaken, or reject the hypothesis. This is active perception: selecting observations to reduce uncertainty about a task. Within the scope of VLM analytics, Degla AI drone make progress towards goal by maximizing matches as new information emerges and with continued observation of people and vehicles. There is no reference to information-gain optimization or model generated subgoals or refinement of queries with human intervention during flight.

Temporal understanding is equally consequential. An aircraft changes viewpoint while the subject may also move, so successive images do not offer interchangeable evidence. A useful analytical system must distinguish a newly observed person from a previously considered candidate, preserve the basis for earlier conclusions, and recognize when a changing viewpoint resolves an ambiguity. Degla advertises detection, tracking, and maintaining visual contact, but its materials do not establish that these functions reside inside the VLM itself. This could mean that mission reasoning consumes both visual interpretations and persistent tracking outputs, with the precise division of responsibility undisclosed. This prevents an exaggerated comparison in which a VLM is credited with all the capabilities of the surrounding autonomy software.

Degla’s reported fine-tuning on expert pilots’ decisions is especially relevant to the quality of these query-response transitions. If the training examples associate observations and mission context with expert judgments, the model can potentially learn not only what an image contains but which interpretation matters next: continue searching, inspect an ambiguous candidate, or maintain observation of a likely match. That would make the learned behavior more task-specific than generic image captioning. 

Running inference locally changes the deployment characteristics of that analytical loop. Degla states that its reasoning, perception, and mission execution remain onboard, including offline missions; its launch announcement specifically says mission decisions do not depend on cloud inference. 

DVSA-API approaches the same semantic problem from a different software boundary. Its extraction workflow persists curated ImageEntity frames, and its chat path grounds responses through curated_frame_urls. Questions are handled by a three-agent arrangement involving knowledge search, a connected function agent, and a computer-vision tool agent within a shared time budget. Multiple Qwen execution options, including local alternatives. Accordingly, the comparison should not be simplified to “Degla is local, DVSA is cloud.” DVSA’s documented distinction is its API-mediated, curated-evidence workflow and tool composition; Degla’s is an onboard workflow specifically coupled to mission adaptation. DVSA could reprocess curated frames in full on each query and identifies motion-aware change detection and spatiotemporal enrichment but they are not completed capabilities.

Degla’s VLM proposition is compelling because it changes the purpose of a visual answer: the answer helps determine what evidence to gather next. DVSA-API’s opportunity is to make that decision-support process declared, inspectable, and accessible through an analytical interface. Degla competes with DVSA at the boundary between semantic interpretation and mission-directed questioning, while DVSA can differentiate through evidence-centered query orchestration. 



Wednesday, November 18, 2026

 Aerial drone video footage to knowledge.

For initiatives such as DVSA, governance at scale becomes economically viable only when video can be managed as knowledge rather than simply stored as footage. The following sample explains how:

index_to_graph.py reads the frame documents in an Azure AI Search index (schema: core/azure/index_schema.py) and builds a spatio-temporal knowledge graph with networkx.

Node From index field

Account, Video account_id, video_id (or the <account>/images/<video>/ blob path)

Frame id, frame_number (or frame<N> in path), timestamp, caption

Object objects plus description.objects from Azure AI Vision

Concept tags, labels, description.tags

Place location

Cell geotags, snapped to a lat/lon grid (default ~110 m)

index_to_graph.py: 


from __future__ import annotations


import argparse

import json

import math

import os

import re

from collections import Counter, defaultdict

from datetime import datetime, timezone

from itertools import combinations

from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple


import networkx as nx


# Fields to pull from the index. ``vector`` is deliberately left out: the graph

# needs semantics and coordinates, not 1536 floats per frame.

SELECT_FIELDS = [

    "id", "account_id", "video_id", "title", "section", "frame_number",

    "prev_id", "next_id", "path", "imageurl", "created", "timestamp",

    "caption", "description", "labels", "tags", "objects", "annotations",

    "boundingbox", "geotags", "location", "user", "session", "version",

    "acl_users", "acl_groups",

]


# 3 decimal places of lat/lon is roughly a 110 m cell: a sensible "same place"

# for low-altitude drone frames. Raise it for tighter neighbourhoods.

DEFAULT_CELL_PRECISION = 3


# A later frame in the same cell counts as a revisit when it belongs to another

# video, or comes this many seconds (or frames, without timestamps) later.

REVISIT_MIN_GAP = 60


_PATH_RE = re.compile(r"(?:^|/)([^/?]+)/images/([^/?]+)/frame(\d+)\.\w+")

_ID_RE = re.compile(r"-(\d+)$")

_NUM_RE = re.compile(r"-?\d+(?:\.\d+)?")



# --------------------------------------------------------------------------

# 1. Fetch documents from Azure AI Search

# --------------------------------------------------------------------------

def fetch_index_documents(search_client, account_id: Optional[str] = None,

                          video_id: Optional[str] = None,

                          user: Optional[str] = None) -> List[Dict[str, Any]]:

    """Page through every frame document for an account (and optionally a video).


    Uses the same scoping the connected search agent uses in

    ``core/azure/agents.py``: ``account_id eq '<id>'`` plus a ``startswith``

    on the blob path for one video. ``user`` adds an ACL check.

    """

    clauses = []

    if account_id:

        clauses.append(f"account_id eq '{_odata(account_id)}'")

        if video_id:

            clauses.append(

                f"startswith(path, '{_odata(account_id)}/images/{_odata(video_id)}/')"

                f" or video_id eq '{_odata(video_id)}'")

    if user:

        clauses.append(f"acl_users/any(u: u eq '{_odata(user)}')")

    flt = " and ".join(f"({c})" for c in clauses) or None

    results = search_client.search(search_text="*", filter=flt,

                                   select=SELECT_FIELDS, top=100000)

    return [dict(r) for r in results] # the SDK pages transparently



def _odata(value: str) -> str:

    return str(value).replace("'", "''")



def search_client_from_env(index_name: str):

    """A ``SearchClient`` from the same env vars ``AzureEnvironmentConfig`` reads."""

    from azure.core.credentials import AzureKeyCredential # noqa: PLC0415

    from azure.search.documents import SearchClient # noqa: PLC0415


    return SearchClient(endpoint=os.environ["AZURE_SEARCH_ENDPOINT"],

                        index_name=index_name,

                        credential=AzureKeyCredential(os.environ["AZURE_SEARCH_ADMIN_KEY"]))



# --------------------------------------------------------------------------

# 2. Normalize one index document into a frame record

# --------------------------------------------------------------------------

def normalize(doc: Dict[str, Any]) -> Dict[str, Any]:

    """Flatten a raw index document into the fields the graph needs."""

    path = doc.get("path") or doc.get("imageurl") or ""

    m = _PATH_RE.search(path.split("?")[0])

    path_account, path_video, path_frame = (m.groups() if m else (None, None, None))


    # Prefer the blob path's frame<N>: the id is <account>-<N:04d> with no video

    # in it, so it is only a fallback.

    frame_number = doc.get("frame_number")

    if frame_number is None:

        idm = _ID_RE.search(doc.get("id", ""))

        frame_number = int(path_frame) if path_frame else (int(idm.group(1)) if idm else None)


    desc = _parse_json(doc.get("description"))

    caption = doc.get("caption") or desc.get("caption") or ""

    tags = _clean(list(doc.get("tags") or []) + list(desc.get("tags") or []))

    labels = _clean(doc.get("labels") or [])

    objects = _clean(_split_objects(doc.get("objects")) + list(desc.get("objects") or []),

                     keep_dupes=True)


    return {

        "id": doc["id"],

        "account_id": str(doc.get("account_id") or path_account or ""),

        "video_id": str(doc.get("video_id") or path_video or "unknown"),

        "title": doc.get("title") or "",

        "section": doc.get("section") or "",

        "frame_number": frame_number,

        "prev_id": doc.get("prev_id"),

        "next_id": doc.get("next_id"),

        "time": _parse_time(doc.get("timestamp")),

        "created": doc.get("created"),

        "caption": caption,

        "dense_captions": list(desc.get("dense_captions") or []),

        "tags": tags,

        "labels": labels,

        "objects": objects, # duplicates kept = instance counts

        "annotations": doc.get("annotations") or "",

        "boundingbox": doc.get("boundingbox") or "",

        "latlon": _parse_latlon(doc.get("geotags")),

        "location": (doc.get("location") or "").strip(),

        "path": path.split("?")[0], # never store SAS tokens in a graph

        "user": doc.get("user") or "",

        "acl_users": list(doc.get("acl_users") or []),

    }



def _parse_json(text: Any) -> Dict[str, Any]:

    if isinstance(text, dict):

        return text

    try:

        value = json.loads(text) if text else {}

        return value if isinstance(value, dict) else {}

    except (TypeError, ValueError):

        return {}



def _split_objects(value: Any) -> List[str]:

    if not value:

        return []

    if isinstance(value, list):

        return value

    parsed = _parse_json(value) if value.strip().startswith("{") else None

    if parsed:

        return list(parsed.get("objects") or [])

    try:

        as_list = json.loads(value)

        if isinstance(as_list, list):

            return [str(v) for v in as_list]

    except ValueError:

        pass

    return re.split(r"[,;|]", value)



def _clean(items: Iterable[Any], keep_dupes: bool = False) -> List[str]:

    out = [str(i).strip().lower() for i in items if str(i).strip()]

    return out if keep_dupes else sorted(set(out))



def _parse_time(value: Any) -> Optional[float]:

    if not value:

        return None

    if isinstance(value, datetime):

        return value.timestamp()

    try:

        return datetime.fromisoformat(str(value).replace("Z", "+00:00")).timestamp()

    except ValueError:

        return None



def _parse_latlon(value: Any) -> Optional[Tuple[float, float]]:

    """``"('47.6', '-122.1')"``, ``"47.6,-122.1"`` or ``"('', '')"`` -> (lat, lon)."""

    nums = _NUM_RE.findall(str(value or ""))

    if len(nums) < 2:

        return None

    lat, lon = float(nums[0]), float(nums[1])

    return (lat, lon) if -90 <= lat <= 90 and -180 <= lon <= 180 else None



# --------------------------------------------------------------------------

# 3. Build the graph

# --------------------------------------------------------------------------

def cell_id(lat: float, lon: float, precision: int = DEFAULT_CELL_PRECISION) -> str:

    f = 10 ** precision

    return f"cell:{math.floor(lat * f)}_{math.floor(lon * f)}@{precision}"



def build_graph(docs: Sequence[Dict[str, Any]],

                precision: int = DEFAULT_CELL_PRECISION) -> nx.MultiDiGraph:

    """Build the knowledge graph from raw index documents.


    Node kinds: Account, Video, Frame, Object, Concept, Place, Cell.

    Edge relations: HAS_VIDEO, HAS_FRAME, NEXT, CONTAINS, DEPICTS, AT_PLACE,

    IN_CELL, ADJACENT, CO_OCCURS, REVISITS.

    """

    G = nx.MultiDiGraph(precision=precision)

    frames = [normalize(d) for d in docs]

    by_video: Dict[str, List[Dict[str, Any]]] = defaultdict(list)


    for f in frames:

        acc = f"account:{f['account_id']}"

        vid = f"video:{f['account_id']}/{f['video_id']}"

        fid = f"frame:{f['id']}"

        G.add_node(acc, kind="Account", name=f["account_id"])

        G.add_node(vid, kind="Video", name=f["video_id"], title=f["title"])

        G.add_node(fid, kind="Frame", name=f["id"], video=f["video_id"],

                   frame_number=f["frame_number"], time=f["time"], caption=f["caption"],

                   dense_captions=f["dense_captions"], boundingbox=f["boundingbox"],

                   path=f["path"], lat=(f["latlon"] or (None, None))[0],

                   lon=(f["latlon"] or (None, None))[1], acl_users=f["acl_users"])

        _edge(G, acc, vid, "HAS_VIDEO")

        _edge(G, vid, fid, "HAS_FRAME")

        by_video[vid].append(f)


        for name, count in Counter(f["objects"]).items():

            _edge(G, fid, _entity(G, "Object", name), "CONTAINS", count=count)

        for name in set(f["tags"]) | set(f["labels"]):

            src = "label" if name in f["labels"] else "tag"

            _edge(G, fid, _entity(G, "Concept", name), "DEPICTS", source=src)

        if f["location"]:

            _edge(G, fid, _entity(G, "Place", f["location"]), "AT_PLACE")

        if f["latlon"]:

            cid = cell_id(*f["latlon"], precision)

            G.add_node(cid, kind="Cell", name=cid, lat=f["latlon"][0], lon=f["latlon"][1])

            _edge(G, fid, cid, "IN_CELL")


        for a, b in combinations(sorted(set(f["objects"])), 2):

            _bump(G, _entity(G, "Object", a), _entity(G, "Object", b), "CO_OCCURS")


    _link_sequences(G, by_video)

    _link_adjacent_cells(G, precision)

    _link_revisits(G)

    return G



def _entity(G: nx.MultiDiGraph, kind: str, name: str) -> str:

    node = f"{kind.lower()}:{name}"

    if node not in G:

        G.add_node(node, kind=kind, name=name)

    return node



def _edge(G: nx.MultiDiGraph, u: str, v: str, rel: str, **attrs: Any) -> None:

    G.add_edge(u, v, key=rel, rel=rel, **attrs)



def _bump(G: nx.MultiDiGraph, u: str, v: str, rel: str) -> None:

    if G.has_edge(u, v, key=rel):

        G[u][v][rel]["weight"] += 1

    else:

        _edge(G, u, v, rel, weight=1)



def _order_key(f: Dict[str, Any]) -> Tuple[float, float]:

    return (f["time"] if f["time"] is not None else math.inf,

            f["frame_number"] if f["frame_number"] is not None else math.inf)



def _link_sequences(G: nx.MultiDiGraph, by_video: Dict[str, List[Dict[str, Any]]]) -> None:

    """NEXT edges: stored ``next_id`` when present, else frame/time order."""

    for frames in by_video.values():

        ids = {f["id"] for f in frames}

        ordered = sorted(frames, key=_order_key)

        for i, f in enumerate(ordered):

            nxt = f["next_id"] if f["next_id"] in ids else (

                ordered[i + 1]["id"] if i + 1 < len(ordered) else None)

            if nxt:

                g = next(x for x in frames if x["id"] == nxt)

                gap = (g["time"] - f["time"]) if None not in (f["time"], g["time"]) else None

                _edge(G, f"frame:{f['id']}", f"frame:{nxt}", "NEXT",

                      seconds=gap, frames=(g["frame_number"] or 0) - (f["frame_number"] or 0))



def _link_adjacent_cells(G: nx.MultiDiGraph, precision: int) -> None:

    cells = {n for n, k in G.nodes(data="kind") if k == "Cell"}

    for c in cells:

        lat_i, lon_i = map(int, c[5:].split("@")[0].split("_"))

        for dlat in (-1, 0, 1):

            for dlon in (-1, 0, 1):

                other = f"cell:{lat_i + dlat}_{lon_i + dlon}@{precision}"

                if other != c and other in cells:

                    _edge(G, c, other, "ADJACENT")



def _link_revisits(G: nx.MultiDiGraph) -> None:

    """REVISITS: a later frame of the same cell, from another flight or after a gap."""

    for cell in [n for n, k in G.nodes(data="kind") if k == "Cell"]:

        visits = sorted((G.nodes[u] for u, _, r in G.in_edges(cell, keys=True) if r == "IN_CELL"),

                        key=lambda n: (n["time"] if n["time"] is not None else math.inf,

                                       n["frame_number"] or 0))

        for first, later in zip(visits, visits[1:]):

            other_flight = first["video"] != later["video"]

            if first["time"] is not None and later["time"] is not None:

                gap = later["time"] - first["time"]

            else:

                gap = (later["frame_number"] or 0) - (first["frame_number"] or 0)

            if other_flight or gap >= REVISIT_MIN_GAP:

                unit = "s" if first["time"] is not None and later["time"] is not None else "frames"

                _edge(G, f"frame:{later['name']}", f"frame:{first['name']}", "REVISITS",

                      cell=cell, gap=gap, gap_unit=unit, other_flight=other_flight)



# --------------------------------------------------------------------------

# 4. Graph-explorer style queries

# --------------------------------------------------------------------------

def _frames_with(G: nx.MultiDiGraph, node: str, rel: str) -> List[str]:

    return [u for u, _, r in G.in_edges(node, keys=True) if r == rel] if node in G else []



def _sort_frames(G: nx.MultiDiGraph, frames: Iterable[str]) -> List[str]:

    return sorted(frames, key=lambda f: (G.nodes[f]["time"] or math.inf,

                                         G.nodes[f]["video"], G.nodes[f]["frame_number"] or 0))



def object_timeline(G: nx.MultiDiGraph, name: str) -> List[Dict[str, Any]]:

    """Every frame where an object (or a concept, as a fallback) appears, in order."""

    name = name.lower()

    hits = _frames_with(G, f"object:{name}", "CONTAINS") or _frames_with(G, f"concept:{name}", "DEPICTS")

    rows = []

    for f in _sort_frames(G, hits):

        n = G.nodes[f]

        count = G.get_edge_data(f, f"object:{name}", "CONTAINS", {}).get("count", 1)

        rows.append({"frame": n["name"], "video": n["video"], "frame_number": n["frame_number"],

                     "time": n["time"], "lat": n["lat"], "lon": n["lon"], "count": count,

                     "caption": n["caption"]})

    return rows



def frames_near(G: nx.MultiDiGraph, lat: float, lon: float, rings: int = 1,

                start: Optional[float] = None, end: Optional[float] = None) -> List[str]:

    """Frames within ``rings`` cells of a point, optionally inside a time window."""

    p = G.graph["precision"]

    f = 10 ** p

    lat_i, lon_i = math.floor(lat * f), math.floor(lon * f)

    hits = set()

    for dlat in range(-rings, rings + 1):

        for dlon in range(-rings, rings + 1):

            hits.update(_frames_with(G, f"cell:{lat_i + dlat}_{lon_i + dlon}@{p}", "IN_CELL"))


    def in_window(fr: str) -> bool:

        t = G.nodes[fr]["time"]

        return t is None or ((start is None or t >= start) and (end is None or t <= end))

    return _sort_frames(G, (h for h in hits if in_window(h)))



def revisited_places(G: nx.MultiDiGraph, min_visits: int = 2) -> List[Dict[str, Any]]:

    """Cells seen by several frames, with which videos saw them."""

    rows = []

    for c, kind in G.nodes(data="kind"):

        if kind != "Cell":

            continue

        frames = _frames_with(G, c, "IN_CELL")

        if len(frames) >= min_visits:

            rows.append({"cell": c, "lat": G.nodes[c]["lat"], "lon": G.nodes[c]["lon"],

                         "frames": len(frames),

                         "videos": sorted({G.nodes[f]["video"] for f in frames})})

    return sorted(rows, key=lambda r: (-len(r["videos"]), -r["frames"]))



def co_occurring(G: nx.MultiDiGraph, name: str, top: int = 5) -> List[Tuple[str, int]]:

    """Objects most often seen in the same frame as ``name``."""

    node = f"object:{name.lower()}"

    if node not in G:

        return []

    counts: Counter = Counter()

    for u, v, r, d in list(G.out_edges(node, keys=True, data=True)) + \

            list(G.in_edges(node, keys=True, data=True)):

        if r == "CO_OCCURS":

            counts[G.nodes[v if u == node else u]["name"]] += d["weight"]

    return counts.most_common(top)



def spatiotemporal_context(G: nx.MultiDiGraph, frame_ids: Sequence[str],

                           max_lines: int = 40) -> str:

    """Context block for the chat agents, grown from frames search already found.


    For each retrieved frame it adds: what it contains, the frames just before

    and after it, and earlier frames of the same place (revisits). Pass the

    ids the connected search agent returned; prepend the result to the prompt

    alongside the ``[Context: account_id=..., video_id=...]`` line.

    """

    lines: List[str] = []

    for raw in frame_ids:

        f = raw if raw.startswith("frame:") else f"frame:{raw}"

        if f not in G:

            continue

        n = G.nodes[f]

        objs = [f"{G.nodes[v]['name']} x{d.get('count', 1)}"

                for _, v, r, d in G.out_edges(f, keys=True, data=True) if r == "CONTAINS"]

        where = f" at ({n['lat']:.5f}, {n['lon']:.5f})" if n["lat"] is not None else ""

        lines.append(f"Frame {n['name']} (video {n['video']}, #{n['frame_number']}"

                     f"{_when(n)}){where}: {n['caption']}"

                     + (f" | objects: {', '.join(objs)}" if objs else ""))

        before, after, revisits = [], [], []

        for u, v, r, d in list(G.in_edges(f, keys=True, data=True)) + \

                list(G.out_edges(f, keys=True, data=True)):

            if r == "NEXT":

                other = G.nodes[u if v == f else v]

                (before if v == f else after).append(

                    f" {'before' if v == f else 'after'}: #{other['frame_number']}"

                    f"{_when(other)} {other['caption']}")

            elif r == "REVISITS":

                other = G.nodes[v if u == f else u]

                kind = "another flight" if d["other_flight"] else f"{d['gap']:.0f} {d['gap_unit']} apart"

                revisits.append(f" same place, {'earlier' if u == f else 'later'} ({kind}): "

                                f"{other['name']}{_when(other)} {other['caption']}")

        lines.extend(before + after + revisits)

        if len(lines) >= max_lines:

            break

    return "\n".join(lines[:max_lines])



def _when(n: Dict[str, Any]) -> str:

    return (", " + datetime.fromtimestamp(n["time"], timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ")

            if n.get("time") is not None else "")



# --------------------------------------------------------------------------

# 5. Exports for graph explorers

# --------------------------------------------------------------------------

def _flat(attrs: Dict[str, Any]) -> Dict[str, Any]:

    """GraphML/Cypher-safe attributes: drop None, join lists."""

    out = {}

    for k, v in attrs.items():

        if v is None:

            continue

        out[k] = "; ".join(map(str, v)) if isinstance(v, (list, tuple)) else v

    return out



def export_graph(G: nx.MultiDiGraph, out_dir: str) -> Dict[str, str]:

    os.makedirs(out_dir, exist_ok=True)

    flat = nx.MultiDiGraph()

    flat.add_nodes_from((n, _flat(d)) for n, d in G.nodes(data=True))

    flat.add_edges_from((u, v, k, _flat(d)) for u, v, k, d in G.edges(keys=True, data=True))


    paths = {"graphml": os.path.join(out_dir, "dvsa_kg.graphml"),

             "json": os.path.join(out_dir, "dvsa_kg.json"),

             "cypher": os.path.join(out_dir, "dvsa_kg.cypher")}

    nx.write_graphml(flat, paths["graphml"])

    with open(paths["json"], "w", encoding="utf-8") as fh:

        json.dump(nx.node_link_data(flat, edges="edges"), fh, indent=1)

    with open(paths["cypher"], "w", encoding="utf-8") as fh:

        fh.write(to_cypher(flat))

    return paths



def to_cypher(G: nx.MultiDiGraph) -> str:

    """openCypher MERGE script (Neo4j Browser; Apache AGE wraps it in cypher())."""

    def lit(v: Any) -> str:

        if isinstance(v, bool):

            return "true" if v else "false"

        if isinstance(v, (int, float)):

            return repr(v)

        return "'" + str(v).replace("\\", "\\\\").replace("'", "\\'") + "'"


    def props(d: Dict[str, Any]) -> str:

        return ", ".join(f"{k}: {lit(v)}" for k, v in d.items() if k != "kind")


    out = []

    for n, d in G.nodes(data=True):

        out.append(f"MERGE (n:{d['kind']} {{key: {lit(n)}}}) SET n += {{{props(d)}}};")

    for u, v, k, d in G.edges(keys=True, data=True):

        extra = {x: y for x, y in d.items() if x != "rel"}

        out.append(f"MATCH (a {{key: {lit(u)}}}), (b {{key: {lit(v)}}}) "

                   f"MERGE (a)-[r:{k}]->(b)" + (f" SET r += {{{props(extra)}}}" if extra else "") + ";")

    return "\n".join(out) + "\n"



# --------------------------------------------------------------------------

# Demo data shaped exactly like build_document() output

# --------------------------------------------------------------------------

def demo_documents() -> List[Dict[str, Any]]:

    """Two short flights over the same parking lot, as the indexer stores them."""

    base = "https://sadronevideo.blob.core.windows.net/input/2/images/{v}/frame{n}.jpg?sv=redacted"

    shots = [ # video, frame, minute, lat, lon, caption, objects, tags

        ("17", 0, 0, 47.67412, -122.12105, "a parking lot with cars",

         ["car", "car", "tree"], ["parking", "outdoor"]),

        ("17", 30, 1, 47.67431, -122.12088, "cars parked near a building",

         ["car", "building"], ["parking", "building"]),

        ("17", 60, 2, 47.67502, -122.12011, "a truck on a road",

         ["truck", "road"], ["road", "outdoor"]),

        ("17", 90, 3, 47.67588, -122.11940, "an empty field next to trees",

         ["tree"], ["field", "grass"]),

        ("21", 0, 120, 47.67415, -122.12101, "a nearly empty parking lot",

         ["car", "tree"], ["parking", "outdoor"]),

        ("21", 30, 121, 47.67505, -122.12015, "a truck and a person on a road",

         ["truck", "person", "road"], ["road"]),

    ]

    docs = []

    for v, n, minute, lat, lon, cap, objs, tags in shots:

        docs.append({

            "id": f"2-{n:04d}" if v == "17" else f"2-{1000 + n:04d}",

            "account_id": "2",

            "description": json.dumps({"caption": cap, "tags": tags, "objects": objs,

                                       "dense_captions": []}),

            "caption": cap, "labels": [], "tags": [], "objects": "",

            "boundingbox": "0,0,1280,720",

            "geotags": str((str(lat), str(lon))),

            "path": base.format(v=v, n=n),

            "timestamp": f"2026-11-01T10:{minute % 60:02d}:00+00:00" if minute < 60

            else f"2026-11-01T12:{minute % 60:02d}:00+00:00",

            "user": "ravi", "session": "demo",

        })

    return docs



# --------------------------------------------------------------------------

# CLI

# --------------------------------------------------------------------------

def main(argv: Optional[Sequence[str]] = None) -> None:

    p = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])

    src = p.add_mutually_exclusive_group(required=True)

    src.add_argument("--demo", action="store_true", help="use built-in sample documents")

    src.add_argument("--from-json", help="JSON file with a list of index documents")

    src.add_argument("--index", help="Azure AI Search index name, e.g. dvsa-index-s-2")

    p.add_argument("--account-id")

    p.add_argument("--video-id")

    p.add_argument("--precision", type=int, default=DEFAULT_CELL_PRECISION)

    p.add_argument("--object", default="car", help="object for the timeline query")

    p.add_argument("--out", default="out/kg")

    a = p.parse_args(argv)


    if a.demo:

        docs = demo_documents()

    elif a.from_json:

        with open(a.from_json, encoding="utf-8") as fh:

            docs = json.load(fh)

    else:

        docs = fetch_index_documents(search_client_from_env(a.index), a.account_id, a.video_id)


    G = build_graph(docs, precision=a.precision)

    kinds = Counter(k for _, k in G.nodes(data="kind"))

    rels = Counter(k for _, _, k in G.edges(keys=True))

    print(f"{len(docs)} documents -> {G.number_of_nodes()} nodes, {G.number_of_edges()} edges")

    print(" nodes:", dict(kinds))

    print(" edges:", dict(rels))


    print(f"\nTimeline of '{a.object}':")

    for r in object_timeline(G, a.object):

        print(f" video {r['video']} #{r['frame_number']} x{r['count']}: {r['caption']}")

    print(f"\nCo-occurs with '{a.object}':", co_occurring(G, a.object))

    print("\nRevisited places:")

    for r in revisited_places(G)[:5]:

        print(f" {r['cell']} seen in {r['frames']} frames by videos {r['videos']}")


    first = object_timeline(G, a.object)

    if first:

        print("\nAgent context for", first[0]["frame"] + ":")

        print(spatiotemporal_context(G, [first[0]["frame"]]))


    print("\nWrote:", export_graph(G, a.out))



if __name__ == "__main__":

    main()




Tuesday, November 17, 2026

 From Drone Video Archives to Governed Autonomy Evidence

Is video search already a competitive market, and where could DVSA still differentiate? Yes, it is already crowded. Generic video search is not the moat. The stronger position is a governed evidence layer. It connects video to missions, telemetry, autonomy decisions, operator interventions, and software lineage. One correction comes first. Camera-based perception does not mean an operator continuously records, streams, or retains video. Amazon states that its delivery-drone cameras are not viewed live by people. Wing describes low-resolution navigation imagery, not a continuous FPV archive. These distinctions shrink the addressable market. Vendor capabilities below are advertised functionality, not independently verified accuracy, completeness, or forensic admissibility. Architectural proposals and strategic inferences are marked as such.

DVSA faces four competitive forces. Public clouds make video understanding and retrieval easy to assemble. Fleet operators already own perception, mission-management, and data pipelines. Independent platforms turn enterprise video libraries into searchable assets. Insider-risk products pair desktop recordings with activity metadata and investigation workflows. AWS publishes architectures for frame-based analysis, shot-based understanding, and multimodal embedding retrieval. Microsoft provides timestamped video insights and search. TwelveLabs separates retrieval models from video-language models. Proofpoint and Teramind link visual recordings to activity logs, risk events, and forensic investigation. The opportunity is therefore not to make video searchable. It is to make operational evidence discoverable, interpretable, policy-controlled, and reproducible across heterogeneous autonomy systems.

Much of the generic pipeline already exists. AWS Bedrock Data Automation extracts chapters, text, and prompt-defined objects, and AWS documents frame, shot, and embedding designs for video search. Azure AI Video Indexer produces time-ranged insights, exact-moment search, and multimodal summaries, in the cloud and on Azure Arc-enabled edge. Google documents Gemini video understanding, timestamped chapters, and multimodal embeddings. Extraction, embeddings, and retrieval are therefore replaceable infrastructure. Mission-aware evidence management must be distinguished from general-purpose indexing. None of these is necessarily a turnkey product that ingests live fleet feeds, synchronizes telemetry, interprets safety events, and exports investigation packages. They do cut the effort to build the generic layer. AWS's own reference architecture combines orchestration, inference, vector search, and timestamp metadata storage.

A new pattern separates retrieval from interpretation. Retrieval models locate candidate video intervals. Video-language models then describe or answer questions about selected intervals. TwelveLabs exposes this through Marengo for embeddings and Pegasus for video-to-text, and AWS offers both through Bedrock. Independent model providers can thus be partners and cloud-enabled competitors at once. For DVSA the proposed flow runs from video and telemetry ingestion to time-aligned segments and event records, then to geospatial, temporal, structured, and vector indexes. It continues to candidate retrieval and targeted video-language analysis, and ends in an evidence-linked answer and investigation package. A generated description is not a complete representation of its source. A description index and a multimodal embedding index preserve different information, and Microsoft's migration documentation distinguishes the two. Product lifecycles are also an architectural risk. Azure AI Vision Video Retrieval was retired on June 30, 2025, with users directed to Content Understanding and Azure AI Search. DVSA should keep a provider-neutral media catalog, segment identifiers, timestamps, and provenance records. Embeddings and vendor insights should be versioned derivatives, never the only description of the archive.

Fleet operators have sophisticated stacks and incompletely disclosed archives. Public evidence shows heavy investment in autonomy and orchestration. It does not show that major platforms keep comprehensive, searchable FPV archives. Retailers and marketplaces must be separated from the operators who supply the vehicles. Walmart works with Wing and Zipline. Uber Eats works with robot partners. DoorDash combines partners with its own Dot platform. Their access to raw sensor data should not be assumed equal. Amazon Prime Air uses onboard cameras and radar for obstacle detection and delivery decisions. AWS describes Step Functions and Batch processing of lidar, aerial imagery, and geospatial inputs, plus ECS and Fargate mission-planning workflows. What is not public is any general-purpose retained FPV archive, its search interface, retention schedule, or third-party access model. Wing supplies Walmart's drone operations and describes low-resolution overhead navigation imagery, flight plans, drone locations, and flight histories. It is unknown whether Walmart receives raw imagery, whether collection is continuous, how long images are kept, or whether internal semantic search exists. Walmart documents its Zipline expansion, but a partnership alone does not show video storage, indexing, evidence export, or retailer access. DoorDash's Dot uses lidar, radar, and cameras, and its Autonomous Delivery Platform orchestrates humans, robots, and drones. Its fleet-wide video search, raw-video availability, retention, and evidentiary lineage are not public. Uber lists Avride, Cartken, Coco, and Serve as partners. A unified Uber-owned sensor archive or common forensic schema is not established.

Amazon shows the strength of the in-house alternative. Its published architecture already spans geospatial processing, delivery-point scoring, trajectory planning, contingency planning, and preflight orchestration. This is far more than a camera stream attached to an order. The strategic inference is that mature operators may build evidence capabilities into existing systems rather than buy a replacement. DVSA would have to offer something complementary: cross-system investigation, portable evidence packages, consistent policy enforcement, or cheaper management of video and its derivatives. Marketplaces add an integration problem. DoorDash picks delivery modes by speed, cost, location, distance, weight, and order traits. Uber integrates services from several autonomy companies. The marketplace holds order and customer-service context. The operator holds mission and sensor context. A remote-assistance system may hold intervention records. Other systems may hold model releases and maintenance history. This is a proposed integration model, not a claim about any company's internal schema. DVSA could target that boundary with standardized incident requests and evidence manifests. A retailer needs a bounded answer to what happened during a delivery, not open access to an operator's camera archive.

Independent platforms already serve enterprise customers. They show both competition and reusable patterns. TwelveLabs offers multimodal retrieval and video-language analysis beyond transcripts. Its Qencode case study adds searchable video as an output of an existing encoding workflow, and searches return timestamped clips. The reusable pattern is enrichment inside an existing media pipeline: encode, generate semantic representations, and publish searchable time ranges. DVSA need not make operators replace their media systems. It can attach evidence generation and governance to ingest, encoding, and mission-closeout workflows. TwelveLabs also publishes a retail-CCTV tutorial that combines library search, structured analysis, and playable timestamped evidence. It overlaps with DVSA's proposed investigation experience, though a tutorial is not a production deployment. Veritone's Digital Media Hub combines media management, AI metadata, search, permissions, sharing, and distribution. Its customers include the San Francisco Giants and Vital Voices, and its PROGRESS announcement describes automated cataloging of a large film archive. These are vendor-reported deployments, not independent evaluations. Buyers purchase workflows around video, not an embedding model. Veritone annotates video segments and exports time-bounded metadata as JSON and XML. DVSA should therefore invest in the full investigator workflow. Find the event. Review the source interval. Inspect the metadata. Record annotations and disagreements. Apply access and disclosure policies. Export a reproducible evidence package. Snowflake adds another pressure. Its TwelveLabs guide stores timestamped embeddings, retrieves them with Cortex Search, and plays results at matching timestamps, with analysis in optional private preview. Video intelligence is becoming a workload inside enterprise data platforms. DVSA should interoperate with them while owning the autonomy-specific evidence model.

Desktop monitoring is a relevant adjacent market. Products already record or reconstruct screen activity, index it, and tie it to security investigations. Video there needs careful reading. Some products use screenshot sequences, activity-triggered capture, and changed-region storage rather than full-motion video, and Proofpoint documents these methods. Teramind advertises live desktop viewing, historical playback, OCR search of recorded screens, indexed activity timelines, policy alerts, and configurable recording and privacy controls. Its newer insider-risk materials describe optional recording around risk events. An investigator searches for sensitive on-screen content and then reviews what happened before and after it. Proofpoint Insider Threat Management, formerly ObserveIT, documents metadata-indexed recording, policy-controlled capture, and investigation workflows. It can switch from metadata-only monitoring to screenshot recording when an alert fires. It manages screenshot data separately, with hot and warm storage tiers. It also audits the people who use the monitoring system, the watch-the-watchers principle. That matters for any platform giving investigators sensitive operational footage. The precedents map directly. Metadata-only monitoring that escalates to visual capture becomes routine telemetry that escalates to protected video when risk triggers fire. A searchable activity timeline linked to replay becomes a mission timeline linking perception, planning, intervention, and video. Application and user recording policies become camera, mission, location, and event policies. Separate visual and metadata lifecycles become separate lifecycles for raw video, tracks, embeddings, summaries, and evidence manifests. Auditing monitoring access becomes auditing who searched, viewed, exported, or unmasked evidence. The analogy has limits. Desktop interfaces expose application names, URLs, commands, and file events. Drone video must infer meaning from moving-camera imagery and imperfect sensor alignment. DVSA should borrow the workflow without assuming comparable accuracy.

The following software patterns are architecture proposals informed by the market. They are not claims that operators already use them. First, separate sensing, recording, transmission, and retention. A sensor operating is not a frame recorded. A frame recorded is not a frame transmitted off the vehicle. A frame transmitted is not a frame retained long term. A frame retained is not a frame available for authorized investigation. This avoids overstating archive size or surveillance capability, as Amazon's onboard processing and Wing's imagery policy show. A mission manifest should state what was captured, retained, discarded, and unavailable. Second, use event-oriented segments, not only video shots. AWS uses shot segmentation to preserve temporal context. Autonomy should extend it to delivery approach and clearance checks, obstacle encounters, localization uncertainty, remote-assistance requests, human interventions, aborted deliveries, and communications interruptions. The key unit is the approach and intervention episode, not a fixed clip. Third, combine structured filtering with semantic retrieval. Use exact filters for mission, time, location, vehicle, software version, and event type, and multimodal retrieval for visual concepts. An illustrative query asks for aborted delivery approaches near pedestrian crossings where a person entered the delivery area after the initial clearance check. The plan should narrow candidates by structured and geospatial constraints, retrieve visual candidates, inspect surrounding intervals, and join operational logs. An answer must separate the video showing a pedestrian, the planner logging an obstacle, and an analyst concluding the pedestrian caused the abort. Fourth, treat semantic outputs as versioned claims. Keep separate records for original sensor evidence, model observations, operator actions and statements, human-reviewed findings, and generated explanations. A VLM narrative should cite its evidence, its model and configuration, and any uncertainty or reviewer correction. Timestamped grounding already exists in commercial tools, and DVSA should extend it into explicit evidence lineage. Fifth, use a durable catalog with rebuildable indexes. The authoritative layer preserves mission and camera identities, source time ranges and synchronization, media integrity records, sensor and software provenance, retention and legal-hold decisions, and derivation relationships. Vector indexes, caches, and summaries stay replaceable, which reduces dependence on one embedding model or cloud product. Sixth, make evidence-preserving retention a first-class policy. The earlier progression from full video to key frames to embeddings to long-lived digests remains an example, not a universal policy. A safer policy distinguishes routine missions, safety events, human interventions, disputes and complaints, legally preserved evidence, and restricted research or training datasets. Deleting raw video while keeping its summary limits what investigators can later verify. A digest can show the integrity of an available object. It cannot rebuild a deleted recording. Semantic compression is a task-dependent information-loss trade-off, not a guaranteed substitute for source evidence.

Where could DVSA differentiate? This is a strategic assessment, not a measured forecast. Generic descriptions, OCR, and summaries face strong cloud and model-provider capabilities, so DVSA should integrate rather than recreate them. Natural-language clip retrieval faces strong specialist and cloud platforms, so DVSA should add mission context and evaluation. Media cataloging and sharing face established enterprise platforms, so DVSA should interoperate with existing archives. Video, telemetry, and decision alignment faces incomplete public operator disclosure, so DVSA should build autonomy-specific connectors and schemas. Cross-operator incident exchange is not established by the reviewed public materials, so DVSA should investigate standardized evidence requests and manifests. Policy propagation across video and derivatives needs application-specific design, so DVSA should make governance consistent across every evidence representation. Reproducible incident reconstruction needs more than semantic search, so DVSA should preserve sources, versions, uncertainty, and analyst actions. The defensible claim is not that competitors lack governance. Cloud, media, and insider-risk vendors already advertise substantial security and control. The narrower hypothesis is that heterogeneous autonomy evidence needs a mission-centered integration and governance layer that general-purpose video products do not automatically supply.

The recommended positioning is this. DVSA is a governed evidence platform for autonomous operations. It connects video, telemetry, perception outputs, interventions, and software lineage into searchable, policy-controlled incident records. It integrates with existing cloud, media, and fleet systems. It should be validated around one bounded workflow before expanding into a horizontal physical-AI platform. Given a disputed delivery or safety event, retrieve the evidence across participating systems, reconstruct the timeline, expose uncertainty and missing data, and produce an auditable review package. The next research step is customer and operator validation of what public documentation leaves open. What camera data actually leaves the vehicle? What remains after routine retention expires? Which organization can authorize access? Can investigators align video with planning and intervention logs? How long does an investigation take today? Can operators export a complete incident package without bespoke engineering? Which parts of the existing workflow would buyers replace, and which must stay?

Public clouds and independent platforms already make video searchable, summarize it, and return relevant moments. Enterprise security products go further by linking visual recordings to activity timelines, risk events, and controlled investigations. DVSA should build on these patterns, not present them as capabilities that do not yet exist. Autonomous delivery adds a more specific evidence problem. A camera image alone may not explain what a vehicle perceived, which route it planned, whether communications degraded, or why a human intervened. The opportunity is to connect these records without confusing original evidence, machine inference, and retrospective explanation. DVSA's next phase should be defined less by the type of camera and more by the integrity of the operational record. Its value lies in preserving the relationships among what was sensed, what was inferred, what was commanded, what occurred, and what can still be verified.


Monday, November 16, 2026

 Delivery Robot Video Forensics and the Opportunity for DVSA Governance

Do four-wheeled food delivery robots in American cities continuously capture video? And could that video become a target for governance, auditability, provenance, compliance, and forensics under a DVSA (Drone Video Security, Safety, and Auditability) initiative? Yes. Every commercially viable sidewalk robot depends on camera-driven perception. Most also add remote supervision, teleoperation, or event recording. Retention policies are rarely disclosed. Even so, substantial video is generated. Sidewalk robots are effectively ground-based drones. They share persistent cameras, edge AI, remote fleet supervision, human intervention, fleet analytics, model-improvement pipelines, and regulatory accountability. They also sit between drones and autonomous vehicles. They are unmanned yet constantly among the public. Coco Robotics operates in Los Angeles, Chicago, Miami, and Helsinki. Its robots navigate busy sidewalks and use low-latency remote operation. This implies that video is streamed to operators and becomes an operational record. Each run yields navigation, pedestrian, crossing, and delivery imagery. It also yields teleoperation video, route and localization traces and intervention records. These matter when questions arise about yielding, collisions, wheelchair access, misclassification, or human override. Serve Robotics grew out of Uber. It publicly advertises live low-latency video, monitoring, and remote control. It offers the richest opportunity. Each trip contains layers of sensor reality, machine perception, navigation reasoning, human oversight, human intervention and communications evidence such as latency, bitrate, and packet loss. Investigators can use these layers to tell algorithm failure from communications failure. The result resembles flight telemetry combined with drone video archives. Starship Technologies has made millions of deliveries. Its robots carry twelve cameras plus radar, ultrasonics, and time-of-flight sensing. It holds a vast pedestrian-height corpus covering congestion, accessibility, weather, and night operation. At that scale raw-video archives give way to event-triggered retention, semantic indexing, object trajectories, scene graphs, VLM summaries and provenance records. These support incident reconstruction, behavioral audits, algorithm audits, safety certification, and synthetic-data validation. Nuro builds road-going vehicles with cameras, lidar, radar and 360-degree perception. It is the closest to advanced drone autonomy. It calls for sensor, fusion, localization, planning, decision, human-review, and software provenance. This converges with AI observability. Smaller operators include Kiwibot, Cartken and campus, hospital, and corporate fleets. They work in semi-private settings with minors, visitors, and sensitive sites. Their concerns shift toward privacy, retention, access, chain of custody and compliance. Standardized DVSA controls would help here. Adoption may come readily. The realistic forensic hierarchy starts with raw video, which has the highest value and the highest cost. It then runs through key frames, sensor metadata, object tracks, perception outputs, and scene graphs. It continues with intervention and teleoperation logs and VLM narratives. It ends with software lineage. DVSA could unify all of these into one evidence graph. Ground robots may prove a stronger governance market than drones. They operate all day among the public. They repeat fixed routes. They produce structured navigation data. They also face growing regulations. This creates recurring risk and recurring demand for evidence and compliance. DVSA is therefore autonomy-video governance. It is a horizontal governance plane for physical AI. This covers delivery, security, warehouse, industrial, inspection and teleoperated ground robots. Each run becomes a governed and cryptographically verifiable record of video, sensors, perception, decisions, operator actions, and model lineage. Drone governance is DVSA 1.0. Delivery robots are a natural route to DVSA 2.0. Enterprise tools have made text and structured data cheap to retain. Video remains hard. Codecs shrink files but do not make them governable. Robot fleets will produce petabytes. The real asset is the meaning in the footage and not the container. This points to semantic compression and deduplication. It also points to event-aware sampling and frame pruning such as NVIDIA's Video Search and Summarization and PruneVid. Related tools include Video RAG, embedding-based vector search, scene graphs, object tracking and AI-written narratives. Retention can then age by meaning rather than time. Full video is kept on day zero. Key frames follow at one month. Embeddings and summaries follow at one year. Governance digests remain for five years. Incidents, safety events, and interventions are kept long-term. Routine runs are kept briefly. Cloud tiering from S3, Azure and Google handles storage but not semantics. The opportunity lies in a software layer above it. Video will become manageable once computers index meaning. Documents became manageable once computers indexed words. This is the trend DVSA most needs to watch.


Sunday, November 15, 2026

 Improving precision and recall in aerial drone images

Implementations of “standard” analytics pipeline on any video fail more often than not in raising the bar for precision and recall due to factors such as quality of data, embeddings, algorithms for retrieval and application of reranking, let alone narrowing the data set down to aerial drone images where the objects don’t even appear in the same façade form as in image search sites. Then, how do we improve the F-score in such analytics.

This comes down primarily to the following relation:

RetrievalQuality≈EmbeddingQuality x IndexRecall x RerankingQuality

Where embedding quality improves semantic understanding, index recall determines true neighbors survive candidate generation, and re-ranking determines final precision. For aerial drone images, the order of priority is domain-specific embeddings, multi-stage retrievals and re-ranking, vector index selection, and hardware scaling.

If the precision and recall is for retrieval quality only, nothing beats Exhaustive-KNN (Exact nearest neighbor search) for its brute force approach but most if not all production systems replace it with an ANN (Approximate nearest neighbor) search. Most researchers establish Ground Truth Recall on datasets such as VisDrone, UAVDT, DOTA, xView and SkyScenes as Exhaustive KNN and then compare ANN against it. And the most favored commercial ANN is the HNSW (Hierarchical Navigable Small World) for its best trade-off between recall, latency, simplicity and maturity with some vector store implementations claiming nearly 95-99% of Exhaustive KNN regardless for 1536 dimension vectors. Applications often tune their retreivals by keeping M between 32-64, efConstruction to be between 400-800 and efSearch to be between 200-1000. These ranges cover products such as Azure AI Search index, Vespa, Weaviate, Qdrant, Milvus and Pinecone.

But as pointed embedding models determine the disambiguity between vectors. Fine-tuned CLIP models such as GeoCLIP, RemoteCLIP, SkyCLIP, SatCLIP, and variations of OpenCLIP out perform regular plain CLIP model and work well as Tier 1 models for aerial drone images. They succeed in detecting drone world objects such as aircraft, runway, warehouses, substation, container yard, and solar farm. But taking it a notch higher is DinoV2 embeddings because they can match buildings, roofs, parking lots and other infrastructures. DinoV2 outperforms both Amazon and Azure for their S3 Vectors and AI Search respectively on similarity search.

If the retrieval is exclusively multimodal, Google SigLIP performs better than other CLIPs on the basis of its zero-shot retrieval, improved ranking, and stronger semantic descriptions. SigLIP + HNSW + Cross-encoder re-ranker significantly raises the game. Similar stages is suitable for image search purposes across domains but the model might need to change from CLIP cross attention to BLIP to Florence to Vision-Language-Models. These improve precision.

GeoVARNet style drone retrieval benefits from adding more stages to the pipeline such as an evidence encoder, higher dimension embedding, filtering the HNSW results and taking the top few even from the re-ranking. This must be combined with metadata retrieval such as GPS, Altitude, Heading, Time, Region, and object detections.

The main takeaway from this article is that focus on embedding and reranking, not so much on the substitute for HNSW. An earlier article explained retrieval escalation techniques for standardizing your pipeline. With these, switching out HNSW rarely moves the needle as much.


Saturday, November 14, 2026

 What percentage of aerial drone videos are top-down nadir videos? Most drone experts would hesitate to give that answer but when pressed they would lean towards a 40-60 split between nadir and oblique/FPV videos. In this article, we explore how accurate that might be. First, there are two di¯erent drone worlds: 1. Photogrammetry/mapping drones 2. Inspection/media/consumer drones The first of these two, benefits from a near-nadir perspective over other angles because it is helpful to create orthomosaics, surface models and measurements. Oblique coverage is helpful to façade or height estimation but not ground/roof coverage. Most commercial shot planning uses a mix, but the percentage varies on a case-by-case basis. Historically photogrammetry and mapping datasets such as OpenDroneMap/ESRI accrued nadir datasets with a dominance of top-down collections but the situation reverses with the ubiquity of hobbyist/consumer drones that generate YouTube drone footage, real-estate marketing, inspections, construction progress, emergency or rescue response, racing and cinematic content. Computer Vision has also switched to the study of these videos as evident from VisDrone/UAV tracking datasets that are forward looking. So, the market determines the ration of these videos. Photogrammetry and Mapping also benefit from still photographs rather than videos. Mapping is easier with one acquisition setup whereas oblique imagery requires using five cameras. Datasets for mapping often have varying ratios depending on use cases but they all benefit from derived orthomosaics and related products. Geospatial analytics and mapping ecosystem among these use cases, have the most skewed ratio and in favor of nadir perspective. Even when the perspectives mix in the same video between takeo¯ and landing, a credible estimate based on duration alone is elusive because of revisits. A single nadir tour of a survey area has high overlaps with those from extended rounds and revisits that usually build up a drone tour flight path. However, nadir videos help with a survey map more than any oblique videos. Any study of such a ratio amongst existing datasets is also subject to the same limitations as with analytics over a single drone video. It is inherently multimodal in nature and the search terms and retrieval frameworks no matter whether hierarchical navigable small world, exhaustive KNN, agentic or tool-based su¯er from significant variations in precision and recall with none coming close to being true to actual. The most credible reply therefore is that the answer might be unknown, but the market determines the choice.

Thursday, November 12, 2026

 PruneVid for aerial drone videos:

The previous article discussed using one of the three supported VLMs for PruneVid to improve token efficiency with the smallest model: PLLaVA. The other two are LLaVA-OneVision and ST-LLM (Liu et al., 2024b).  

ST-LLM feeds all raw, unpooled spatial-temporal tokens directly into the LLM backbone. For an egomotion camera where changes between consecutive frames are incremental, preserving the raw spatial-temporal progression allows the LLM to model subtle frame-to-frame displacements, motion cues, and panning trajectories. To prevent the massive context length from slowing down the system, it uses a dynamic masking strategy during training. This means it can drop redundant tokens where frames overlap significantly while forcing the model to retain a highly granular understanding of the continuous stream

The following implementation uses LLaVA-OneVision instead of the PLLaVA and ST-LLM from the previous article:

"""Standalone LLaVA-OneVision aerial-video runner. No PruneVid/ST-LLM imports."""

import argparse

import faulthandler

import json

import logging

import math

from pathlib import Path

import platform

import time


LOG = logging.getLogger('onevision')

DEFAULT_MODEL = 'llava-hf/llava-onevision-qwen2-0.5b-ov-hf'



def sample_indices(total, fps, start, seconds, frames):

    if total < 1 or not math.isfinite(fps) or fps <= 0:

        raise ValueError('Invalid video frame count or FPS. Transcode to constant-frame-rate MP4.')

    if not math.isfinite(start) or not math.isfinite(seconds) or start < 0 or seconds <= 0 or frames < 1:

        raise ValueError('Require start >= 0, seconds > 0 and frames >= 1.')

    first = int(start * fps)

    last = min(total, int((start + seconds) * fps)) - 1

    if last - first + 1 < frames:

        raise ValueError('Clip is out of range or contains fewer source frames than --frames.')

    return [first] if frames == 1 else [round(first + i * (last-first)/(frames-1)) for i in range(frames)]



def save_json(path, value):

    temporary = path.with_suffix(path.suffix + '.tmp')

    temporary.write_text(json.dumps(value, indent=2), encoding='utf-8')

    temporary.replace(path)



def parse_args():

    p = argparse.ArgumentParser(description=__doc__)

    p.add_argument('--video', required=True)

    p.add_argument('--model-dir', '--model-id', dest='model_dir', default=DEFAULT_MODEL,

                   help='Complete Hugging Face OneVision checkpoint directory or repository ID.')

    p.add_argument('--start', type=float, default=0)

    p.add_argument('--seconds', type=float, default=20)

    p.add_argument('--frames', type=int, default=16)

    p.add_argument('--question', default='Describe the visible terrain, structures, roads, and moving objects. Distinguish camera motion from object motion. State uncertainty; do not invent GPS coordinates or metric distances.')

    p.add_argument('--max-new-tokens', type=int, default=160)

    p.add_argument('--output', default='runs/drone_onevision')

    p.add_argument('--prepare-only', action='store_true')

    p.add_argument('--device', choices=['cuda', 'cpu'], default='cuda')

    p.add_argument('--dtype', choices=['auto', 'float16', 'bfloat16', 'float32'], default='auto')

    p.add_argument('--load-in-4bit', action='store_true')

    p.add_argument('--local-files-only', action='store_true')

    p.add_argument('--revision', default='main', help='Hub revision; use a commit hash for reproducibility.')

    a = p.parse_args()

    if a.max_new_tokens < 1 or a.frames < 1 or not a.question.strip():

        p.error('Require positive frames/max-new-tokens and a nonempty question.')

    if a.device == 'cpu' and a.load_in_4bit:

        p.error('This adapter supports bitsandbytes 4-bit loading on CUDA only.')

    return a



def prepare(a, target):

    import cv2

    from PIL import Image, ImageDraw

    source = Path(a.video).resolve()

    if not source.is_file():

        raise FileNotFoundError(source)

    cap = cv2.VideoCapture(str(source))

    images, samples = [], []

    try:

        if not cap.isOpened():

            raise RuntimeError('OpenCV cannot open the video; check codec and path.')

        fps = float(cap.get(cv2.CAP_PROP_FPS))

        total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

        indices = sample_indices(total, fps, a.start, a.seconds, a.frames)

        for i, index in enumerate(indices):

            cap.set(cv2.CAP_PROP_POS_FRAMES, index)

            ok, bgr = cap.read()

            if not ok:

                raise RuntimeError(f'Failed to decode frame {index}.')

            image = Image.fromarray(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))

            images.append(image)

            image.save(target / f'frame_{i:03d}.jpg')

            samples.append({'frame_index': index, 'nominal_seconds': index / fps})

    finally:

        cap.release()

    sheet = Image.new('RGB', (1280, math.ceil(a.frames / 4) * 205), 'white')

    draw = ImageDraw.Draw(sheet)

    for i, image in enumerate(images):

        thumb = image.copy()

        thumb.thumbnail((320, 180))

        x, y = (i % 4)*320, (i // 4)*205

        sheet.paste(thumb, (x, y))

        draw.text((x+4, y+182), f'{samples[i]["nominal_seconds"]:.3f}s', fill='black')

    sheet.save(target / 'contact_sheet.jpg')

    meta = {'arguments': vars(a), 'video': str(source), 'fps': fps,

            'source_frame_count': total, 'timestamps_are_nominal': True, 'samples': samples}

    save_json(target / 'sampling.json', meta)

    return images, meta



def infer(a, images, metadata):

    import numpy as np

    import torch

    import transformers

    from transformers import AutoConfig, AutoProcessor, LlavaOnevisionForConditionalGeneration

    device = 'cuda:0' if a.device == 'cuda' else 'cpu'

    if a.device == 'cuda':

        if not torch.cuda.is_available():

            raise RuntimeError('CUDA unavailable. Install a CUDA-enabled PyTorch wheel; no implicit CPU fallback.')

        LOG.info('GPU: %s; capability: %s; torch: %s; runtime: %s',

                 torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0), torch.__version__, torch.version.cuda)

        if torch.cuda.get_device_capability(0) == (12, 0) and 'sm_120' not in torch.cuda.get_arch_list():

            LOG.warning('sm_120 not listed in this wheel. The following real GPU computation must pass.')

        x = torch.randn(128, 128, device=device)

        _ = x @ x

        torch.cuda.synchronize()

        del x, _

    dtype_name = a.dtype if a.dtype != 'auto' else ('float32' if a.device == 'cpu' else 'float16')

    if a.device == 'cpu' and dtype_name != 'float32':

        raise ValueError('Use float32 on CPU in this adapter.')

    if dtype_name == 'bfloat16' and a.device == 'cuda' and not torch.cuda.is_bf16_supported():

        raise RuntimeError('This GPU does not report BF16 support.')

    dtype = getattr(torch, dtype_name)

    shared = {'revision': a.revision, 'local_files_only': a.local_files_only, 'trust_remote_code': False}

    config = AutoConfig.from_pretrained(a.model_dir, **shared)

    if config.model_type != 'llava_onevision':

        raise ValueError(f'Expected llava_onevision, got {config.model_type!r}; PLLaVA/ST-LLM weights cannot be reused.')

    processor = AutoProcessor.from_pretrained(a.model_dir, use_fast=False, **shared)

    processor.tokenizer.padding_side = 'left'

    options = dict(config=config, torch_dtype=dtype, low_cpu_mem_usage=True,

                   device_map={'': device}, attn_implementation='sdpa', **shared)

    if a.load_in_4bit:

        from transformers import BitsAndBytesConfig

        options['quantization_config'] = BitsAndBytesConfig(

            load_in_4bit=True, bnb_4bit_quant_type='nf4',

            bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=dtype)

    LOG.info('Loading %s onto %s; no automatic RAM/disk offload.', a.model_dir, device)

    started = time.perf_counter()

    model = LlavaOnevisionForConditionalGeneration.from_pretrained(a.model_dir, **options).eval()

    load_seconds = time.perf_counter()-started

    times = ', '.join(f'{row["nominal_seconds"]:.3f}' for row in metadata['samples'])

    question = (f'The supplied video consists of {a.frames} sampled frames, in chronological order, '

                f'at nominal source-video times [{times}] seconds. Events between samples may be unobserved.\n' + a.question)

    messages = [{'role': 'user', 'content': [{'type': 'video'}, {'type': 'text', 'text': question}]}]

    prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

    clip = np.stack([np.asarray(image) for image in images])

    inputs = processor(text=prompt, videos=[clip], padding=True, return_tensors='pt')

    inputs = inputs.to(device=device, dtype=dtype)

    input_length = inputs['input_ids'].shape[1]

    limit = getattr(config.text_config, 'max_position_embeddings', None)

    if limit and input_length + a.max_new_tokens > limit:

        raise ValueError(f'Context overflow: {input_length}+{a.max_new_tokens}>{limit}; reduce --frames.')

    video_token_count = int((inputs['input_ids'] == config.video_token_index).sum().item())

    if a.device == 'cuda':

        torch.cuda.synchronize()

        torch.cuda.reset_peak_memory_stats()

    started = time.perf_counter()

    with torch.inference_mode():

        generated = model.generate(**inputs, max_new_tokens=a.max_new_tokens,

                                   do_sample=False, num_beams=1, use_cache=True)

    if a.device == 'cuda':

        torch.cuda.synchronize()

    elapsed = time.perf_counter()-started

    answer = processor.batch_decode(generated[:, input_length:], skip_special_tokens=True,

                                    clean_up_tokenization_spaces=False)[0].strip()

    return {**metadata, 'answer': answer, 'model_type': config.model_type,

            'resolved_model_commit': getattr(config, '_commit_hash', None),

            'model_load_seconds': load_seconds, 'generation_seconds': elapsed,

            'input_tokens': input_length, 'video_placeholder_tokens': video_token_count,

            'generated_tokens': generated.shape[1]-input_length,

            'input_shapes': {k:list(v.shape) for k,v in inputs.items() if hasattr(v, 'shape')},

            'peak_allocated_gpu_gib': torch.cuda.max_memory_allocated()/2**30 if a.device == 'cuda' else None,

            'gpu': torch.cuda.get_device_name(0) if a.device == 'cuda' else None,

            'torch_version': torch.__version__, 'transformers_version': transformers.__version__,

            'python_version': platform.python_version(), 'resolved_dtype': dtype_name,

            'note': 'Standard OneVision video inference, not PruneVid pruning or GeoVARNet evidence reasoning. Timestamps are frame-index/FPS approximations.'}



def main():

    faulthandler.enable()

    logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')

    a = parse_args()

    target = Path(a.output)

    target.mkdir(parents=True, exist_ok=True)

    try:

        started = time.perf_counter()

        images, metadata = prepare(a, target)

        preparation_seconds = time.perf_counter()-started

        if a.prepare_only:

            LOG.info('Prepared %d frames, sampling.json and contact sheet; no model loaded.', len(images))

            return

        result = infer(a, images, metadata)

        result['preparation_seconds'] = preparation_seconds

        result['total_seconds'] = time.perf_counter()-started

        save_json(target / 'result.json', result)

        print(result['answer'])

    except Exception as error:

        save_json(target / 'error.json', {'error_type': type(error).__name__, 'message': str(error), 'arguments': vars(a)})

        LOG.exception('Run failed. No fallback answer was generated.')

        raise



if __name__ == '__main__':

    main()



Some caveat remain as before:

1. Libraries I used:

# Approximate environment for run_onevision_drone.py

# Use in a clean environment. GPU inference has not been validated here.

# --extra-index-url https://download.pytorch.org/whl/cu128

 torch==2.7.1+cu128

 torchvision==0.22.1+cu128

 transformers==4.51.3

 accelerate>=1.6,<2

 huggingface-hub>=0.30,<1

 numpy>=1.26,<2

 opencv-python==4.9.0.80

 pillow>=10,<12

 sentencepiece>=0.2,<0.3

 protobuf>=4.25,<6

 safetensors>=0.4.3,<1


 # Optional: uncomment for --load-in-4bit (not needed for the default 0.5B run).

 bitsandbytes>=0.47,<1


2. 

There is a checkpoint/code compatibility issue.


prunevid is not a generic VLM wrapper. It is tightly coupled to the downloaded checkpoint. 

and there are three checkpoints involved which is suspicious:

LlaVa-OneVision

llava-v1.6-vicuna-7b-hf

llava-1.5-7b-hf

The checkpoint architecture and the code architecture do not match.


1. Last known state:

Environment built successfully.

Model weights downloaded successfully.

Preprocessing works.

Model loading partially works.

Current blocker = intermittent native crash LlaVA-OneVision compatibility issue.