Friday, November 20, 2026

 When should a system stop answering from the evidence it has and go get better evidence? Degla and DVSA-API are two answers to it. Degla's onboard model gets better evidence by changing what the aircraft looks at next. DVSA-API's curated-frame, multi-agent pipeline organizes the evidence already captured so it can answer questions about it. The six workflows below are publicly available cases on what the loop does.

The first pattern is iterative evidence gathering over a long video, and VideoAgent from Stanford, published at ECCV 2024, documents it clearly. The system doesn't caption every frame and then answer. A language model acts as the central agent. It looks at a handful of evenly spaced frames, drafts an answer, and judges whether it is confident. If it isn't, it states what information is missing. A vision-language retrieval model then fetches the frames most likely to contain that information. Those frames are captioned and added to what the agent knows, and the loop repeats until the agent is confident or runs out of budget. The authors report competitive accuracy on long-video question answering while using only about eight frames per question on average. Selective sampling is a property of the reasoning loop instead of a fixed sampling rate. For DVSA-API, this could mean a VideoAgent-style loop would start with a small set of frames and pull in more only when the agent's confidence is low.

The second pattern is guided search inside a single high-resolution image, documented by V* (SEAL). The problem it targets is that a multimodal model shrinks a large image down to its input size, so small objects vanish. In V*, when the model can't answer because the needed detail isn't visible, it names the objects it needs to find. A search model then uses the model's world knowledge as a guide. For example, if it is looking for a small sign, it checks near roads and storefronts before open sky. It works through regions of the full-resolution image, and the crops it finds are kept in a visual working memory used to produce the final answer. The model asks itself where something like this would plausibly be and zooms there. This matters for aerial imagery, because the objects in bounding-box and color queries are often a few dozen pixels in a large frame. The workflow is a digital version of Degla's "take another look"

The third pattern is turning a query into a distinct program, documented by ViperGPT from Columbia. Here the language model doesn't answer the question. It writes a short Python program that calls a fixed set of vision functions, such as finding objects, checking a property of an image patch, comparing positions, or answering a simple visual question. The program then runs, and its output is the answer. A question like "how many cars are parked to the left of the building with the circular roof" becomes a few readable steps: find the building, find the cars, filter by position, count. What this offers a product is inspection. When an answer is wrong, you can see whether the plan or a particular perception step failed, and the steps can be reused across many images. For DVSA-API, this is a design for the computer-vision tool agent. It would make counting, spatial-relation, and multi-step questions auditable, rather than leaving them inside one opaque VLM call.

The fourth pattern is the continuous video-analytics agent, documented by NVIDIA's Video Search and Summarization (VSS) Blueprint. NVIDIA describes it as a reference architecture for video analytics agents that combines VLMs, LLMs and retrieval-augmented generation to deliver verified real-time alerts, visual question answering and automated reports. In plain language, the workflow has two speeds. As video comes in, it is cut into chunks, a VLM writes dense descriptions of each chunk, and those descriptions are stored in a searchable form. Later questions and summaries draw on that store instead of re-watching the video. Meanwhile, standing conditions such as "alert me if X happens" are checked as the video arrives, and a candidate alert is confirmed before it is raised. The business value is that one ingestion pass serves several products: search, Q&A, alerting, and reporting. DVSA-API already has ingestion and Q&A. What this pattern adds is standing queries: an operator registers a question once, and every new flight is checked against it, which reuses your existing evidence instead of repeating questions per flight.

The fifth pattern is fleet-wide content search to mine rare events, documented by Waymo's Content Search. Waymo describes indexing its driving logs so engineers can search what amounted to billions of observed objects from about 20 million miles of driving. It supports searching by similarity to an example as well as by text or category. The stated purpose is to find training data for rare situations quickly and turn data mining into a search problem. The workflow runs across videos rather than within one. An engineer notices a weakness, searches the whole fleet's history for similar cases, gathers them, and uses them to retrain models, so what one vehicle encounters improves the whole fleet. For a business, the archive becomes a training asset instead of a storage cost. For DVSA-API, this suggests a cross-flight query mode, such as "show me every flight where something like this frame appears." Its results would feed evaluation sets and fine-tuning, much as Degla turns expert decisions into training data.

The sixth pattern is natural-language investigation across many fixed cameras, documented by Verkada's AI-powered search. Users of its Command platform can type free-text descriptions such as "person holding a mop" or "white Tesla model Y with tinted windows." Search is no longer limited to a fixed menu of person and vehicle attributes, and results come from people and vehicle detections across the organization's cameras. Two parts of the documented workflow are worth noting beyond the search itself. First, it only searches over people and vehicle detections the system has already indexed; the guide says results won't appear if those analytics are disabled. Second, it pairs the search with guidance on how to phrase queries, stated limitations, query moderation and role-based permissions. A broad description narrows a large camera fleet to a short list of candidates for a person to review, inside a governed workflow. This is a suggestion for DVSA-API to index entities first, let language query those entities, and publish what the system can't reliably answer and who is allowed to ask

Each pattern answers the opening question differently:

• VideoAgent: gathers more frames from the same video.

• V*: looks closer at pixels already captured.

• ViperGPT: breaks the question into explicit, checkable steps.

• NVIDIA VSS: does the expensive interpretation once at ingestion and keeps watching with standing queries.

• Waymo: widens the search to the whole fleet's history.

• Verkada: narrows a fleet to candidates and hands the final judgment to a person under governance.

The thread connecting them is the one the earlier essay ended on: a good system knows when its answer isn't good enough and can state the next useful question to ask of the evidence.


No comments:

Post a Comment