Improving precision and recall in aerial drone images

Implementations of “standard” analytics pipeline on any video fail more often than not in raising the bar for precision and recall due to factors such as quality of data, embeddings, algorithms for retrieval and application of reranking, let alone narrowing the data set down to aerial drone images where the objects don’t even appear in the same façade form as in image search sites. Then, how do we improve the F-score in such analytics.

This comes down primarily to the following relation:

RetrievalQuality≈EmbeddingQuality x IndexRecall x RerankingQuality

Where embedding quality improves semantic understanding, index recall determines true neighbors survive candidate generation, and re-ranking determines final precision. For aerial drone images, the order of priority is domain-specific embeddings, multi-stage retrievals and re-ranking, vector index selection, and hardware scaling.

If the precision and recall is for retrieval quality only, nothing beats Exhaustive-KNN (Exact nearest neighbor search) for its brute force approach but most if not all production systems replace it with an ANN (Approximate nearest neighbor) search. Most researchers establish Ground Truth Recall on datasets such as VisDrone, UAVDT, DOTA, xView and SkyScenes as Exhaustive KNN and then compare ANN against it. And the most favored commercial ANN is the HNSW (Hierarchical Navigable Small World) for its best trade-off between recall, latency, simplicity and maturity with some vector store implementations claiming nearly 95-99% of Exhaustive KNN regardless for 1536 dimension vectors. Applications often tune their retreivals by keeping M between 32-64, efConstruction to be between 400-800 and efSearch to be between 200-1000. These ranges cover products such as Azure AI Search index, Vespa, Weaviate, Qdrant, Milvus and Pinecone.

But as pointed embedding models determine the disambiguity between vectors. Fine-tuned CLIP models such as GeoCLIP, RemoteCLIP, SkyCLIP, SatCLIP, and variations of OpenCLIP out perform regular plain CLIP model and work well as Tier 1 models for aerial drone images. They succeed in detecting drone world objects such as aircraft, runway, warehouses, substation, container yard, and solar farm. But taking it a notch higher is DinoV2 embeddings because they can match buildings, roofs, parking lots and other infrastructures. DinoV2 outperforms both Amazon and Azure for their S3 Vectors and AI Search respectively on similarity search.

If the retrieval is exclusively multimodal, Google SigLIP performs better than other CLIPs on the basis of its zero-shot retrieval, improved ranking, and stronger semantic descriptions. SigLIP + HNSW + Cross-encoder re-ranker significantly raises the game. Similar stages is suitable for image search purposes across domains but the model might need to change from CLIP cross attention to BLIP to Florence to Vision-Language-Models. These improve precision.

GeoVARNet style drone retrieval benefits from adding more stages to the pipeline such as an evidence encoder, higher dimension embedding, filtering the HNSW results and taking the top few even from the re-ranking. This must be combined with metadata retrieval such as GPS, Altitude, Heading, Time, Region, and object detections.

The main takeaway from this article is that focus on embedding and reranking, not so much on the substitute for HNSW. An earlier article explained retrieval escalation techniques for standardizing your pipeline. With these, switching out HNSW rarely moves the needle as much.