From Detection to Coordinates: Building Our Position Estimation Pipeline

By this point in the build we already had real-time object detection running on the aircraft — a TensorRT YOLO model on the Jetson Orin Nano turning camera frames into bounding boxes. But a bounding box only answers where the target appears in the image. It is a pixel. For the drone to actually do anything with a target — fly to it, hold over it, report it — it needs the answer to a different question: where is that target on the ground? Closing that gap, from pixel to a real-world latitude and longitude, is what our Position Estimation pipeline does.

The pipeline has four moving parts. Every camera frame is matched against a PoseBuffer — a short history of timestamped aircraft poses — so we know exactly where the camera was and how it was pointed at the instant the frame was captured. pixel_to_ground then casts the detection's centre pixel as a ray through the camera model and intersects it with the ground plane, producing one geolocated sample. As the drone sweeps its search lane past the object, those samples accumulate, and the estimator fuses them into a single coordinate at CONFIRM. Finally the aircraft flies over that coordinate and enters a closed-loop hover, refining the estimate while it holds station.

We did not want to trust the geometry on paper alone, so we drove the real pipeline with simulated flights over a known object — 40 randomized trials per scenario, plus an animated replay of one representative trial each. Four scenarios cover the failure modes we actually worried about: a clean baseline with only pixel noise, a wrong field-of-view calibration, a physical camera-mount tilt, and a realistic combination of clock lag, FOV error and tilt together. The replay below is that recorded data — switch scenarios and watch the estimate settle (or not).

Three findings came out of it. First, the projection itself is exact: a 12-combination yaw/offset round-trip unit test recovers the object to under a millimetre, and with clean inputs the live estimate lands within a centimetre. Second, field-of-view (calibration) error is erased by the hover. A 20%-wrong FOV badly biases the sweep estimate, but once the drone is hovering with the object at image centre the intrinsics barely matter, and the confirmed error collapses from 0.42 m to 0.10 m. Third, and least forgiving: camera tilt is the error that survives.

A tilted camera biases every sample in the same direction, so no amount of averaging removes it. Empirically the final error sits exactly on error = altitude · tan(tilt) — about 0.5 m per degree at 30 m altitude, reaching 6.4 m at 12° off nadir. That is the whole argument for a blunt operational rule: lock the gimbal at 90° straight down before every flight.

Beyond the four scenarios, a SITL end-to-end rehearsal — confirm, fly over, hold, return to launch — put arrival within 2.1–2.2 m of the target, consistent with our full-mission integration result. Two items stay honestly open: clock-skew sensitivity is under-tested (the harness samples synchronously; theory says ~0.5 m per 100 ms of lag at 5 m/s, and an asynchronous harness is proposed), and real-flight ground truth — CONFIRM against a GPS placed on the object — is scheduled for the next flight.