Seeing at 150 Feet: The Perception Pipeline

The object-detection task drops four known objects — including a person-sized mannequin — somewhere inside the search boundary, and the aircraft has to find, classify, and localize them on its own while sweeping at 150 feet. The constraints are the whole story: the compute is a Jetson Orin Nano capped at 15 watts, the camera is a 16-megapixel Basler pushing 5320×3040 frames over USB 3, and the same little box has to run neural-network inference and encode a live video stream for the safety pilot — on a chip that, we learned the hard way, has no hardware video encoder.

Our first pipeline demosaiced each frame to color inside the camera callback, which cost about 50 ms per frame right in the capture path and capped the whole system in the single digits of FPS. The fix is an old producer-consumer rule: do nothing in the capture path except move bytes. The grab thread copies the raw Bayer mosaic — one byte per pixel, a third the size of color — into a size-1 latest-frame slot; a separate consumer thread demosaics, resizes, and runs the detector at whatever pace the model allows, always on the newest frame. Every frame is stamped with the same monotonic clock the telemetry pump uses for poses, so a detection can later be paired with the aircraft's attitude at exposure time.

Inference runs as a TensorRT FP16 engine — several times faster than the PyTorch checkpoint on the same silicon — and the budget is dominated by it: about 125 ms for the large YOLO26 model versus roughly 10 ms for everything else combined, giving around 7–8 detections a second. That is still a fresh look at every meter of ground at the 8 m/s sweep, so we default to the accurate large model and keep a faster small one as a per-flight option. The GPU runs exactly one tenant, the CPU cores absorb capture, demosaic, and a software x264 encode, and that encode feeds a small RTSP server the Herelink pulls over Ethernet — so the pilot sees exactly what the model sees, with its detections drawn on top, while the drone is still airborne. That overlay stream quietly became our most valuable field-debugging tool.