Achieving Real-Time GPU Inference in URVIS

Starting point: a simple system that doesn't scale

URVIS started with a straightforward question: can a consumer GPU process 4K underwater video fast enough to be useful in real ROV operations?

The first version was intentionally simple:

• Load a frame
• Run inference (PyTorch)
• Display result

It worked, at ~8 FPS at 1080p. Fine for batch processing, useless if an ROV operator needs real-time feedback.

Identifying the real bottleneck: PCIe transfers

Profiling revealed the culprit immediately:

The frame was crossing the PCIe bus six times per inference. Every enhancement stage uploaded its input and downloaded its result.

PCIe Gen 4 practical throughput: ~25 GB/s. At that rate, 144 MB = ~5.8 ms just moving data, before any computation runs.

All measurements were taken with an RTX 5070 Ti limited to PCIe Gen 4 by the motherboard, giving ~25 GB/s practical bandwidth.

Eliminating redundant transfers

The first meaningful optimisation was structural. Instead of each pipeline stage owning its own transfer, the orchestrator uploads the frame once, passes a GPU-resident array through every stage, and downloads once at the end:

def _apply_stages_gpu(self, image, stages, ...):
    gpu_image = self.device_context.upload(image)   # one upload
    for stage in stages:
        gpu_image = stage(gpu_image, self.device_context)  # stays on GPU
    return self.device_context.download(gpu_image)  # one download

This reduced PCIe traffic: ~144 MB → ~48 MB per frame. More importantly, transfer cost became predictable (~2 ms total), independent of the number of stages in the pipeline.

Switching to TensorRT

PyTorch is flexible but carries overhead: Python dispatch, autograd bookkeeping, a runtime not designed for single-frame latency. TensorRT compiles the model into a hardware-specific execution plan, fuses operations, and eliminates most per-layer overhead.

The model is an adapted version of DNnet (Cao et al., 2025), a lightweight network designed for underwater enhancement. Converting it to a TensorRT engine at INT8 gave a model that is ~25 MB in memory (vs ~100 MB at FP32), running native integer kernels directly on the tensor cores.

Why INT8 over FP16?

FP16 is the obvious first step. INT8 is more aggressive: it requires a calibration pass to determine activation ranges and introduces small quantisation errors. The observed tradeoff:

• ~0.5–1 dB PSNR difference, barely perceptible in a live feed
• Significant throughput improvement over FP16
• Halved GPU memory bandwidth requirements relative to FP16

Reducing CUDA dispatch overhead with graphs

After TensorRT, the bottleneck shifted somewhere unexpected: CUDA API call overhead. At the ~73 FPS target, the frame budget is ~13.6 ms. TensorRT takes ~11.6 ms, while PCIe transfers account for ~2 ms combined. Within the inference pipeline, every kernel launch, every memory copy, every synchronisation point carries a small fixed cost, typically a few microseconds each. Across dozens of operations those costs add up to a meaningful fraction of the frame budget.

CUDA graphs solve this by recording the full sequence of GPU operations on the first inference call and replaying the graph as a single dispatch on every subsequent call:

# Capture once
with cuda.graph_capture():
    self._context.execute_async_v3(stream_handle)

# Replay on every frame, one launch for the whole graph
self._cuda_graph.launch(stream=self._stream)

• Launch overhead: microseconds → nanoseconds per operation
• Capture cost: ~50 ms (paid once per resolution change)
• At 73 FPS: ~4,380 replays per minute, so the capture cost is amortised within the first second

Pinned memory: small but important

GPU transfers use DMA, bypassing the CPU entirely. But DMA requires that the source memory is page-locked (pinned), meaning the OS cannot swap it out. Standard heap allocations are pageable, so the driver never DMAs from them directly. It first copies the data into an internal pinned staging buffer, adding an extra copy to every transfer.

URVIS allocates pinned host buffers for TensorRT I/O using cuda.pagelocked_empty(). The throughput gain is ~2% on its own: small, but free, and it removes a source of non-deterministic latency. Used only on the hot path to avoid exhausting pinned memory system-wide.

Knowing when not to optimise

Not every stage belongs on the GPU.

Thread affinity and CUDA contexts

CUDA contexts are thread-affine. If the context is created on the GUI thread and inference runs on a worker thread, TensorRT will fail in non-obvious ways. The fix is lazy initialisation: defer context creation to the first inference call on the worker thread:

def _ensure_initialized(self):
    if self._initialized:
        return
    cuda.init()
    self._cuda_device = cuda.Device(0)
    self._cuda_ctx = self._cuda_device.make_context()  # binds to current thread
    self._initialized = True

The worker thread owns CUDA. The GUI thread never touches it. A common production pitfall that is easy to miss when everything runs on a single thread in early development.

Safe model hot-swapping

Switching enhancement modes while the camera is running required careful locking, with the heavy work outside the lock and only the pointer swap inside:

new_stage = TensorRTModelStage(engine_path=new_path)  # load outside lock (~500 ms)

self._mutex.lock()
old_stage = self._model_stage
self._model_stage = new_stage                          # atomic swap
self._mutex.unlock()

old_stage.close()                                      # cleanup outside lock

Results

On an RTX 5070 Ti, the fully optimised ML pipeline breaks down like this:

The classical GPU pipeline at 1080p reaches 10–15 FPS, 5–8× faster than a CPU-only baseline, primarily limited by the stages that remain on CPU (CLAHE, bilateral filter).

Key takeaway

Compute is fast. Memory movement is the constraint.

INT8 quantisation, CUDA graphs, pinned memory, and single-transfer pipelines are all different answers to the same question: how much data moves, how many times, and across which bus? They are not independent tricks. They compound.

The difference between 8 FPS at 1080p and 73 FPS at 4K wasn't any single optimisation. It was eliminating inefficiencies across the entire pipeline.

Final note

Profiling is the only reliable guide. Several things I expected to be bottlenecks were not. Several things I would not have guessed, like CUDA API call overhead and OS page staging, were significant.

The bottleneck started in the data. Once eliminated, compute became the only thing left.