Sub-200ms Video Pipeline on Jetson Orin NX

GStreamer and NVENC Hardware Acceleration

Introduction

Video latency is invisible until it isn't. On a development bench a couple of hundred milliseconds glass-to-glass feels acceptable. In field operation it makes an operator correct against an image that is already several frames stale.

The pipeline is usually not broken. It is simply not designed for real-time field operation.

What \"Glass-to-Glass\" Really Means

Glass-to-glass is sensor exposure to pixel on the operator screen. It includes sensor readout and ISP processing, encode, network transport, receive buffer, decode and display. Most teams optimise only the network transport. The real bottleneck is usually encode latency and pipeline buffering.

Root Cause Analysis

1. Software encoding. libx264 at 1080p consumes a large share of a CPU core and adds encode latency before anything reaches the network.
2. Excessive pipeline buffering. GStreamer's default queue depth is sized for smoothness, not for live viewing, and adds latency unpredictably.
3. Pipeline stage bloat. Every element boundary is a potential scheduling delay.
4. ISP to encoder format mismatch. An NV12 ISP output feeding an I420 encoder input forces a colorspace conversion on the CPU.

The Fix

Switch to the hardware encoder:

# Before:
... ! x264enc tune=zerolatency bitrate=4000 speed-preset=ultrafast ! ...

# After:
... ! nvv4l2h264enc maxperf-enable=1 insert-sps-pps=1 iframeinterval=30 ! ...

Minimise the queue:

queue max-size-buffers=1 max-size-bytes=0 max-size-time=0 leaky=downstream

One buffer maximum. If the consumer cannot keep up, drop, never buffer.

Keep the native NV12 path into the encoder, cut the pipeline down to the elements that earn their place, and configure the RTP payloader for live use:

rtph264pay config-interval=1 aggregate-mode=zero-latency pt=96

Measurement Methodology

Wrong: software timestamps at pipeline entry and exit, encode time measured in isolation, single-run measurements.

Correct: an external time reference in front of the camera compared against the operator screen, repeated under full operational load, with P50, P95 and P99 reported together rather than an average alone.

Engineering Note

The problem was not the network. It was pipeline stages that earned nothing, and software encoding where hardware was available.

Optimization is subtraction, not addition.

Any latency figure we publish comes with the board, the resolution, the network path and the load it was measured under.