Read RTSP Camera Streams in Python with OpenCV, PyAV, and GStreamer

When building computer vision applications that process live CCTV footage, one of the first things we need to figure out is how to receive video frames from an RTSP stream.

In Python, OpenCV is probably the most familiar option. A few lines of code with cv2.VideoCapture and we can start receiving frames. But OpenCV isn't our only option.

We also have PyAV, which provides direct access to FFmpeg's decoding capabilities, and GStreamer, which lets us build more configurable video processing pipelines.

Interestingly, OpenCV itself supports both FFmpeg and GStreamer as video capture backends.

In this article, we'll capture frames from the same RTSP camera using OpenCV, PyAV, and GStreamer, look at how each approach works, and understand their differences.

Goal

Our goal is to connect to an RTSP camera, receive its video frames, and make them available in Python for processing.

For example, suppose we have a camera available at:

rtsp://192.168.1.100:554/stream_1

We want to receive frames as NumPy arrays so we can use them for object detection, tracking, or other computer vision tasks.

We'll explore four approaches:

Approach	Python interface	Backend
OpenCV + FFmpeg	OpenCV	FFmpeg
OpenCV + GStreamer	OpenCV	GStreamer
PyAV	PyAV	FFmpeg
Native GStreamer	PyGObject	GStreamer

One thing worth understanding is that we're not using four completely different video decoders.

OpenCV relies on backends such as FFmpeg and GStreamer to receive and decode video. PyAV provides Python bindings to FFmpeg libraries, while native GStreamer lets us interact directly with GStreamer's media pipeline.

For consistency, we'll convert every decoded frame into a NumPy array in BGR format.

Prerequisites

You'll need Python and an RTSP camera URL. If you have a CCTV camera, use its RTSP URL. Otherwise, follow the guide on streaming MP4 videos as virtual RTSP cameras which explains how to create one for testing.

We'll use the following URL throughout this article:

RTSP_URL = "rtsp://192.168.1.100:554/stream_1"

Replace it with your actual camera URL.

We'll use OpenCV to display video frames, including those received through PyAV and native GStreamer.

First, install GStreamer, its Python bindings, and the dependencies required to build OpenCV on Ubuntu:

sudo apt update

sudo apt install -y \
  build-essential cmake ninja-build pkg-config \
  python3-dev libcairo2-dev libgirepository-2.0-dev \
  gir1.2-gstreamer-1.0 gir1.2-gst-plugins-base-1.0 \
  libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev \
  gstreamer1.0-tools gstreamer1.0-plugins-base \
  gstreamer1.0-plugins-good gstreamer1.0-plugins-bad \
  gstreamer1.0-libav \
  libavcodec-dev libavformat-dev libavutil-dev libswscale-dev

Next, install the Python dependencies:

uv pip install av pygobject

CMAKE_ARGS="-DWITH_FFMPEG=ON -DWITH_GSTREAMER=ON" uv pip install --no-binary :all: opencv-python

The examples use Python on Ubuntu Linux and assume you're running them in a desktop environment where OpenCV can display video windows.

Reading RTSP streams with OpenCV and FFmpeg

If you've worked with computer vision in Python, you've probably used cv2.VideoCapture to read video files or webcams. The same function can connect to RTSP streams.

import cv2

RTSP_URL = "rtsp://192.168.1.100:554/stream_1"

cap = cv2.VideoCapture(RTSP_URL, cv2.CAP_FFMPEG)

if not cap.isOpened():
    raise RuntimeError("Failed to connect to RTSP stream")

try:
    while True:
        ret, frame = cap.read()

        if not ret:
            print("Failed to receive frame")
            break

        cv2.imshow("OpenCV RTSP", frame)

        if cv2.waitKey(1) & 0xFF == ord("q"):
            break
finally:
    cap.release()
    cv2.destroyAllWindows()

Run the script, and you should see the camera footage in a window. Press q to stop.

cv2.VideoCapture() opens the RTSP stream. Selecting cv2.CAP_FFMPEG tells OpenCV to use FFmpeg for the incoming video.

cap.read() reads and decodes the next video frame. It returns two values: a boolean indicating whether the operation succeeded and the decoded frame.

The frame is already a NumPy array in BGR format, ready for processing.

For example, we can check its shape:

print(frame.shape)

For a 1920 × 1080 video, we would typically get:

(1080, 1920, 3)

The three channels represent: Blue, Green, and Red.

OpenCV handles most of the streaming and decoding details, including decoder configuration and pixel format conversion.

Reading RTSP streams with OpenCV and GStreamer

OpenCV can also use GStreamer as its video capture backend.

The Python interface remains the same, but we'll provide a GStreamer pipeline instead of passing the camera URL directly.

Installing GStreamer doesn't automatically mean OpenCV can use it. The standard opencv-python package from PyPI generally doesn't include GStreamer support. We need an OpenCV build with the GStreamer backend enabled.

We can check our current installation:

import cv2

print(cv2.getBuildInformation())

Look for the Video I/O section:

FFMPEG: YES
GStreamer: YES

If GStreamer shows NO, we'll need an OpenCV build compiled with GStreamer support.

Check that your OpenCV build supports GStreamer before continuing.

Creating the pipeline

GStreamer uses pipelines made of individual elements.

For an H.264 RTSP stream, we can construct the following pipeline:

RTSP Camera
    ↓
rtspsrc
    ↓
rtph264depay
    ↓
h264parse
    ↓
avdec_h264
    ↓
videoconvert
    ↓
appsink
    ↓
OpenCV

Each element handles part of the receiving or decoding process. In Python, we pass the pipeline to OpenCV:

import cv2

RTSP_URL = "rtsp://192.168.1.100:554/stream_1"

pipeline = (
    f'rtspsrc location="{RTSP_URL}" protocols=tcp ! '
    "rtph264depay ! "
    "h264parse ! "
    "avdec_h264 ! "
    "videoconvert ! "
    "video/x-raw,format=BGR ! "
    "appsink drop=true max-buffers=1 sync=false"
)

cap = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)

if not cap.isOpened():
    raise RuntimeError("Failed to open GStreamer pipeline")

try:
    while True:
        ret, frame = cap.read()

        if not ret:
            print("Failed to receive frame")
            break

        cv2.imshow("OpenCV + GStreamer", frame)

        if cv2.waitKey(1) & 0xFF == ord("q"):
            break
finally:
    cap.release()
    cv2.destroyAllWindows()

The frame-reading code is almost identical to the previous OpenCV example. The pipeline controls how video reaches OpenCV, with each element handling a step:

Element	Purpose
rtspsrc	Connects to the camera and receives RTP packets
rtph264depay	Extracts H.264 data from RTP packets
h264parse	Parses the H.264 bitstream
avdec_h264	Decodes the H.264 video
videoconvert	Converts decoded frames to the requested pixel format
appsink	Makes the decoded frames available to OpenCV

We've also configured a few parameters.

protocols=tcp uses TCP for RTSP media transport.

max-buffers=1 limits the number of buffers queued inside appsink.

drop=true allows the sink to drop queued buffers when its queue is full.

sync=false disables clock synchronization at the sink.

These settings give us some control over buffering, although they don't guarantee a particular end-to-end latency.

The example assumes our camera uses H.264. For H.265, replace the depayloader, parser, and decoder:

pipeline = (
    f'rtspsrc location="{RTSP_URL}" protocols=tcp ! '
    "rtph265depay ! "
    "h265parse ! "
    "avdec_h265 ! "
    "videoconvert ! "
    "video/x-raw,format=BGR ! "
    "appsink drop=true max-buffers=1 sync=false"
)

The advantage of this approach is that we can customize the GStreamer pipeline without changing how our Python application reads frames.

Reading RTSP streams with PyAV

PyAV provides Python bindings for FFmpeg libraries. Unlike running FFmpeg through a subprocess, we can use FFmpeg's decoding functionality directly from Python.

Open the RTSP stream using av.open():

import av
import cv2

RTSP_URL = "rtsp://192.168.1.100:554/stream_1"

container = av.open(
    RTSP_URL,
    options={"rtsp_transport": "tcp"},
)

try:
    for video_frame in container.decode(video=0):
        frame = video_frame.to_ndarray(format="bgr24")

        cv2.imshow("PyAV RTSP", frame)

        if cv2.waitKey(1) & 0xFF == ord("q"):
            break
finally:
    container.close()
    cv2.destroyAllWindows()

The result should look like the OpenCV examples: frames from the same camera displayed in a window. PyAV uses a different interface to access those frames.

av.open() opens the RTSP stream using FFmpeg, with TCP selected as the transport.

container.decode(video=0) decodes frames from the first video stream and yields VideoFrame objects.

We convert it to a BGR NumPy array using:

frame = video_frame.to_ndarray(format="bgr24")

Now we have the same type of BGR NumPy array that OpenCV provides.

PyAV gives us more direct access to video information such as timestamps, encoded packets, codec details, and decoding behavior.

For example, we can inspect a decoded frame:

print(video_frame.width, video_frame.height)
print(video_frame.pts)
print(video_frame.time)

Here, pts is the presentation timestamp in the stream's time base, while time represents that timestamp in seconds when available.

This can be useful when we need to work with frame timing or build applications that require more control over decoding.

Both OpenCV + FFmpeg and PyAV use FFmpeg internally, so their different Python interfaces can rely on the same decoding libraries.

Reading RTSP streams with native GStreamer

We've already used GStreamer through OpenCV. But we can also use GStreamer directly from Python through PyGObject.

This gives us access to GStreamer's pipelines, elements, buffers, and events without going through OpenCV's VideoCapture.

We'll continue using OpenCV only to display the decoded frames.

Creating the pipeline

This time, we'll use uridecodebin to automatically select suitable components for receiving and decoding the video.

Our pipeline will look like this:

RTSP Camera
    ↓
uridecodebin
    ↓
videoconvert
    ↓
BGR Frames
    ↓
appsink
    ↓
NumPy

Here's how we can implement this pipeline in Python:

import gi
import cv2
import numpy as np

gi.require_version("Gst", "1.0")
gi.require_version("GstVideo", "1.0")

from gi.repository import Gst, GstVideo

RTSP_URL = "rtsp://192.168.1.100:554/stream_1"

Gst.init(None)

pipeline = Gst.parse_launch(
    f'uridecodebin uri="{RTSP_URL}" name=decoder '
    "decoder. ! queue ! videoconvert ! "
    "video/x-raw,format=BGR ! "
    "appsink name=sink sync=false max-buffers=1 drop=true"
)

appsink = pipeline.get_by_name("sink")
pipeline.set_state(Gst.State.PLAYING)

try:
    while True:
        sample = appsink.emit("try-pull-sample", 5 * Gst.SECOND)

        if sample is None:
            print("No frame received")
            break

        buffer = sample.get_buffer()
        caps = sample.get_caps()

        info = GstVideo.VideoInfo.new_from_caps(caps)

        success, map_info = buffer.map(Gst.MapFlags.READ)

        if not success:
            break

        try:
            frame = np.ndarray(
                shape=(info.height, info.width, 3),
                dtype=np.uint8,
                buffer=map_info.data,
                strides=(info.stride[0], 3, 1),
            ).copy()
        finally:
            buffer.unmap(map_info)

        cv2.imshow("Native GStreamer", frame)

        if cv2.waitKey(1) & 0xFF == ord("q"):
            break

finally:
    pipeline.set_state(Gst.State.NULL)
    cv2.destroyAllWindows()

This requires more code than the other approaches. Most of it comes from handling GStreamer's samples and buffers.

uridecodebin connects to the RTSP stream and automatically selects suitable receiving and decoding components.

appsink lets us retrieve decoded video samples inside our Python application.

GstVideo.VideoInfo provides information about the frame, including its dimensions and memory layout.

We map the decoded buffer into memory and copy its pixel data into a NumPy array.

The stride information is important because decoded frames can contain additional padding between rows.

We also use .copy() so the resulting NumPy array remains valid after we unmap the GStreamer buffer.

This example uses automatic decoding for simplicity. In a more advanced application, we could explicitly configure the receiving and decoding elements.

Comparing the approaches

All four examples produce decoded frames as NumPy arrays. Their interfaces and configuration options differ:

Approach	Complexity	Frame output	Pipeline control
OpenCV + FFmpeg	Low	NumPy array	Limited
OpenCV + GStreamer	Moderate	NumPy array	Configurable GStreamer pipeline
PyAV	Moderate	VideoFrame → NumPy	FFmpeg decoding controls
Native GStreamer	Higher	GstBuffer → NumPy	Direct GStreamer pipeline access

OpenCV + FFmpeg is a convenient starting point when we simply need to retrieve frames and process them.

OpenCV + GStreamer lets us configure a GStreamer pipeline while using OpenCV's familiar interface.

PyAV is useful when we want access to decoding details, timestamps, codec information, or encoded packets.

Native GStreamer gives us the most direct control over GStreamer elements and pipelines, although it requires more code.

The level of control alone doesn't tell us which approach is faster. Performance depends on the codec, decoding implementation, buffering configuration, hardware, and how frames are transferred into Python.

Practical considerations

Receiving RTSP frames is only part of building a CCTV application. Once we start processing those frames, a few other things become important.

Buffering: If our application processes frames slower than the camera produces them, frames may accumulate in internal buffers. This can increase the delay between the real camera feed and what our application processes.

Frame dropping: Sometimes we care more about receiving recent frames than processing every frame. Dropping older frames can help control buffering, although the strategy depends on the application.

Decoding: H.264 and H.265 video must be decoded before we can work with raw pixels. Higher resolutions and more concurrent cameras increase decoding requirements.

Stream interruptions: Network connections can fail, cameras can restart, and frames can stop arriving. A long-running CCTV application needs to detect these interruptions and reconnect when necessary.

Multiple cameras: Receiving frames from one camera is straightforward. With 10, 25, or even 100 cameras, we also need to consider memory, CPU usage, stream management, and processing throughput.

For this article, we've kept the examples focused on receiving frames.

Conclusion

For a simple computer vision application, I'd probably start with OpenCV and FFmpeg. It requires very little code and gives us frames directly as NumPy arrays.

PyAV gives us more direct access to FFmpeg's decoding capabilities, while GStreamer lets us build more configurable media pipelines, either through OpenCV or directly from Python.

The choice depends on what the application needs from the video pipeline. To compare how these approaches perform with multiple cameras, we'll separately benchmark frame capture performance, CPU usage, and memory consumption across different codecs, resolutions, and camera counts.