Streaming RTSP Video into Multimodal LLMs: A ChatGPT Example

Large language models (LLMs) like ChatGPT are now multimodal , able to process text, images, audio, and video. This post shows how to connect a live RTSP video stream to a multimodal LLM, extract frames with FFmpeg, and optimize token usage for efficient analysis.

0:00 / 0:00

Understanding Multimodal LLMs

LLMs like GPT-4o (used in ChatGPT) can process text, images, audio, and video. For example, GPT-4 Vision or GPT-4o uses a visual module (based on Vision Transformers) to split images into patches (typically 14x14 pixels), generate embeddings for each patch, and project them into a shared space with text tokens. This allows the model to perform tasks such as:

Object detection (with bounding boxes)

Segmentation (pixel-level labeling)

Image captioning

Visual Question Answering (VQA)

OpenAI’s API supports sending images as base64 or URLs, with token costs scaling by resolution and tiling. Efficient preprocessing is key for video streams.

What is an RTSP Stream?

RTSP (Real-Time Streaming Protocol) is an application-layer protocol for controlling the delivery of real-time multimedia. It works over TCP (reliable) or UDP (low latency) and uses RTP/RTCP for transport and quality monitoring.

An RTSP session typically involves:

DESCRIBE: Returns an SDP (Session Description Protocol) describing media formats and ports

SETUP: Establishes transport parameters

PLAY: Starts the stream

TEARDOWN: Ends the session

Unlike HLS or DASH, RTSP focuses on interactive control, making it ideal for IP cameras and surveillance systems.

RTSP streams are continuous, accessible via URLs like rtsp://host:port/path. To integrate with LLMs, streams must be converted into discrete frames, as models don’t natively support live video input.

Extracting Frames with FFmpeg

FFmpeg is a powerful tool for multimedia processing. We use it to slice RTSP streams into individual frames (images), which are then sent to OpenAI’s API for analysis in ChatGPT.

Process:

Connect to the stream: FFmpeg connects to the RTSP URL and captures video.

Extract frames: Save frames at a set frequency (e.g., 1 frame per second) to avoid overloading the system.

Save and send: Frames are saved as JPGs and uploaded to OpenAI. The prompt can instruct the model to analyze changes between frames for tasks like motion detection or event description.

Example command:

Terminal

Loading editor...

ffmpeg -y -loglevel error -rtsp_transport tcp -i streamUrl -vf fps=1 -q:v 2 -strftime 1 path.join(streamOutputDir, '%Y-%m-%d_%H-%M-%S.jpg')

Parameters explained:

y: Overwrites existing files automatically

loglevel error: Logs only errors

rtsp_transport tcp: Uses TCP for reliable transport (UDP is an option for lower latency)

i streamUrl: Input RTSP URL

vf fps=1: Extracts 1 frame per second

q:v 2: Image quality (1 = best, 31 = worst; 2 is a good balance)

strftime 1: Adds timestamp to filenames

path.join(streamOutputDir, '%Y-%m-%d_%H-%M-%S.jpg'): Output path with timestamp

After extracting frames, you can send them to OpenAI using their library in a loop to analyze the stream in near real-time.

Optimizing Token Usage

Sending each frame individually can be costly, e.g., 10 frames might use \~9000 tokens. Combining multiple frames into a single grid (2x5 or 3x3) drastically reduces token usage (\~1256 tokens for 10 frames) while preserving temporal context, making analysis more efficient.

Conclusion

Integrating RTSP streams with multimodal LLMs like ChatGPT enables real-time monitoring, automation, and video analysis. Using FFmpeg and token optimization techniques makes the process efficient and scalable. Start with a local stream to avoid exceeding API quotas.