Nvidia Deepstream ultimate details: 3 Deepstream Python RTSP video output display

This chapter will detail the official case: deepstream_test_1_rtsp_out.py for interpretation. deepstream_ test_ 1_ rtsp_ out. The main function of Py is to input RTSP format video stream. When we successfully run the Python file, we won't see the video on the screen, but the system will generate an RTSP address. When we use VLC or flash in this case to read, we will see the video marked by object recognition. There are some modules in Previous case explanation As explained in, I'll just take it here.

Favorite friends remember to collect some zanha.

1. How to run

First, let's solve a problem that we are most concerned about: how to run this py program. If you have not successfully installed deepstream6 0 and Deepstream Python 1.1.0, please check the first blog post in this series: Nvidia Deepstream ultimate details: installation of Deepstream 6.0 and Deepstream Python 1.1.0.

About how to run deep stream_ test_ 1_ rtsp_ out. py. First, we can find the README file in the official folder. If you can understand the official documents, it is strongly recommended to follow the official documents. Blogs are always inevitably missing some or outdated information.

After we installed the corresponding dependencies according to the official blog:

$ sudo apt update
$ sudo apt-get install libgstrtspserver-1.0-0 gstreamer1.0-rtsp
For gst-rtsp-server (and other GStreamer stuff) to be accessible in
Python through gi.require_version(), it needs to be built with
gobject-introspection enabled (libgstrtspserver-1.0-0 is already).
Yet, we need to install the introspection typelib package:
$ sudo apt-get install libgirepository1.0-dev
$ sudo apt-get install gobject-introspection gir1.2-gst-rtsp-server-1.0

Terminal commands are as follows:

cd /opt/nvidia/deepstream/deepstream-6.0/sources/apps/deepstream_python_apps
python3 deepstream_test1_rtsp_out.py -i /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_qHD.h264

There are two things to note here. The first is dstest1_pgie_config.txt and related paths. The official path is relative:

model-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel
proto-file=../../../../samples/models/Primary_Detector/resnet10.prototxt
model-engine-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
labelfile-path=../../../../samples/models/Primary_Detector/labels.txt
int8-calib-file=../../../../samples/models/Primary_Detector/cal_trt.bin

But if we move dstest1_pgie_config.txt. It is recommended to use an absolute path, such as:

model-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel
proto-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.prototxt
model-engine-file=/opt/nvidia/deepstream/deepstream-6.0/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
labelfile-path=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/labels.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/cal_trt.bin

In addition, in deepstream_test_1_rtsp_out.py file has a sentence sys path. append('../'). This is because you need to call some functions in the common package under the previous path.

After the code runs, let's take a look at the record of Terminal:

Creating Pipeline
Creating Source
Creating H264Parser
Creating Decoder
Creating H264 Encoder
Creating H264 rtppay
Playing file /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_qHD.h264
Adding elements to Pipeline
Linking elements in the Pipeline
 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

Opening in BLOCKING MODE
Opening in BLOCKING MODE
Starting pipeline
Opening in BLOCKING MODE
Opening in BLOCKING MODE
0:00:00.720879172 15042     0x2996b530 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1161> [UID = 1]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
ERROR: Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.0/models open error
0:00:02.190228549 15042     0x2996b530 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/models failed
0:00:02.190342602 15042     0x2996b530 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/models failed, try rebuild
0:00:02.190377035 15042     0x2996b530 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: Detected invalid timing cache, setup a local cache instead
0:00:35.550098071 15042     0x2996b530 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1947> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine successfully
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x368x640
1   OUTPUT kFLOAT conv2d_bbox     16x23x40
2   OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40

0:00:35.607265055 15042     0x2996b530 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:dstest1_pgie_config.txt sucessfully
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
Frame Number=0 Number of Objects=6 Vehicle_count=5 Person_count=1
H264: Profile = 66, Level = 0
NVMEDIA_ENC: bBlitMode is set to TRUE
Frame Number=1 Number of Objects=6 Vehicle_count=5 Person_count=1

In essence, this example is the same as the previous deepstream_test_1.py there is no difference except RTSP output. The output RTSP address is: rtsp://localhost:8554/ds-test.

2. Pipeline

The Gst pipeline consists of the following parts:

• filesrc: import of video files. It has been described in the last blog and will not be repeated here;
• h264parse: parse a video file in H264 format. It has been described in the last blog and will not be repeated here;
• nvv4l2decoder: decode. It has been described in the last blog and will not be repeated here. More detailed parameter descriptions can be found through the instruction gst-inspect-1.0 nvv4l2coder;
• nvstreammux: bring together video streams. It has been described in the last blog and will not be repeated here;
• nvinfer: Call of CNN module. It has been described in the last blog and will not be repeated here;
• nvvideoconvert: image format conversion, from NV12 to RGBA, for nvosd module. It has been described in the last blog and will not be repeated here;
• nvdsosd: functions such as drawing bounding box and text on the image. It has been described in the last blog and will not be repeated here;
• nvvideoconvert: re conversion of image format
• capsfilter: The element does not modify data as such, but can enforce limitations on the data format.
• nvv4l2h264enc/nvv4l2h265enc: encoding, ready for output to rtsp stream. More detailed parameter descriptions can be found through the instruction gst-inspect-1.0 nvv4l2h264enc;
• rtph264pay/rtph265pay:
• udpsink: output.

When we instantiate all the above plug-in modules, we add them through pipeline:

pipeline.add(source)
pipeline.add(h264parser)
pipeline.add(decoder)
pipeline.add(streammux)
pipeline.add(pgie)
pipeline.add(nvvidconv)
pipeline.add(nvosd)
pipeline.add(nvvidconv_postosd)
pipeline.add(caps)
pipeline.add(encoder)
pipeline.add(rtppay)
pipeline.add(sink)

And link:

source.link(h264parser)
h264parser.link(decoder)
sinkpad = streammux.get_request_pad("sink_0")
if not sinkpad:
    sys.stderr.write(" Unable to get the sink pad of streammux \n")

srcpad = decoder.get_static_pad("src")
if not srcpad:
    sys.stderr.write(" Unable to get source pad of decoder \n")

srcpad.link(sinkpad)
streammux.link(pgie)
pgie.link(nvvidconv)
nvvidconv.link(nvosd)
nvosd.link(nvvidconv_postosd)
nvvidconv_postosd.link(caps)
caps.link(encoder)
encoder.link(rtppay)
rtppay.link(sink)

3. nvvideoconvert and capsfilter

The nvvideoconvert plug-in module is used to convert the image format. Many are used in pipeline, such as from NV12 to RGBA for osd module, and then from RGBA to I420 for video coding.

Caps filter is generally used together with nvvideoconvert to define what type of image format to convert. For example, caps set_ property("caps", Gst.Caps.from_string("video/x-raw(memory:NVMM), format=I420")). This link There are similar explanations: caps filters are often placed after converter elements like audioconvert, audiosample, videoconvert or videoscale to force those converters to convert data to a specific output format at a certain point in a stream.

4. nvv4l2h264enc/nvv4l2h265enc

These two modules are V4l2 H.264/H. 265 video encoder. More detailed parameter descriptions can be found through the instruction gst-inspect-1.0 nvv4l2h264enc. We can also see some in the following pipeline diagram:

It should be noted that the input of nvv4l2h264enc/nvv4l2h265enc is in I420 format and the output is byte stream.

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
          stream-format: byte-stream
              alignment: au
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
                 format: { (string)I420, (string)NV12, (string)P010_10LE, (string)NV24 }
              framerate: [ 0/1, 2147483647/1 ]

Some parameters are set in the code:

encoder.set_property('bitrate', bitrate)
if is_aarch64():
    encoder.set_property('preset-level', 1)
    encoder.set_property('insert-sps-pps', 1)
    encoder.set_property('bufapi-version', 1)

Separate explanations:

- bitrate: Set bitrate for v4l2 encode. Unsigned Integer. Range: 0 - 4294967295 Default: 4000000;
- preset-level: HW preset level for encoder flags: readable, writable, Enum "GstV4L2VideoEncHwPreset" Default: 1, "UltraFastPreset"
            (0): DisablePreset    - Disable HW-Preset
            (1): UltraFastPreset  - UltraFastPreset for high perf
            (2): FastPreset       - FastPreset
            (3): MediumPreset     - MediumPreset
            (4): SlowPreset       - SlowPreset
- insert-sps-pps: Insert H.264 SPS, PPS at every IDR frame. Default: false
- bufapi-version: Set to use new buf API. Default: false

I don't particularly understand these parameters. I have familiar friends who can add rumors. I consulted some data about the bufapi version parameter. It seems that it must be set to True in the Deepstream scenario.

5. rtppay and udpsink

rtppay is used to make the payload encode video into RTP packets. I don't understand this parameter very well. Interested students can refer to it This link.

About the explanation of RTP and UDP, I think This website Not much. If you are interested in the concepts, you can learn them. Generally speaking, RTP is between the transport layer and the application layer, and UDP is used as the transport protocol by default. UDP - user datagram protocol is a simple datagram oriented transport layer protocol. UDP does not provide reliability. It just sends the datagrams sent by the application to the IP layer, but it does not guarantee that they can reach the destination. Because UDP does not need to establish a connection between the client and the server before transmitting datagrams, and there is no timeout retransmission mechanism, the transmission speed is very fast. In the network architecture of TCP/IP protocol suite, RTP is in the application layer, UDP is in the transmission layer, IP is in the network layer, and the encapsulation and transmission of data come down from one layer to another. Therefore, this is why in this pipeline, the system first converts the data into I420 format through nvvideoconvert and capsfilter, then converts it into byte stream through nvv4l2h264enc/nvv4l2h265enc, then starts encapsulation, goes through rtppay to udpsink, and finally generates an rtsp address.

Relevant parts of the code:

if codec == "H264":
    rtppay = Gst.ElementFactory.make("rtph264pay", "rtppay")
    print("Creating H264 rtppay")
elif codec == "H265":
    rtppay = Gst.ElementFactory.make("rtph265pay", "rtppay")
    print("Creating H265 rtppay")
if not rtppay:
    sys.stderr.write(" Unable to create rtppay")

Finally, the UDP sink part. Parts covered in the code:

# Make the UDP sink
updsink_port_num = 5400
sink = Gst.ElementFactory.make("udpsink", "udpsink")
if not sink:
    sys.stderr.write(" Unable to create udpsink")

sink.set_property('host', '224.224.255.255')
sink.set_property('port', updsink_port_num)
sink.set_property('async', False)
sink.set_property('sync', 1)

For the detailed description of udpsink, we enter gst-inspect-1.0 udpsink in terminal.

The most confusing parameters are actually sync and async. Quoted here This website Interpretation of:

Gstreamer sets a timestamp for when a frame should be played, if sync=true it will block the pipeline and only play the frame after that time. This is useful for playing from a video file, or other non-live source. If you play a video file with sync=false it would play back as fast as it can be read and processed. Note that for a live source this doesn't matter, because you are only getting frames in at the capture rate of the camera anyway.

Use sync=true if:
There is a human watching the output, e.g. movie playback
Use sync=false if:
You are using a live source
The pipeline is being post-processed, e.g. neural net
As to your other question, async=false tells the pipeline not to wait for a state change before continuing. Seems mostly useful for debugging.

Therefore, if we have high requirements for real-time performance, these two parameters should be set to False.

6. Read the video stream on flash

Finally, we read the video stream on flash. The code is very simple. I wrote it in another blog.

The results are as follows: