Deepstream provides nvinfer plugin s for encapsulating call models. Deepstream's sample code, deepstream app, implements the use of one primary inference engine for invoking multiple models, and one or more secondary inference engines, because the models are converted to TensorRT's engine files before they are used. So it's generally called engine.

Deepstream app uses four models, one main model, three secondary models, the main model to identify the car, and three secondary subordinate models to identify the color, type, and manufacturer of the car. The model file is located under /opt/nvidia/deepstream/deepstream/samples/models/and when running deepstream app, specify the config file as/opt/nvidia/deepstream/deepstream/samples/config/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt,

cd/opt/nvidia/deepstream/deepstream/samples/config/deepstream-app
deepstream-app -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt

You can run four models, source4_ 1080p_ Dec_ Infer-resnet_ Tracker_ Sgie_ Tiled_ Display_ Int8. The TXT specifies the configuration file config-file for each model and the model-engine-file for the model engine file for each model, and enables or turns off the use of the model by setting enable=1 or 0. The video played by the deepstream app is the demo video that comes with the deepstream samples. When you run on the Jetson Nano, you can see that the effect is comparable. In addition to the low-end hardware resources of Nano and the number of models invoked, it is also relevant that the three secondary models are invoked in parallel in different sub-bins, because only when all three secondary models are invoked and the inference results are written in meta data can the frame data be allowed to go down. So here you need to tactfully add pad probe code on pads that contain tee before sub-bin of the three secondary models and on pads of queue that branch from frame data in sub-bin to synchronize parallel calls of the three secondary models based on check buffer reference counts and locks:

static GstPadProbeReturn
wait_queue_buf_probe (GstPad *pad, GstPadProbeInfo *info, gpointer u_data)
{
  if (info->type & GST_PAD_PROBE_TYPE_EVENT_BOTH) {
    GstEvent *event = (GstEvent *) info->data;
    if (event->type == GST_EVENT_EOS) {
      return GST_PAD_PROBE_OK;
    }
  }

  if (info->type & GST_PAD_PROBE_TYPE_BUFFER) {
    g_mutex_lock (&g_bin_wait_lock);
    // printf("locked, buffer ref count %d\n",GST_OBJECT_REFCOUNT_VALUE (GST_BUFFER (info->data)));
   
       while (GST_OBJECT_REFCOUNT_VALUE (GST_BUFFER (info->data)) > 1
           && !g_bin_stop) {
          gint64 end_time;
          end_time = g_get_monotonic_time () + G_TIME_SPAN_SECOND / 1000;
          g_cond_wait_until (&g_bin_wait_cond, &g_bin_wait_lock, end_time);
       }
   
    g_mutex_unlock (&g_bin_wait_lock);
    // printf("unlocked, buffer ref count %d\n",GST_OBJECT_REFCOUNT_VALUE (GST_BUFFER (info->data)));
  }

  return GST_PAD_PROBE_OK;
}

/**
 * Probe function on sink pad of tee element. It is being used to
 * capture EOS event. So that wait for all secondary to finish can be stopped.
 * see ::wait_queue_buf_probe
 */
static GstPadProbeReturn
wait_queue_buf_probe1 (GstPad * pad, GstPadProbeInfo * info, gpointer u_data)
{
  if (info->type & GST_PAD_PROBE_TYPE_EVENT_BOTH) {
    GstEvent *event = (GstEvent *) info->data;
    if (event->type == GST_EVENT_EOS) {
      g_bin_stop = TRUE;
    }
  }

  return GST_PAD_PROBE_OK;
}

In fact, the code here only works for deepstream app, if it's for your own app, and downstream there's a sink plugin for a branch like sink for recording or http live, wait_above Queue_ Buf_ The code in probe () still needs to be revamped, otherwise the pipeline will get stuck during startup and it will not enter the PLAYING state at all, because *sink plugin relies on the first frame of data to change its state from READY to PLAYING, but when playing video, it usually goes to tee before sub-bin a few frames at a time. This leads to three secondary models requiring continuous inference of multiple frames to satisfy the foot (GST_OBJECT_REFCOUNT_VALUE (GST_BUFFER (info->data)== 1, thus exiting the while loop unlock to send the frame data down.