##Series articles
Analysis of Qualcomm msm-V4L2-Camera driver 1- first knowledge
Qualcomm msm-V4L2-Camera driver analysis 2-framework details
Analysis of Qualcomm msm-V4L2-Camera driver 3-session

1, Design idea of stream

Why use the design of stream?
If you are a developer, now you have a task: assuming that the camera pixel is 500W, you need to transmit 500W images per frame to users. What should you do?

• Method 1: access through [frame IO]
  Read and write are used to read each frame of data through read. The data needs to be copied between the kernel and the user. But the problem is: access speed in this way may be very slow.
  The amount of data in a frame is 500W. In general, our frame rate is at least more than 20 frames. This access speed is certainly unacceptable.

  Then what shall I do? Can I apply for a piece of memory (buf) in the kernel and map it (mmap) to the user space so that users can access the data in our kernel space like reading memory directly.

• Method 2: access through [stream IO]:

  • Memory mapped buffer (V4L2_MEMORY_MMAP): a buffer is opened in the kernel space, and the application is mapped to the user address space through the mmap() system call
  • User space buffer (V4L2_MEMORY_USERPTR): open up a buffer in user space applications, and exchange buffer pointers between users and kernel space.

Therefore, the concept of * * stream * * was born!
stream also involves VB2 of image_ buffer，vb2_buffer is provided by vb2_queue management

About vb2_buffer and vb2_queue is discussed in the next article.

2, Creation of stream

2.1 stream data structure

struct msm_stream {
	struct list_head list;//Chain header

	/* stream index per session, same
	 * as stream_id but set through s_parm
	 */
	unsigned int stream_id;//Stream id
	/* vb2 buffer handling */
	struct vb2_queue *vb2_q;//buf queue
	spinlock_t stream_lock;
	struct list_head queued_list;
};

2.2 session,stream,vb2_buffer relationship

As mentioned in the previous article, the session contains the stream queue:

struct msm_session {
    struct msm_queue_head stream_q;
}

stream is embedded with struct vb2_queue *vb2_q queue;
And VB2_ The queue also contains buf - struct VB2, which is used to save and transmit images_ buffer *bufs[VB2_MAX_FRAME];

• session management stream
• steam management vb2_queue
• vb2_queue management vb2_buffer

2.3 creation of flow

int msm_create_stream(unsigned int session_id,
	unsigned int stream_id, struct vb2_queue *q)
{
	struct msm_session *session;
	struct msm_stream  *stream;
    //According to session_id determines whether the session has been created
	session = msm_queue_find(msm_session_q, struct msm_session,
		list, __msm_queue_find_session, &session_id);
	if (!session)
		return -EINVAL;
    //Request memory
	stream = kzalloc(sizeof(*stream), GFP_KERNEL);
	if (!stream)
		return -ENOMEM;
    //Assignment stream_id
	stream->stream_id = stream_id;
    //Assignment vb2_queue queue
	stream->vb2_q = q;
    //Initialize spin lock
	spin_lock_init(&stream->stream_lock);
    //Add the newly created stream to the stream queue managed by the session
	msm_enqueue(&session->stream_q, &stream->list);
    //Queue length of stream + 1
	session->stream_q.len++;

	INIT_LIST_HEAD(&stream->queued_list);

	return 0;

There are many streams, such as preview stream, photo stream, metadata stream, etc. these streams are composed of a unified stream_q queue management, every time a stream is created,
Will join the team.

2.4 flow creation process - from hal to kernel

Detailed call process, refer to [Camera topic] HAL layer - Analysis of addChannel and startChannel
Here is the key interface

User space: HAL layer
hardware/qcom/camera/QCamera2/HAL/QCamera2HWI.cpp

int32_t QCamera2HardwareInterface::addPreviewChannel()
{
`
    CDBG_HIGH("%s :zcf camera_handle=%d ops=%p", __func__,mCameraHandle->camera_handle,mCameraHandle->ops);
    pChannel = new QCameraChannel(mCameraHandle->camera_handle,
                                  mCameraHandle->ops);
···

    // preview only channel, don't need bundle attr and cb
    rc = pChannel->init(NULL, NULL, NULL);
··· 

    // meta data stream always coexists with preview if applicable
    rc = addStreamToChannel(pChannel, CAM_STREAM_TYPE_METADATA,
                            metadata_stream_cb_routine, this);
···

    if (isNoDisplayMode()) {
        rc = addStreamToChannel(pChannel, CAM_STREAM_TYPE_PREVIEW,
                                nodisplay_preview_stream_cb_routine, this);
    } else {
        rc = addStreamToChannel(pChannel, CAM_STREAM_TYPE_PREVIEW,
                                preview_stream_cb_routine, this);
    }
···
    if(···)
         rc = addStreamToChannel(pChannel, CAM_STREAM_TYPE_ANALYSIS,
                NULL, this);

    m_channels[QCAMERA_CH_TYPE_PREVIEW] = pChannel;
    return rc;
}

When the preview is started, three data streams are added, namely:

• CAM_STREAM_TYPE_METADATA: metadata flow
• CAM_STREAM_TYPE_PREVIEW: preview flow
• CAM_STREAM_TYPE_ANALYSIS: face analysis stream

User space: Hal mm camera interface
hardware/qcom/camera/QCamera2/stack/mm-camera-interface/src/mm_camera_stream.c

Call process:
mm_channel_init
mm_camera_intf_add_channel
mm_camera_intf_add_stream
mm_channel_fsm_fn
mm_channel_fsm_fn_stopped
mm_channel_add_stream
mm_stream_fsm_fn/*Get stream */
mm_stream_fsm_inited
mm_stream_init
mm_stream_set_ext_mode

Finally, it is called to mm_stream_set_ext_mode

int32_t mm_stream_set_ext_mode(mm_stream_t * my_obj)
{

    s_parm.type =  V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
    //Call to the kernel through IOCTL.
    rc = ioctl(my_obj->fd, VIDIOC_S_PARM, &s_parm);

    return rc;
}

Call to the kernel through IOCTL.
ioctl(my_obj->fd, VIDIOC_S_PARM, &s_parm);

Kernel space

kernel/msm-4.9/drivers/media/v4l2-core/v4l2-ioctl.c

static struct v4l2_ioctl_info v4l2_ioctls[] = {
···
IOCTL_INFO_FNC(VIDIOC_S_PARM, v4l_s_parm, v4l_print_streamparm, INFO_FL_PRIO),
···
}

static int v4l_s_parm(const struct v4l2_ioctl_ops *ops,
				struct file *file, void *fh, void *arg)
{
    //Call vidioc_s_parm
	return ops->vidioc_s_parm(file, fh, p);
}

kernel/msm-4.9/drivers/media/platform/msm/camera_v2/camera/camera.c

static const struct v4l2_ioctl_ops camera_v4l2_ioctl_ops = {
···
	/* Stream type-dependent parameter ioctls */
	.vidioc_g_parm = camera_v4l2_g_parm,
	.vidioc_s_parm = camera_v4l2_s_parm,
···
};

In the previous article, I talked about ioctl_ops this operation set:
Among them vidioc_s_parm = camera_v4l2_s_parm.

static int camera_v4l2_s_parm(struct file *filep, void *fh,
	struct v4l2_streamparm *parm)
{
    //Create stream
	rc = msm_create_stream(event_data->session_id,
		event_data->stream_id, &sp->vb2_q);

	return rc;
}

3, log

Definition of data flow in HAL layer:

typedef enum {
    CAM_STREAM_TYPE_DEFAULT,       /* default stream type */
    CAM_STREAM_TYPE_PREVIEW,       /* preview */
    CAM_STREAM_TYPE_POSTVIEW,      /* postview */
    CAM_STREAM_TYPE_SNAPSHOT,      /* snapshot */
    CAM_STREAM_TYPE_VIDEO,         /* video */
    CAM_STREAM_TYPE_CALLBACK,      /* app requested callback */
    CAM_STREAM_TYPE_IMPL_DEFINED, /* opaque format: could be display, video enc, ZSL YUV */
    CAM_STREAM_TYPE_METADATA,      /* meta data */
    CAM_STREAM_TYPE_RAW,           /* raw dump from camif */
    CAM_STREAM_TYPE_OFFLINE_PROC,  /* offline process */
    CAM_STREAM_TYPE_PARM,         /* mct internal stream */
    CAM_STREAM_TYPE_ANALYSIS,     /* analysis stream */
    CAM_STREAM_TYPE_DEPTH,        /* Depth stream for depth sensor*/
    CAM_STREAM_TYPE_MAX,
} cam_stream_type_t;

• CAM_STREAM_TYPE_PREVIEW = 1
• CAM_STREAM_TYPE_METADATA = 7
• CAM_STREAM_TYPE_ANALYSIS = 11

//hal layer
D QCamera : <MCI><DBG> mm_stream_fsm_inited: 596: open dev name =/dev/video1 , fd = 78
D QCamera : <MCI><DBG> mm_stream_set_ext_mode: 1894: E, my_handle = 0x900, fd = 78, state = 1
D QCamera : <MCI><DBG> mm_stream_set_ext_mode: 1901: stream fd=78, rc=0, extended_mode=1

//Metadata flow
D QCamera : <MCI><HIGH> mm_stream_calc_offset: 4876: Stream type 7 num_planes 1
D QCamera : <MCI><HIGH> mm_stream_calc_offset: 4883: Plane 0, stride 764376, scanline 1, width 764376, height 1,length 764416

//Preview stream
D QCamera : <MCI><HIGH> mm_stream_calc_offset: 4876: Stream type 1 num_planes 2
D QCamera : <MCI><HIGH> mm_stream_calc_offset: 4883: Plane 0, stride 384, scanline 320, width 360, height 320, length 122880
D QCamera : <MCI><HIGH> mm_stream_calc_offset: 4883: Plane 1, stride 384, scanline 160, width 360, height 160, length 61440

//Face analysis stream
D QCamera : <MCI><HIGH> mm_stream_calc_offset: 4876: Stream type 11 num_planes 1
D QCamera : <MCI><HIGH> mm_stream_calc_offset: 4883: Plane 0, stride 544, scanline 480, width 540, height 480,length 261120

//kernel layer
camera_v4l2_s_parm:session_id = 1,event_data->stream_id=1,sp->stream_id=1
camera_v4l2_s_parm:session_id = 1,event_data->stream_id=2,sp->stream_id=2
camera_v4l2_s_parm:session_id = 1,event_data->stream_id=3,sp->stream_id=3

Stay Hungry，Stay Foolish!