[network communication -- live broadcast] Introduction to basic structure and function of FFMPEG

[1] FFMPEG function and structure in typical player framework

[2] Brief introduction of FFMPEG common function API

[2.1] register relevant function API s

• av_register_all(): register all components, 4.0 has been deprecated
• avdevice_register_all(): register the device, such as V4L2
• avformat_network_init(): initialize the network library and the library related to the network encryption protocol (such as openssl)

[2.2] encapsulation format related function API

• avformat_alloc_context(): responsible for applying for memory of an AVFormatContext structure and simple initialization
• avformat_free_context(): release everything in the AVFormatContext structure and the structure itself
• avformat_open_input(): open the input video file
• avformat_close_input(): turn off the demultiplexer. After that, you no longer need to use avformat_free_context: release
• avformat_find_stream_info(): get audio and video file information
• av_read_frame(): read audio and video packets
• avformat_seek_file(): locate the file
• av_seek_frame(): locate the file

[2.3] decoder correlation function API

• avcodec_alloc_context3(): allocate decoder context
• avcodec_find_decoder(): find the decoder according to the ID
• avcodec_find_decoder_by_name(): according to the decoder name
• avcodec_open2(): open codec
• avcodec_decode_video2(): decode a frame of video data (the new version is discarded)
• avcodec_decode_audio4(): decode one frame of audio data (the new version is discarded)
• avcodec_send_packet(): send encoded data packet
• avcodec_receive_frame(): receive decoded data
• avcodec_free_context(): release the decoder context, including avcodec_close()
• avcodec_close(): close the decoder

[3] FFMPEG component registration

[3.1]FFMPEG 3.X component registration method

Execute av_register_all registers the global decoder, encoder and other structures into their global object linked list for later search and call;

[3.2]FFMPEG 4.X component registration method

FFmpeg performs internal registration without requiring users to call API for manual registration;

For codec codec

• 1. Generate components to be registered during configure
  • ./configure:7203:print_enabled_components libavcodec/codec_list.c
  • AVCodec codec_list $CODEC_LIST
  • A codec will be generated here_ list. C file, only static const AVCodec * const codec_list [] array
• 2. In libavcodec / allcodecs C set static const avcodec * const codec_ The codecs in list [] are organized in a linked list

For demuxer/muxer

• 1.libavformat/muxer_list.c and libavformat/demuxer_list.c these two files are generated during configure
• 2. In libavformat / allformats C demoxer_ List [] and muexr_list [] is organized as a linked list

Other components are registered in a similar way

[4] Introduction to FFMPEG data structure

[4.1] common data structures of FFMPEG

• AVFormatContext encapsulates the format context structure, which is also the structure that commands the whole world, and stores the information related to the video file encapsulation format

AVFormatContext Field introduction
	iformat : Enter the name of the media AVInputFormat
	nb_streams : Enter the name of the media AVStream number
	streams : Enter the name of the media AVStream [] array
	duration : Enter the duration of the media(In microseconds)
	bit_rate : Bit rate of input media

• AVInputFormat (unpacking, demuxer); Each packaging format (FLV, MKV, MP4, AVI) corresponds to one such structure

AVInputFormat Field introduction
	name : Encapsulation format name
	extensions : Extension of encapsulation format
	id : Encapsulation format ID
	Some interface functions that encapsulate format processing, such as read_packet()

• AVOutputFormat (encapsulation, muxer); Each packaging format (FLV, MKV, MP4, AVI) corresponds to one such structure
  • AVOutputFormat indicates the format of the output file container. The AVOutputFormat structure mainly contains the following information:
    • Package name Description
    • Encoding format information (video/audio default encoding format, list of supported encoding formats)
    • Some operation functions for encapsulation (write_header, write_packet, write_tail, etc.)

Ffmpeg supports various output file formats, such as MP4, FLV, 3GP, etc; The AVOutputFormat structure saves the information of these formats and some general settings; Each package corresponds to an AVOutputFormat structure, and ffmpeg stores AVOutputFormat in a linked list

AVOutputFormat Field introduction
    const char *name; 				// Multiplexer name
    const char *long_name;			// The descriptive name of the format is easy to read
    enum AVCodecID audio_codec; 	// Default audio codec
    enum AVCodecID video_codec; 	// Default video codec
    enum AVCodecID subtitle_codec; 	// Default caption codec

// Write header information
int (*write_header)(struct AVFormatContext *); 					
// Write a packet; If avfmt is set in the flag_ ALLOW_ Flush, then pkt can be NULL;
int (*write_packet)(struct AVFormatContext *, AVPacket *pkt);	
// Write tail information
int (*write_trailer)(struct AVFormatContext *);					
// Currently only used to format pixels
int (*interleave_packet)(struct AVFormatContext *, AVPacket *out, AVPacket *in, int flush); 
// Allow messages from applications to devices
int (*control_message)(struct AVFormatContext *s, int type, void *data, size_t data_size);
// Write an uncoded AVFrame	
int (*write_uncoded_frame)(struct AVFormatContext *, int stream_index, AVFrame **frame, unsigned flags);	
// Initialization format; You can allocate data here and set any AVFormatContext or AVStream parameters that need to be set before sending data packets;
int (*init)(struct AVFormatContext *);		
// Cancel initialization format;
void (*deinit)(struct AVFormatContext *);
// Set any necessary bitstream filtering and extract any additional data required by the global header
int (*check_bitstream)(struct AVFormatContext *, const AVPacket *pkt);

• Each video (audio) stream in AVStream audio and video file corresponds to this structure

AVStream Field introduction
	index : Identify this video/Audio stream
	time_base : The time base of the stream, PTS * time_base = Real time(second)
	avg_frame_rate : The frame rate of the stream
	duration : This video/Audio stream length
	codecpar : Codec parameter properties

• AVCodecParameters saves codec parameter information

AVCodecParameters Field introduction
	codec_type : Media type, e.g AVMEDIA_TYPE_VIDEO,AVMEDIA_TYPE_AUDIO etc.
	codec_id : Codec type, such as AV_CODEC_ID_H264,AV_CODEC_ID_AAC etc.

• AVCodecContext codec context structure, which stores information related to video (audio) codec

AVCodecContext Field introduction
	codec : Codec AVCodec，Such as pointing AVCodec ff_aac_latm_decoder
	width, height : Width and height of image(Video only)
	pix_fmt : Pixel format(Video only)
	sample_rate : sampling rate(Audio only)
	channels : Channels (Audio only)
	sample_fmt : Sampling format(Audio only)

• Each video (audio) codec (such as H.264 decoder) of AVCodec corresponds to this structure

AVCodec Field introduction
	name : Codec name
	type : codec type 
	id : Codec ID
	Some codec interface functions, such as int (*decode)()

• AVPacket stores a frame of compressed encoded data

AVPacket Field introduction
	pts : presentation time stamp 
	dts : Decoding timestamp
	data : Compressed encoded data
	size : Compressed encoded data size
	pos : Offset address of data
	stream_index : Belonging to AVStream

• AVFrame stores one frame of decoded pixel (sampling) data

AVFrame Field introduction
	data : Decoded image pixel data(Audio sampling data)
	linesize : For video, it is the size of a row of pixels in the image; For audio, it is the size of the whole audio frame
	width, height : Width and height of image(Video only)
	key_frame : Is it a keyframe(Video only)
	pict_type : Frame type(Video only)，for example I，P，B
	sample_rate : Audio sampling rate(Audio only)
	nb_samples : Audio samples per channel(Audio only)
	pts : presentation time stamp 

[4.2] relationship between AVFormatContext, AVStream and AVCodecContext

You can use avformat_find_stream_info() API to find audio and video stream parameter information through avcodec_ parameters_ to_ The context () API copies the parameter information in the audio and video stream to the AVCodecContext structure;

Via av_find_best_stream API distinguishes audio stream from video stream;

[4.3] relationship between AVPacket and AVFrame

Reference thanks

This blog is a summary of the study and practice of bloggers, and has referred to the blogs of many bloggers. I would like to express my gratitude here. If there are any deficiencies, please criticize and correct them.

[1]The relationship between the most critical structures in FFMPEG