Recently, in the study of ffmpeg and ffplay, it is found that the source code analysis of ffmpeg unpacking on the Internet is not much and complete. Based on the ffplay source code, let's explain the ffmpeg decoding process. Here we mainly explain the key variables in the data structures of AVFormatContext, AVStream and AVIOContext, such as void*priv_data, AVStreamInternal *internal, AVFormatInternal *internal and other internal variables are the most important, because it is these internal non-public variables that are the key to storing file content, important interfaces or attributes and transferring between interfaces. To understand the true meaning of ffmpeg library to define these internal variables is the key to learning ffmpeg.
However, there are too many interfaces to write in one blog post, so students need to pay attention to other blog posts
All interfaces and data structures are written in great detail, but they have been studied for a long time. It's very troublesome and tiring to write. After reading it, give me some attention, ha ha ha ha
Key Tips:
-
Many structures in ffmpeg (AVStream, URLContext, AVFormatContext) like to use void *priv_data or void *opaque variable
In fact, this variable is used to store the "substructure" of the structure. I prefer to call it external protocol or external resource (interface or data). For example, in ffplay, AVFormatContext needs to be unpacked. By detecting the file format, it is concluded that AVInputFormat is mov(MP4) format, and AVFormatContext - > priv_data = movcontext variable (mov.c file), priv in AVStream_ Data is used to store mov protocol movstramcontext structure variables, priv in URLContext_ Data is used to store the FileContext structure in the file protocol. In fact, this is to separate the protocol interface function or data from the backbone interface, so as to make the whole library scalable. So you will find that the priv of the backbone will be mentioned at the beginning of various protocol interfaces_ Data is assigned to the structure of the protocol itself. For example: mov_ read_ In header
MOVContext *mov = s->priv_data; Writing in this way is also a kind of grammatical sugar. sc will not be affected by priv_data name.
Even if the name of external variables changes, it will have little impact on the internal interface. ffmpeg interfaces mostly use this method, especially involving some external protocols
rtmp streaming media, file file, mov format, etc. -
For the naming of Context, such as URLContext, FileContext, AVFormatContext, etc., my personal understanding is the data + method (Interface) required to complete the function. For example, there is a file protocol in URLContext. In the FileContext structure, there are open, close, read and other methods and uint*data variables to store the data read from the file. This is stored level by level. In order to make the code more extensible, this kind of library is written by many people. I don't know if I can explain clearly, ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha.
-
For internal, such naming as AVStreamInternal is generally used to store data and pass it to the interface. For example, AVStreamInternal is used to store the information (location, size, pts, dts) and codec data interface of audio and video samples, and AVFormatInternal is used to store (obtained through AVStreamInternal index sample) AVPacket content. It is mainly used to store and transfer data.
Here I would like to focus on my personal understanding of AVFormatContext:
AVFormatContext structure is the structure of ffmpeg docking and unpacking, that is, directly reading (whether local file file or network streaming media) files or resources, and according to certain structure requirements (AVPacket is used to store a frame or a sample information)
To organize and store the data read out from the multimedia file. Decode and store the information in the form of AVPacket structure. Therefore, in AVFormatContext, since it is necessary to unpack, it is necessary to have the file IO interface AVIOContext and the audio and video metadata information generated when reading the multimedia file head information in AVStream. It must be noted that both AVIOContext and AVStream are lower layer protocols for docking, such as AVIOContext - > priv_ data=FileContext(file.c)
URLProtocol ff_file_protocol stores the interface for reading files, and FileContext can store file handles, avstream - > internal - > index_ Entries is used to store everyone's sample information. However, after reading a sample (such as mov(mp4)), find the information of each sample by reading the head and store it in avstream - > internal - > index_ Entries), which need to be stored in avformatinternal - > avpacketlist * packet in the form of AVPacket_ buffer_ End, and then decode.
Therefore, from this point of view, the upper structure of AVFormatContext is not directly connected with the lower protocols (FileContext, MOVContext, MOVStreamContext), which I mentioned in the above Tips and previous blog posts. The ffmpeg library needs to be written by many people and must be extensible. Changes in the lower layer will not affect the upper structure.
In fact, the naming of ffmpeg is also very reasonable. AVFormatContext is simple to read files and put them in the linked list avformatinternal - > avpacketlist * packet_ buffer_ End, wait for decoding. AVStream and AVIOContext serve this purpose.
Key interfaces ha ha ha
Small Tips: avformat_find_stream_info jumps out of if (ST - > internal - > Info - > frame_delay_evidence & & count < 2 & & ST - > internal - > avctx - > has_b_frames = = 0) when reading packets circularly. When a pts= dts, that is, when the sample has both pts and dts, the frame_ delay_ The instance is assigned a value of 1. It's too important here. I always couldn't find out where to jump out of the loop because I ignored it. In fact, to read a packet, you only need to read one packet. Here, you must pay attention to yayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayaya
//Read some packet s, assign the decoder interface, and assign some parameters from AVStream to AVCodecContext*Internal for subsequent decoding
int avformat_find_stream_info(AVFormatContext *ic, AVDictionary **options)
{
int i, count = 0, ret = 0, j;
int64_t read_size;
AVStream *st;
AVCodecContext *avctx;
AVPacket pkt1;
int64_t old_offset = avio_tell(ic->pb); //After reading the file head, the current pointer points to the offset from the file start position
// new streams might appear, no options for those
int orig_nb_streams = ic->nb_streams; //Read from the head information, usually 2 stream s
int flush_codecs;
int64_t max_analyze_duration = ic->max_analyze_duration;
int64_t max_stream_analyze_duration;
int64_t max_subtitle_analyze_duration;
int64_t probesize = ic->probesize;
int eof_reached = 0;
int *missing_streams = av_opt_ptr(ic->iformat->priv_class, ic->priv_data, "missing_streams");
flush_codecs = probesize > 0;
av_opt_set(ic, "skip_clear", "1", AV_OPT_SEARCH_CHILDREN);
max_stream_analyze_duration = max_analyze_duration;
max_subtitle_analyze_duration = max_analyze_duration;
if (!max_analyze_duration) {
max_stream_analyze_duration =
max_analyze_duration = 5*AV_TIME_BASE;
max_subtitle_analyze_duration = 30*AV_TIME_BASE;
if (!strcmp(ic->iformat->name, "flv"))
max_stream_analyze_duration = 90*AV_TIME_BASE;
if (!strcmp(ic->iformat->name, "mpeg") || !strcmp(ic->iformat->name, "mpegts"))
max_stream_analyze_duration = 7*AV_TIME_BASE;
}
//Assign internal - > avctx in AVStream, including the parameter time_base, which is used to facilitate code writing
for (i = 0; i < ic->nb_streams; i++) {
const AVCodec *codec;
AVDictionary *thread_opt = NULL;
st = ic->streams[i];
avctx = st->internal->avctx;
if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO ||
st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) {
/* if (!st->time_base.num)
st->time_base = */
if (!avctx->time_base.num)
avctx->time_base = st->time_base;
}
...
// only for the split stuff
//Assign the corresponding parser according to the decoder id in AVStream*st
if (!st->parser && !(ic->flags & AVFMT_FLAG_NOPARSE) && st->internal->request_probe <= 0) {
st->parser = av_parser_init(st->codecpar->codec_id);
if (st->parser) {
if (st->need_parsing == AVSTREAM_PARSE_HEADERS) {
st->parser->flags |= PARSER_FLAG_COMPLETE_FRAMES;
} else if (st->need_parsing == AVSTREAM_PARSE_FULL_RAW) {
st->parser->flags |= PARSER_FLAG_USE_CODEC_TS;
}
} else if (st->need_parsing) {
av_log(ic, AV_LOG_VERBOSE, "parser not found for codec "
"%s, packets or times may be invalid.\n",
avcodec_get_name(st->codecpar->codec_id));
}
}
if (st->codecpar->codec_id != st->internal->orig_codec_id)
st->internal->orig_codec_id = st->codecpar->codec_id;
//Assign some parameter values in AVStream*st to AVCodecContext *avctx
ret = avcodec_parameters_to_context(avctx, st->codecpar);
if (ret < 0)
goto find_stream_info_err;
if (st->internal->request_probe <= 0)
st->internal->avctx_inited = 1;
//Assignment decoder interface
codec = find_probe_decoder(ic, st, st->codecpar->codec_id);
/* Force thread count to 1 since the H.264 decoder will not extract
* SPS and PPS to extradata during multi-threaded decoding. */
av_dict_set(options ? &options[i] : &thread_opt, "threads", "1", 0);
if (ic->codec_whitelist)
av_dict_set(options ? &options[i] : &thread_opt, "codec_whitelist", ic->codec_whitelist, 0);
...
for (i = 0; i < ic->nb_streams; i++) {
#if FF_API_R_FRAME_RATE
ic->streams[i]->internal->info->last_dts = AV_NOPTS_VALUE;
#endif
ic->streams[i]->internal->info->fps_first_dts = AV_NOPTS_VALUE;
ic->streams[i]->internal->info->fps_last_dts = AV_NOPTS_VALUE;
}
read_size = 0;//Read file size
//Small Tips: for (;) Because in the compilation phase, the assembler will put for (;) Optimize it and enter the cycle directly
for (;;) {
const AVPacket *pkt;
int analyzed_all_streams;
if (ff_check_interrupt(&ic->interrupt_callback)) {
ret = AVERROR_EXIT;
av_log(ic, AV_LOG_DEBUG, "interrupted\n");
break;
}
...
Be a pts!=dts，in other words sample Existing pts Again dts Time frame_delay_evidence Is assigned a value of 1.
It's too important here. I always couldn't find out where to jump out of the loop because I ignored it. Actually read packet,Read only one packet All right,
You must pay attention to yayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayayaya
//Jump out of read packet loop
if (st->internal->info->frame_delay_evidence && count < 2 && st->internal->avctx->has_b_frames == 0)
//Jump out of read packet loop
break;
if (!st->internal->avctx->extradata &&
(!st->internal->extract_extradata.inited ||
st->internal->extract_extradata.bsf) &&
extract_extradata_check(st))
break;
if (st->first_dts == AV_NOPTS_VALUE &&
!(ic->iformat->flags & AVFMT_NOTIMESTAMPS) &&
st->codec_info_nb_frames < ((st->disposition & AV_DISPOSITION_ATTACHED_PIC) ? 1 : ic->max_ts_probe) &&
(st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO ||
st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO))
break;
}
analyzed_all_streams = 0;
...
...
/* NOTE: A new stream can be added there if no header in file
* (AVFMTCTX_NOHEADER). */
//Read a sample from the file and assign it to avpacket - > pkt1
ret = read_frame_internal(ic, &pkt1);
if (ret == AVERROR(EAGAIN))
continue;
if (ret < 0) {
/* EOF or error*/
eof_reached = 1;
break;
}
//Assign pkt1 to avformatcontext * internal - > packet_buffer, which is read from the packet every time_ Read from buffer
//When packet_buffer_ When end is NULL, add packet to packet_ In buffer
if (!(ic->flags & AVFMT_FLAG_NOBUFFER)) {
ret = avpriv_packet_list_put(&ic->internal->packet_buffer,
&ic->internal->packet_buffer_end,
&pkt1, NULL, 0);
if (ret < 0)
goto unref_then_goto_end;
pkt = &ic->internal->packet_buffer_end->pkt;
} else {
pkt = &pkt1;
}
st = ic->streams[pkt->stream_index];
if (!(st->disposition & AV_DISPOSITION_ATTACHED_PIC))
read_size += pkt->size;
avctx = st->internal->avctx;
...
if (pkt->dts != AV_NOPTS_VALUE && st->codec_info_nb_frames > 1) {
/* check for non-increasing dts */
if (st->internal->info->fps_last_dts != AV_NOPTS_VALUE &&
st->internal->info->fps_last_dts >= pkt->dts) {
...
st->internal->info->fps_first_dts =
st->internal->info->fps_last_dts = AV_NOPTS_VALUE;
}
/* Check for a discontinuity in dts. If the difference in dts
* is more than 1000 times the average packet duration in the
* sequence, we treat it as a discontinuity. */
if (st->internal->info->fps_last_dts != AV_NOPTS_VALUE &&
st->internal->info->fps_last_dts_idx > st->internal->info->fps_first_dts_idx &&
(pkt->dts - (uint64_t)st->internal->info->fps_last_dts) / 1000 >
(st->internal->info->fps_last_dts - (uint64_t)st->internal->info->fps_first_dts) /
(st->internal->info->fps_last_dts_idx - st->internal->info->fps_first_dts_idx)) {
...
st->internal->info->fps_first_dts =
st->internal->info->fps_last_dts = AV_NOPTS_VALUE;
}
/* update stored dts values */
if (st->internal->info->fps_first_dts == AV_NOPTS_VALUE) {
st->internal->info->fps_first_dts = pkt->dts;
st->internal->info->fps_first_dts_idx = st->codec_info_nb_frames;
}
st->internal->info->fps_last_dts = pkt->dts;
st->internal->info->fps_last_dts_idx = st->codec_info_nb_frames;
}
if (st->codec_info_nb_frames>1) {
int64_t t = 0;
int64_t limit;
if (st->time_base.den > 0)
t = av_rescale_q(st->internal->info->codec_info_duration, st->time_base, AV_TIME_BASE_Q);
if (st->avg_frame_rate.num > 0)
t = FFMAX(t, av_rescale_q(st->codec_info_nb_frames, av_inv_q(st->avg_frame_rate), AV_TIME_BASE_Q));
if ( t == 0
&& st->codec_info_nb_frames>30
&& st->internal->info->fps_first_dts != AV_NOPTS_VALUE
&& st->internal->info->fps_last_dts != AV_NOPTS_VALUE)
t = FFMAX(t, av_rescale_q(st->internal->info->fps_last_dts - st->internal->info->fps_first_dts, st->time_base, AV_TIME_BASE_Q));
if (analyzed_all_streams) limit = max_analyze_duration;
else if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) limit = max_subtitle_analyze_duration;
else limit = max_stream_analyze_duration;
if (t >= limit) {
av_log(ic, AV_LOG_VERBOSE, "max_analyze_duration %"PRId64" reached at %"PRId64" microseconds st:%d\n",
limit,
t, pkt->stream_index);
if (ic->flags & AVFMT_FLAG_NOBUFFER)
av_packet_unref(&pkt1);
break;
}
if (pkt->duration) {
if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE && pkt->pts != AV_NOPTS_VALUE && st->start_time != AV_NOPTS_VALUE && pkt->pts >= st->start_time) {
st->internal->info->codec_info_duration = FFMIN(pkt->pts - st->start_time, st->internal->info->codec_info_duration + pkt->duration);
} else
st->internal->info->codec_info_duration += pkt->duration;
st->internal->info->codec_info_duration_fields += st->parser && st->need_parsing && avctx->ticks_per_frame ==2 ? st->parser->repeat_pict + 1 : 2;
}
}
if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) {
#if FF_API_R_FRAME_RATE
ff_rfps_add_frame(ic, st, pkt->dts);
#endif
if (pkt->dts != pkt->pts && pkt->dts != AV_NOPTS_VALUE && pkt->pts != AV_NOPTS_VALUE)
This piece is too important. I haven't noticed it for a long time. The assignment here is to
Used to jump out of the loop. Normally, only one is read sample Just jump out
st->internal->info->frame_delay_evidence = 1;
}
if (!st->internal->avctx->extradata) {
ret = extract_extradata(st, pkt);
if (ret < 0)
goto unref_then_goto_end;
}
/* If still no information, we try to open the codec and to
* decompress the frame. We try to avoid that in most cases as
* it takes longer and uses more memory. For MPEG-4, we need to
* decompress for QuickTime.
*
* If AV_CODEC_CAP_CHANNEL_CONF is set this will force decoding of at
* least one frame of codec data, this makes sure the codec initializes
* the channel configuration and does not only trust the values from
* the container. */
//Here is just decoding the packet. In fact, decoding here feels useless. Ha ha
//Because the decoding of the first few sample s failed
try_decode_frame(ic, st, pkt,
(options && i < orig_nb_streams) ? &options[i] : NULL);
if (ic->flags & AVFMT_FLAG_NOBUFFER)
av_packet_unref(&pkt1);
st->codec_info_nb_frames++; //Record how many packet s to read
count++;
}
....
Actually, here, avformat_find_stream_info The main function of is to read some packet,Assignment decoder interface,
from AVStream Assign some parameters to AVCodecContext*Internal,For the next decoding work, the following code,
Is in head In case of incompleteness, calculate the frame rate and assign some parameters. The most important thing is the above code,
in especial st->internal->info->frame_delay_evidence = 1 After assignment
if (st->internal->info->frame_delay_evidence && count < 2 &&
st->internal->avctx->has_b_frames == 0)Jump out, it's too critical. I've ignored it for 2 weeks. You must pay attention,
There are too many variables, you still have to watch them several times, hahaha
//Calculate a frame rate
ff_rfps_calculate(ic);
//This is mainly based on the fact that there is no head or the frame rate information is not read in the head, and the estimated frame rate needs to be calculated
for (i = 0; i < ic->nb_streams; i++) {
st = ic->streams[i];
avctx = st->internal->avctx;
if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
if (avctx->codec_id == AV_CODEC_ID_RAWVIDEO && !avctx->codec_tag && !avctx->bits_per_coded_sample) {
uint32_t tag= avcodec_pix_fmt_to_codec_tag(avctx->pix_fmt);
if (avpriv_find_pix_fmt(avpriv_get_raw_pix_fmt_tags(), tag) == avctx->pix_fmt)
avctx->codec_tag= tag;
}
/* estimate average framerate if not set by demuxer */
if (st->internal->info->codec_info_duration_fields &&
!st->avg_frame_rate.num &&
st->internal->info->codec_info_duration) {
...
}
} else if (avctx->codec_type == AVMEDIA_TYPE_AUDIO) {
if (!avctx->bits_per_coded_sample)
avctx->bits_per_coded_sample =
av_get_bits_per_sample(avctx->codec_id);
...
}
}
}
if (probesize)
estimate_timings(ic, old_offset);
av_opt_set(ic, "skip_clear", "0", AV_OPT_SEARCH_CHILDREN);
if (ret >= 0 && ic->nb_streams)
/* We could not have all the codec parameters before EOF. */
ret = -1;
for (i = 0; i < ic->nb_streams; i++) {
const char *errmsg;
st = ic->streams[i];
...
}
compute_chapters_end(ic);
//Assign some parameters from AVStream
/* update the stream parameters from the internal codec contexts */
for (i = 0; i < ic->nb_streams; i++) {
st = ic->streams[i];
if (st->internal->avctx_inited) {
int orig_w = st->codecpar->width;
int orig_h = st->codecpar->height;
ret = avcodec_parameters_from_context(st->codecpar, st->internal->avctx);
if (ret < 0)
goto find_stream_info_err;
ret = add_coded_side_data(st, st->internal->avctx);
if (ret < 0)
goto find_stream_info_err;
#if FF_API_LOWRES
// The decoder might reduce the video size by the lowres factor.
if (st->internal->avctx->lowres && orig_w) {
st->codecpar->width = orig_w;
st->codecpar->height = orig_h;
}
#endif
}
....
st->internal->avctx_inited = 0;
}
find_stream_info_err:
for (i = 0; i < ic->nb_streams; i++) {
st = ic->streams[i];
if (st->internal->info)
av_freep(&st->internal->info->duration_error);
avcodec_close(ic->streams[i]->internal->avctx);
av_freep(&ic->streams[i]->internal->info);
av_bsf_free(&ic->streams[i]->internal->extract_extradata.bsf);
av_packet_free(&ic->streams[i]->internal->extract_extradata.pkt);
}
if (ic->pb)
av_log(ic, AV_LOG_DEBUG, "After avformat_find_stream_info() pos: %"PRId64" bytes read:%"PRId64" seeks:%d frames:%d\n",
avio_tell(ic->pb), ic->pb->bytes_read, ic->pb->seek_count, count);
return ret;
unref_then_goto_end:
av_packet_unref(&pkt1);
goto find_stream_info_err;
}
Read a packet interface
static int read_frame_internal(AVFormatContext *s, AVPacket *pkt)
{
int ret, i, got_packet = 0;//The obtained packet ID in order to jump out of the loop
AVDictionary *metadata = NULL;
//got_packet is used to jump out of the loop
while (!got_packet && !s->internal->parse_queue) {
AVStream *st;
/* read next packet */
//Only one packet is read at a time
ret = ff_read_packet(s, pkt);
..
ret = 0;
st = s->streams[pkt->stream_index];
st->event_flags |= AVSTREAM_EVENT_FLAG_NEW_PACKETS;
/* update context if required */
//Assign some parameters to AVCodecContext*Internal
if (st->internal->need_context_update) {
if (avcodec_is_open(st->internal->avctx)) {
av_log(s, AV_LOG_DEBUG, "Demuxer context update while decoder is open, closing and trying to re-open\n");
avcodec_close(st->internal->avctx);
st->internal->info->found_decoder = 0;
}
/* close parser, because it depends on the codec */
if (st->parser && st->internal->avctx->codec_id != st->codecpar->codec_id) {
av_parser_close(st->parser);
st->parser = NULL;
}
ret = avcodec_parameters_to_context(st->internal->avctx, st->codecpar);
if (ret < 0) {
av_packet_unref(pkt);
return ret;
}
......
st->internal->need_context_update = 0;
}
if (!st->need_parsing || !st->parser) {
/* no parsing needed: we just output the packet as is */
//Correct some packet time parameters
compute_pkt_fields(s, st, NULL, pkt, AV_NOPTS_VALUE, AV_NOPTS_VALUE);
if ((s->iformat->flags & AVFMT_GENERIC_INDEX) &&
(pkt->flags & AV_PKT_FLAG_KEY) && pkt->dts != AV_NOPTS_VALUE) {
ff_reduce_index(s, st->index);
av_add_index_entry(st, pkt->pos, pkt->dts,
0, 0, AVINDEX_KEYFRAME);
}
got_packet = 1; //Get the packet ID and assign it to 1 in order to jump out of the loop
} else if (st->discard < AVDISCARD_ALL) {
if ((ret = parse_packet(s, pkt, pkt->stream_index, 0)) < 0)
return ret;
st->codecpar->sample_rate = st->internal->avctx->sample_rate;
st->codecpar->bit_rate = st->internal->avctx->bit_rate;
st->codecpar->channels = st->internal->avctx->channels;
st->codecpar->channel_layout = st->internal->avctx->channel_layout;
st->codecpar->codec_id = st->internal->avctx->codec_id;
} else {
/* free packet */
av_packet_unref(pkt);
}
if (pkt->flags & AV_PKT_FLAG_KEY)
st->internal->skip_to_keyframe = 0;
if (st->internal->skip_to_keyframe) {
av_packet_unref(pkt);
got_packet = 0;
}
}
if (!got_packet && s->internal->parse_queue)
ret = avpriv_packet_list_get(&s->internal->parse_queue, &s->internal->parse_queue_end, pkt);
//It mainly deals with some discarded sample s, which can be ignored for the time being
if (ret >= 0) {
AVStream *st = s->streams[pkt->stream_index];
int discard_padding = 0;
if (st->internal->first_discard_sample && pkt->pts != AV_NOPTS_VALUE) {
int64_t pts = pkt->pts - (is_relative(pkt->pts) ? RELATIVE_TS_BASE : 0);
int64_t sample = ts_to_samples(st, pts);
int duration = ts_to_samples(st, pkt->duration);
int64_t end_sample = sample + duration;
if (duration > 0 && end_sample >= st->internal->first_discard_sample &&
sample < st->internal->last_discard_sample)
discard_padding = FFMIN(end_sample - st->internal->first_discard_sample, duration);
}
if (st->internal->start_skip_samples && (pkt->pts == 0 || pkt->pts == RELATIVE_TS_BASE))
st->internal->skip_samples = st->internal->start_skip_samples;
if (st->internal->skip_samples || discard_padding) {
uint8_t *p = av_packet_new_side_data(pkt, AV_PKT_DATA_SKIP_SAMPLES, 10);
if (p) {
AV_WL32(p, st->internal->skip_samples);
AV_WL32(p + 4, discard_padding);
av_log(s, AV_LOG_DEBUG, "demuxer injecting skip %d / discard %d\n", st->internal->skip_samples, discard_padding);
}
st->internal->skip_samples = 0;
}
if (st->internal->inject_global_side_data) {
for (i = 0; i < st->nb_side_data; i++) {
AVPacketSideData *src_sd = &st->side_data[i];
uint8_t *dst_data;
if (av_packet_get_side_data(pkt, src_sd->type, NULL))
continue;
dst_data = av_packet_new_side_data(pkt, src_sd->type, src_sd->size);
if (!dst_data) {
av_log(s, AV_LOG_WARNING, "Could not inject global side data\n");
continue;
}
memcpy(dst_data, src_sd->data, src_sd->size);
}
st->internal->inject_global_side_data = 0;
}
}
av_opt_get_dict_val(s, "metadata", AV_OPT_SEARCH_CHILDREN, &metadata);
if (metadata) {
s->event_flags |= AVFMT_EVENT_FLAG_METADATA_UPDATED;
av_dict_copy(&s->metadata, metadata, 0);
av_dict_free(&metadata);
av_opt_set_dict_val(s, "metadata", NULL, AV_OPT_SEARCH_CHILDREN);
}
...
/* A demuxer might have returned EOF because of an IO error, let's
* propagate this back to the user. */
if (ret == AVERROR_EOF && s->pb && s->pb->error < 0 && s->pb->error != AVERROR(EAGAIN))
ret = s->pb->error;
return ret;
}
Call the protocol interface to read the packet as follows
Analysis of source code of ffmpeg unpacking mov/mp4 format unpacking_ read_ Header (read metadata), mov_ read_ Packet (read sample data), mov_read_trak) It is introduced in
//Call the protocol interface to read the packet
int ff_read_packet(AVFormatContext *s, AVPacket *pkt)
{
int ret, i, err;
AVStream *st;
pkt->data = NULL;
pkt->size = 0;
av_init_packet(pkt); //Assign initial value to AVPacket
//Small Tips: for (;) Because in the compilation phase, the assembler will put for (;) Optimize it and enter the cycle directly
for (;;) {
AVPacketList *pktl = s->internal->raw_packet_buffer;
const AVPacket *pkt1;
if (pktl) {
st = s->streams[pktl->pkt.stream_index];
if (s->internal->raw_packet_buffer_remaining_size <= 0)
if ((err = probe_codec(s, st, NULL)) < 0)
return err;
if (st->internal->request_probe <= 0) {
//Get raw_ packet_ Packet in buffer
avpriv_packet_list_get(&s->internal->raw_packet_buffer,
&s->internal->raw_packet_buffer_end, pkt);
s->internal->raw_packet_buffer_remaining_size += pkt->size;
return 0;
}
}
//Call the corresponding external read packet interface according to AVInputFormat
//See the link above for details
//Analysis of source code of ffmpeg unpacking mov/mp4 format unpacking_ read_ Header (read metadata), mov_ read_ Packet (read sample data), mov_read_trak
ret = s->iformat->read_packet(s, pkt);
if (ret < 0) { //Less than 0 free packet
...
}
//Pkt - > buf open up memory, initialize
//Assign the content of pkt - > data to pkt - > buf - > data. My personal understanding of this interface is to take the content of packet from AVPacket and take it from pkt - > buf uniformly
err = av_packet_make_refcounted(pkt);
if (err < 0) {
av_packet_unref(pkt);
return err;
}
st = s->streams[pkt->stream_index];
...
pkt->dts = wrap_timestamp(st, pkt->dts);
pkt->pts = wrap_timestamp(st, pkt->pts);
force_codec_ids(s, st);
/* TODO: audio: time filter; video: frame reordering (pts != dts) */
if (s->use_wallclock_as_timestamps)
pkt->dts = pkt->pts = av_rescale_q(av_gettime(), AV_TIME_BASE_Q, st->time_base);
if (!pktl && st->internal->request_probe <= 0)
//Return here
return ret;
In general, you return here when you get there packet And not request_probe Return in case of,therefore s->internal->raw_packet_buffer Generally NULL
//Assign packet to raw_packet_buffer
//When raw_packet_buffer_ When end is null, raw_packet_buffer is assigned
err = avpriv_packet_list_put(&s->internal->raw_packet_buffer,
&s->internal->raw_packet_buffer_end,
pkt, NULL, 0);
if (err < 0) {
av_packet_unref(pkt);
return err;
}
pkt1 = &s->internal->raw_packet_buffer_end->pkt;
s->internal->raw_packet_buffer_remaining_size -= pkt1->size;
if ((err = probe_codec(s, st, pkt1)) < 0)
return err;
}
}
//Pkt - > buf open up memory, initialize
//Assign the content of pkt - > data to pkt - > buf - > data. My personal understanding of this interface is to take the content of packet from AVPacket and take it from pkt - > buf uniformly
int av_packet_make_refcounted(AVPacket *pkt)
{
int ret;
if (pkt->buf)
return 0;
ret = packet_alloc(&pkt->buf, pkt->size); //Open up memory
if (ret < 0)
return ret;
av_assert1(!pkt->size || pkt->data);
if (pkt->size)
memcpy(pkt->buf->data, pkt->data, pkt->size);//assignment
pkt->data = pkt->buf->data;
return 0;
}
Important Tips:
As mentioned above, avformatinternal - > AVPacketList is used to store the unpacked data. The assignment of AVPacketList adopts the method of queue, and ffplay imitates this method, so let's combine the two here
ffmpeg queue assignment and acquisition
//plast_pktl variable is the tail pointer of AVPacketList. After inserting elements into the new linked list, plast_pktl moves backward automatically
//When plast_ When pktl is NULL, add packet to packet_ In the buffer, you need to read the packet every time_ Buffer content
int avpriv_packet_list_put(AVPacketList **packet_buffer, AVPacketList **plast_pktl , AVPacket *pkt,
int (*copy)(AVPacket *dst, const AVPacket *src), int flags)
{
AVPacketList *pktl = av_mallocz(sizeof(AVPacketList));//Open up memory space
int ret;
if (!pktl)
return AVERROR(ENOMEM);
if (copy) {
ret = copy(&pktl->pkt, pkt);
if (ret < 0) {
av_free(pktl);
return ret;
}
} else {
//Assign the value of pkt - > data to pkt - > buf - > data
ret = av_packet_make_refcounted(pkt);
if (ret < 0) {
av_free(pktl);
return ret;
}
av_packet_move_ref(&pktl->pkt, pkt);
}
//plast_pktl variable is the tail pointer of AVPacketList. After inserting elements into the new linked list, plast_pktl moves backward automatically
//When plast_ When pktl is NULL, add packet to packet_ In the buffer, you need to read the packet every time_ Buffer content
if (*packet_buffer)
(*plast_pktl)->next = pktl;
else
*packet_buffer = pktl; //When plast_ When pktl is empty, pktl is assigned to packet_buffer
/* Add the packet in the buffered packet list. */
*plast_pktl = pktl; //Move linked list pointer
return 0;
}
//Get queue packet
//When pkt_ When buffer is null, AVPacketList tail pointer pkt_buffer_end assignment is null
int avpriv_packet_list_get(AVPacketList **pkt_buffer,
AVPacketList **pkt_buffer_end,
AVPacket *pkt)
{
AVPacketList *pktl;
if (!*pkt_buffer)
return AVERROR(EAGAIN);
pktl = *pkt_buffer;
*pkt = pktl->pkt;
*pkt_buffer = pktl->next;//Move linked list pointer
if (!pktl->next)
//When pkt_ When buffer is null, the AVPacketList tail pointer pkt_buffer_end assignment is null
*pkt_buffer_end = NULL;
av_freep(&pktl);
return 0;
}
ffplay queue assignment and acquisition
//ffplay queue structure
typedef struct PacketQueue {
//first_pkt is the head of the queue. Every time a packet is taken from the queue, it is taken from the first_pkt inside
//last_pkt variable is the tail pointer of MyAVPacketList. After inserting elements into the new linked list, q - > last_pkt automatically moves backward,
//When last_ When pkt is empty, it will be given to first_pkt assignment packet
MyAVPacketList *first_pkt, *last_pkt;
int nb_packets;//How many packet s are there in the queue
int size; //Queue size
int64_t duration; //Queue duration, that is, the duration of the packet
int abort_request; //Discard assignment ID
int serial; //Queue serial number
SDL_mutex *mutex; //The lock of the queue, because the assignment of the queue is in multithreading
SDL_cond *cond; //Semaphores are used for synchronization
} PacketQueue;
//To packetqueue * q - > first_ Pkt linked list assignment
//Activate semaphore for decoding
//This interface is modeled after avpriv in ffmpeg_ packet_ list_ Put interface
//Where Q - > last_ Pkt variable is the tail pointer of MyAVPacketList. After inserting elements into the new linked list, q - > last_ Pkt auto move backward
//When last_ When pkt is NULL, add packet to first_ In pkt, you need to read first every time_ Pkt content
static int packet_queue_put_private(PacketQueue *q, AVPacket *pkt)
{
MyAVPacketList *pkt1;
if (q->abort_request)//Discard assignment ID
return -1;
pkt1 = av_malloc(sizeof(MyAVPacketList));//Linked list opens up memory
if (!pkt1)
return -1;
pkt1->pkt = *pkt;
pkt1->next = NULL;
if (pkt == &flush_pkt)//If the packet is refreshed, the sequence number of the queue is increased by one, that is, it is regarded as a new queue
q->serial++;
pkt1->serial = q->serial; //Update the queue where pkt1 is located
//q->last_ Pkt variable can be regarded as an identifier
//When last_ When pkt is NULL, add packet to first_ In pkt, you need to read first every time_ Pkt content
if (!q->last_pkt)
q->first_pkt = pkt1;
else
q->last_pkt->next = pkt1;
q->last_pkt = pkt1;//Move last_pkt linked list pointer
q->nb_packets++; //The number of packs this queue already has
q->size += pkt1->pkt.size + sizeof(*pkt1);
q->duration += pkt1->pkt.duration; //Queue duration
/* XXX: should duplicate packet data in DV case */
SDL_CondSignal(q->cond); //Activate semaphore for decoding
return 0;
}
//Get packet from queue
//When first_ When pkt is null, the tail pointer of MyAVPacketList is last_pkt assignment is null
//If it's first_ No value in pkt, thread waiting, waiting for packet_ queue_ put_ First in private_ SDL after pkt assignment_ Condsignal (Q - > cond) to activate
static int packet_queue_get(PacketQueue *q, AVPacket *pkt, int block, int *serial)
{
MyAVPacketList *pkt1;
int ret;
SDL_LockMutex(q->mutex); //Lock
//Small Tips: for (;) Because in the compilation phase, the assembler will put for (;) Optimize it and enter the cycle directly
for (;;) {
if (q->abort_request) { //Whether to abandon the queue
ret = -1;
break;
}
pkt1 = q->first_pkt; //List first_pkt removal
if (pkt1) {
q->first_pkt = pkt1->next;//Move linked list pointer
if (!q->first_pkt)
//When first_ When pkt is empty, last will be identified_ Pkt assignment is null
q->last_pkt = NULL;
q->nb_packets--;//Packets in the queue minus one
q->size -= pkt1->pkt.size + sizeof(*pkt1);//Subtract the number of packet s
q->duration -= pkt1->pkt.duration;//Minus total time
*pkt = pkt1->pkt;//packet assignment
if (serial)
//Assign a value to the serial number of the retrieved packet
*serial = pkt1->serial;
av_free(pkt1);
ret = 1;
break;
} else if (!block) {
ret = 0;
break;
} else {
//If it's first_ No value in pkt, thread waiting, waiting for packet_ queue_ put_ First in private_ SDL after pkt assignment_ Condsignal (Q - > cond) to activate
SDL_CondWait(q->cond, q->mutex);
}
}
SDL_UnlockMutex(q->mutex);
return ret;
}
av_ read_ The frame interface is often used in ffplay. In fact, it only calls read_frame_internal
int av_read_frame(AVFormatContext *s, AVPacket *pkt)
{
const int genpts = s->flags & AVFMT_FLAG_GENPTS;
int eof = 0;
int ret;
AVStream *st;
if (!genpts) {
ret = s->internal->packet_buffer
? avpriv_packet_list_get(&s->internal->packet_buffer,
&s->internal->packet_buffer_end, pkt)
: read_frame_internal(s, pkt);
if (ret < 0)
return ret;
//Jump statement directly to return_packet
goto return_packet;
}
....
ret = avpriv_packet_list_put(&s->internal->packet_buffer,
&s->internal->packet_buffer_end,
pkt, NULL, 0);
if (ret < 0) {
av_packet_unref(pkt);
return ret;
}
}
return_packet:
st = s->streams[pkt->stream_index];
if ((s->iformat->flags & AVFMT_GENERIC_INDEX) && pkt->flags & AV_PKT_FLAG_KEY) {
ff_reduce_index(s, st->index);
av_add_index_entry(st, pkt->pos, pkt->dts, 0, 0, AVINDEX_KEYFRAME);
}
if (is_relative(pkt->dts))
pkt->dts -= RELATIVE_TS_BASE;
if (is_relative(pkt->pts))
pkt->pts -= RELATIVE_TS_BASE;
return ret;
}