ffplay is a self-contained open source player instance in the ffmpeg source code. It also supports the playback of local video files and online streaming media. It is very powerful.
FFplay: FFplay is a very simple and portable media player using the FFmpeg libraries and the SDL library. It is mostly used as a testbed for the various FFmpeg APIs.
The code in ffplay fully calls the function library in ffmpeg. Therefore, if you want to learn the use of ffmpeg or develop your own player based on ffmpeg, ffplay is a good entry point.
Because the development documents of ffmpeg itself are relatively few, and the implementation of the source code of ffplay player is relatively complex, in addition to the basic ffmpeg component calls, it also includes the rendering of video frames, playing of audio frames, audio-video synchronization strategy and thread scheduling.
Therefore, here we take the development routine of a simplified version of ffplay player recommended by ffmpeg official website as the basis, step by step, from simple to deep, and finally discuss the complete logic of realizing a video player.
In previous articles, we discussed the concept of timestamp related to audio and video synchronization, such as through video_ clock & audio_ Clock tracks the timestamp of the current playback progress, how to synchronize with the master clock through interpolation and frame loss during audio decoding, and how to dynamically predict the delayed refresh display time of the next frame to synchronize video playback.
So far, the main technical principle and function realization of a video player have been introduced. At the end of this series, we will discuss how to implement the [fast forward] / [fast back] function of the player.
Official account: breakpoint laboratory audio and video development series
Construction of ffmpeg source code compilation environment
ffplay source code compilation
Implementation details of ffmpeg player - Framework Construction
ffmpeg player implementation details - video display
ffmpeg player implementation details - audio playback
ffmpeg player implementation details - create thread
Implementation of detailed video synchronization control in ffmpeg player
Implementation of detailed audio synchronization control in ffmpeg player
Detailed explanation of ffmpeg player - fast forward and fast backward control
1. Fast forward and fast backward event handling
Let's first look at the trigger and processing mechanism of [fast forward] / [fast back] events. Here, we still realize [fast forward] / [fast back] control through asynchronous event mechanism.
Firstly, the [fast forward] / [fast back] event is triggered by the up, down, left and right direction keys. The left and right keys set [fast forward] / [fast back] 10s, and the up and down keys set [fast forward] / [fast back] 60s.
Then, in the event loop processing logic of the main function, the message corresponding to each key is monitored and captured through sdl, and then jump to do through goto_ Seek executes specific event processing logic.
int main(int argc, char *argv[]) {
...
for (;;) {
double incr, pos;
SDL_WaitEvent(&event);//Use this function to wait indefinitely for the next available event
switch (event.type) {//After the event arrives, wake up the main thread, check the event type and perform corresponding operations
case SDL_KEYDOWN://Check keyboard operation events
switch (event.key.keysym.sym) {//Check the keyboard operation type, which key get hit
case SDLK_LEFT://[left key]
incr = -10.0;//Back 10s
goto do_seek;
case SDLK_RIGHT://[right click]
incr = 10.0;//Fast forward 10s
goto do_seek;
case SDLK_UP://[up]
incr = 60.0;//Fast forward 60s
goto do_seek;
case SDLK_DOWN://[down]
incr = -60.0;//Backward 60s
goto do_seek;
With the monitoring of [fast forward] / [fast back] events, let's take a look at the processing process of specific events.
We first add several fields in the VideoState global state parameter set to track the states related to [fast forward] / [fast back]. Other components judge these state values in their own logic and perform corresponding actions.
typedef struct VideoState {
...
int seek_req;//[fast forward] / [back] operation start flag bit
int seek_flags;//[fast forward] / [back] operation type flag bit
int64_t seek_pos;//Reference timestamp after [fast forward] / [back] operation
...
}VideoState;
Next, let's look at the event processing logic.
int main(int argc, char *argv[]) {
...
do_seek://Processing requests
if (global_video_state) {
pos = get_master_clock(global_video_state);//Gets the timestamp of the current primary synchronization source
pos += incr;//Update the main synchronization source timestamp according to the keyboard operation (AV_TIME_BASE is the timestamp reference value)
stream_seek(global_video_state,(int64_t)(pos*AV_TIME_BASE),incr);//Set the lookup location based on the primary synchronization source timestamp
}
break;
In do_ Key events are handled in seek.
- First, get the current master synchronization source clock through getMasterClock.
- Then, the time difference value of [fast forward] / [fast backward] is updated to the time type of the main synchronization source as the new synchronization target time.
- Finally, we are in the stream_ Update the global status information of [fast forward] / [fast backward] in seek.
//Set [fast forward] / [fast backward] status parameters
void stream_seek(VideoState *is, int64_t pos, int rel) {
if (!is->seek_req) {//Check whether the [fast forward] / [reverse] operation flag is turned on
is->seek_pos = pos;//Update reference timestamp after [fast forward] / [back]
is->seek_flags = rel < 0 ? AVSEEK_FLAG_BACKWARD : 0;//Determine [fast forward] or [reverse] operation
is->seek_req = 1;//Turn on the [fast forward] / [back] flag bit
}
}
These global state parameters are in stream_ After the update in seek, other components can perform corresponding actions in their own logic according to these status information.
2. Realization of fast forward and fast backward function
We are in the stream_ The global status parameters of [fast forward] / [fast back] are updated in seek, and the parse function of the encoded packet parsing thread is used_ Judge whether the [fast forward] / [fast backward] operation flag bit is turned on in the thread. When it is judged that there is a [fast forward] / [fast backward] request, perform the following three actions
- First, convert the time unit to seek_ The unit of target is determined by AV_TIME_BASE_Q to time_base
- Via AV_ seek_ The frame function jumps to the specified frame according to the target timestamp
- Clear the current encoded data cache queue
The specific implementation is as follows
int parse_thread(void *arg) {
...
// Seek stuff goes here
if (is->seek_req) {//Check whether the [fast forward] / [fast reverse] operation flag is on
int stream_index= -1;//Initialize audio and video stream type label
int64_t seek_target = is->seek_pos;//Gets the reference timestamp after the [fast forward] / [fast backward] operation
if (is->videoStream >= 0) {//Check whether the video stream type label is obtained
stream_index = is->videoStream;//Get video stream type label
} else if (is->audioStream >= 0) {//Check whether the audio stream type label is obtained
stream_index = is->audioStream;//Gets the audio stream type label
}
if (stream_index >= 0){//Check whether the audio and video stream type label is obtained
//Time unit conversion, set seek_ The unit of target is determined by AV_TIME_BASE_Q to time_base
seek_target= av_rescale_q(seek_target, AV_TIME_BASE_Q, pFormatCtx->streams[stream_index]->time_base);
}
//Jump to the specified frame according to the timestamp after the [fast forward] / [fast backward] operation (this function can only jump to the key frame closest to the specified frame)
if (av_seek_frame(is->pFormatCtx, stream_index, seek_target, is->seek_flags) < 0) {
fprintf(stderr, "%s: error while seeking\n", is->pFormatCtx->filename);
} else {//After [fast forward] / [fast backward] operation, immediately empty the cache queue and reset the audio and video decoder
if (is->audioStream >= 0) {//Check whether the audio stream type label is obtained
packet_queue_flush(&is->audioq);//Clear the audio queue cache and release all dynamically allocated memory in the queue
packet_queue_put(&is->audioq, &flush_pkt);//flush_pkt inserts the audio packet queue and performs the operation of resetting the audio decoder avcodec_flush_buffers
}
if (is->videoStream >= 0) {//Get video stream type label
packet_queue_flush(&is->videoq);//Clear the video queue cache and release all dynamically allocated memory in the queue
packet_queue_put(&is->videoq, &flush_pkt);//flush_pkt inserts the video packet queue and performs the operation of resetting the video decoder avcodec_flush_buffers
}
}
is->seek_req = 0;//Turn off the [fast forward] / [fast backward] operation flag bit
}//end for if (is->seek_req)
Note here that AV_ seek_ Input parameters of frame seek_target is not a specific time length, such as seconds, but a value based on avcodec's internal time base unit. Therefore, time unit conversion is required here.
The principle of emptying the queue is as follows: insert a special flush packet into the queue. After the decoding thread detects this packet, it immediately empties the current encoded data cache, and then waits for the arrival of the next packet.
After clearing the encoded data queue, parse_thread will be based on AV_ seek_ The target timestamp input by frame re parses the encoded data from the specified frame to start the next round of parsing process.
3. Source code compilation verification
The compilation method of the source code is exactly the same as the previous routine. The source code can be compiled by using the Makefile script as follows
tutorial07: tutorial07.c
gcc -o tutorial07 -g3 tutorial07.c -I${FFMPEG_INCLUDE} -I${SDL_INCLUDE} \
-L${FFMPEG_LIB} -lavutil -lavformat -lavcodec -lswscale -lswresample -lz -lm \
`sdl-config --cflags --libs`
clean:
rm -rf tutorial07
Execute the make command to start compilation. After compilation, an executable file named [tutorial07] can be generated in the source directory.
Similar to the use method of ffplay, execute the [tutorial07 url] command. The URL can select a local video file or a media stream address.
./tutorial07 ./xxx.mp4
Enter Ctrl+C to end the program
4. Source list
So far, a complete video player model is introduced. Although this is only the basic model of a player, the content is not checked, involving all aspects of content. In order to simplify the difficulty of learning, almost all codes are annotated here. You can debug by yourself to deepen your understanding.
// tutorial07.c
// A pedagogical video player that really works! Now with seeking features.
//
// This tutorial was written by Stephen Dranger (dranger@gmail.com).
//
// Code based on FFplay, Copyright (c) 2003 Fabrice Bellard,
// and a tutorial by Martin Bohme (boehme@inb.uni-luebeckREMOVETHIS.de)
// Tested on Gentoo, CVS version 5/01/07 compiled with GCC 4.1.1
//
// Updates tested on:
// Mac OS X 10.11.6
// Apple LLVM version 8.0.0 (clang-800.0.38)
//
// Use
//
// $ gcc -o tutorial07 tutorial07.c -lavutil -lavformat -lavcodec -lswscale -lswresample -lz -lm `sdl-config --cflags --libs`
//
// to build (assuming libavutil/libavformat/libavcodec/libswscale/libswresample are correctly installed your system).
//
// Run using
//
// $ tutorial07 myvideofile.mpg
//
// to play the video.
/*---------------------------
//1,Before processing messages, the message queue processing function locks the mutex to protect the critical area resources in the message queue
//2,If the message queue is empty, pthread is called_ cond_ Wait temporarily unlocks the mutex and waits for other threads to insert message data into the message queue
//3,After other threads insert message data into the message queue, use pthread_cond_signal sends a qready signal like a waiting thread
//4,After receiving the qready signal, the message queue processing thread is awakened and regains exclusive access to the resources in the critical area of the message queue
#include <pthread.h>
struct msg{//Message queue structure
struct msg *m_next;//Message queue successor node
//more stuff here
}
struct msg *workq;//Message queue pointer
pthread_cond_t qready=PTHREAD_COND_INITIALIZER;//Message queue readiness condition variable
pthread_mutex_t qlock=PTHREAS_MUTEX_INITIALIZER;//Message queue mutex to protect message queue data
//Message queue processing function
void process_msg(void){
struct msg *mp;//Message structure pointer
for(;;){
pthread_mutex_lock(&qlock);//Message queue mutex is locked to protect message queue data
while(workq==NULL){//Check whether the message queue is empty. If it is empty
pthread_cond_wait(&qready,&qlock);//Wait for the message queue ready signal qready and unlock the mutex temporarily. When the function returns, the mutex is locked again
}
mp=workq;//The thread wakes up and fetches data from the message queue for processing
workq=mp->m_next;//Update the message queue and move the pointer back to clear the fetched messages
pthread_mutex_unlock(&qlock);//Release lock
//now process the message mp
}
}
//Insert message into message queue
void enqueue_msg(struct msg *mp){
pthread_mutex_lock(&qlock);//Message queue mutex is locked to protect message queue data
mp->m_next=workq;//The original queue header is used as the successor node of the inserted message
workq=mp;//Insert a new message into the queue
pthread_mutex_unlock(&qlock);//Release lock
pthread_cond_signal(&qready);//Send a qready message to the waiting thread to inform it that the message queue is ready
}
---------------------------*/
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libavformat/avio.h>
#include <libswresample/swresample.h>
#include <libswscale/swscale.h>
#include <libavutil/avstring.h>
#include <libavutil/opt.h>
#include <libavutil/time.h>
#include <SDL.h>
#include <SDL_thread.h>
#ifdef __MINGW32__
#undef main // Prevents SDL from overriding main().
#endif
#include <stdio.h>
#include <math.h>
#define SDL_AUDIO_BUFFER_SIZE 1024
#define MAX_AUDIO_FRAME_SIZE 192000
#define MAX_AUDIOQ_SIZE (5 * 16 * 1024)
#define MAX_VIDEOQ_SIZE (5 * 256 * 1024)
#define AV_SYNC_THRESHOLD 0.01
#define AV_NOSYNC_THRESHOLD 10.0
#define SAMPLE_CORRECTION_PERCENT_MAX 10
#define AUDIO_DIFF_AVG_NB 20
#define FF_ALLOC_EVENT (SDL_USEREVENT)
#define FF_REFRESH_EVENT (SDL_USEREVENT + 1)
#define FF_QUIT_EVENT (SDL_USEREVENT + 2)
#define VIDEO_PICTURE_QUEUE_SIZE 1
#define DEFAULT_AV_SYNC_TYPE AV_SYNC_VIDEO_MASTER
SDL_Surface *screen;//SDL drawing surface, a structure that contains a collection of pixels used in software painting
//AVPacket type object whose data member is marked as "FLUSH". When the data member of a packet is "FLUSH" in the packet queue during decoding, reset the decoder
AVPacket flush_pkt;//After [fast forward] / [fast backward] operation, ffmpeg needs to reset the decoder
uint64_t global_video_pkt_pts = AV_NOPTS_VALUE;
enum {//Time Synchronisation Source
AV_SYNC_AUDIO_MASTER,//Audio clock master synchronization source
AV_SYNC_VIDEO_MASTER,//Master clock synchronization source
AV_SYNC_EXTERNAL_MASTER,//External clock master synchronization source
};
/*-------Get the current system time--------
int64_t av_gettime(void){
#if defined(CONFUG_WINCE)
return timeGetTime()*int64_t_C(1000);
#elif defined(CONFIG_WIN32)
struct _timeb tb;
_ftime(&tb);
return ((int64_t)tb.time*int64_t_C(1000)+(int64_t)tb.millitm)*int64_t_C(1000);
#else
struct timeval tv;
gettimeofday(&tv,NULL);//Get the current system time
return (int64_t)tv.tv_sec*1000000+tv.tv_usec;//In 1 / 1000000 second, it is easy to migrate on various platforms
#endif
}
---------------------------*/
/*-------Linked list node structure--------
typedef struct AVPacketList {
AVPacket pkt;//Linked list data
struct AVPacketList *next;//Linked list successor node
} AVPacketList;
---------------------------*/
//Encoded packet queue (linked list) structure
typedef struct PacketQueue {
AVPacketList *first_pkt, *last_pkt;//Queue head and tail node pointer
int nb_packets;//queue length
int size;//Cache length for saving encoded data, size = packet - > size
SDL_mutex *qlock;//Queue mutex to protect queue data
SDL_cond *qready;//Queue ready condition variable
} PacketQueue;
//Image frame structure
typedef struct VideoPicture {
SDL_Overlay *bmp;//SDL canvas overlay
int width, height;//Source height & width
int allocated;//Whether to allocate memory space and convert video frames to SDL overlay ID
double pts;//Absolute display timestamp of the current image frame
} VideoPicture;
//Global maintenance video playback status structure
typedef struct VideoState {
AVFormatContext *pFormatCtx;//Structure for storing file container packaging information and code stream parameters
AVPacket audio_pkt;//Save encoded packets extracted from the queue
AVFrame audio_frame;//Save the audio data decoded from the packet
AVStream *video_st;//Video stream information structure
AVStream *audio_st;//Audio stream information structure
struct SwsContext *sws_ctx;//Structure describing converter parameters
struct SwsContext *sws_ctx_audio;
AVIOContext *io_context;
PacketQueue videoq;//Video coding packet queue (coding data queue, implemented in the form of linked list)
//After decoding the image frame queue (decoding data queue, implemented in array), the rendering logic will obtain data from pictq, and the decoding logic will write data to pictq
VideoPicture pictq[VIDEO_PICTURE_QUEUE_SIZE];
int pictq_size, pictq_rindex, pictq_windex;//Queue length, read / write location index
SDL_mutex *pictq_lock;//Queue read / write lock object to protect image frame queue data
SDL_cond *pictq_ready;//Queue ready condition variable
PacketQueue audioq;//Audio encoded data packet queue (encoded data queue, implemented in linked list)
uint8_t audio_buf[(MAX_AUDIO_FRAME_SIZE*3)/2];//Save multiple frames of original audio data after decoding a packet (decoded data queue, implemented in array)
unsigned int audio_buf_size;//Length of decoded multi frame audio data
unsigned int audio_buf_index;//Cumulative write sound card cache length
uint8_t *audio_pkt_data;//Encoded data cache pointer position
int audio_pkt_size;//The length of encoded data remaining in the cache (whether a complete pakcet packet has been decoded, and a packet may contain multiple audio encoded frames)
int audio_hw_buf_size;
int videoStream, audioStream;//Audio and video stream type label
SDL_Thread *parse_tid;//Unpacking thread id
SDL_Thread *video_tid;//Video decoding thread id
char filename[1024];//Enter the full pathname of the file
int quit;//The global exit process ID tells the thread to exit after clicking exit on the interface
//video/audio_clock save pts of last decoded frame/predicted pts of next decoded frame
//keep track of how much time has passed according to the video/audio
double video_clock;//Video clock, tracking video display timestamp
double audio_clock;//Audio clock, tracking audio playback timestamp
double frame_timer;//The cumulative played time when the video is played to the current frame. Track the actual playing time of the video. Take the system time as the benchmark, frame_timer is what time it should be when we display the next frame
double frame_last_pts;//The display timestamp of the previous frame image, which is used in video_ refersh_ The pts value of the previous frame is saved in timer
double frame_last_delay;//Dynamic refresh delay time of last frame image
int av_sync_type;//Primary sync source type
double audio_diff_cum;//U cumulative time difference between audio clock and synchronization source, sed for AV difference average calculation
double audio_diff_avg_coef;//Average time difference weighting coefficient between audio clock and synchronization source
double audio_diff_threshold;//Average time difference threshold between audio clock and synchronization source
int audio_diff_avg_count;//Audio out of sync count (the number of times the audio clock is out of sync with the main synchronization source)
double video_current_pts;//Current displayed pts (different from video_clock if frame fifos are used)
//time (av_gettime) at which we updated video_current_pts - used to have running video pts
int64_t video_current_pts_time;//Get video_ current_ Time value of PTS (system absolute time)
double external_clock;//External clock timestamp, External clock base
int64_t external_clock_time;//Obtain the time value of the external clock
int seek_req;//[fast forward] / [fast reverse] operation on flag bit
int seek_flags;//[fast forward] / [fast backward] operation type flag bit
int64_t seek_pos;//Reference timestamp after [fast forward] / [fast backward] operation
} VideoState;// Since we only have one decoding thread, the Big Struct can be global in case we need it.
VideoState *global_video_state;
//Encoding data queue initialization function
void packet_queue_init(PacketQueue *q) {
memset(q, 0, sizeof(PacketQueue));//All zero initialization queue structure object
q->qlock = SDL_CreateMutex();//Create mutex object
q->qready = SDL_CreateCond();//Create condition variable object
}
//Insert encoded packets into the queue
int packet_queue_put(PacketQueue *q, AVPacket *pkt) {
/*-------Prepare queue (linked list) node object------*/
AVPacketList *pktlist= av_malloc(sizeof(AVPacketList));
if (!pktlist) {//Check whether the linked list node object is created successfully
return -1;
}
pktlist->pkt = *pkt;//Assign the input packet to the packet object in the new linked list node object
pktlist->next = NULL;//The subsequent pointer of the linked list is null
// if (pkt != &flush_pkt && av_packet_ref(pkt, pkt)<0) {
// return -1;
// }
/*---------Insert new node into queue-------*/
SDL_LockMutex(q->qlock);//Queue mutex is locked to protect queue data
if (!q->last_pkt) {//Check whether the queue tail node exists (check whether the queue is empty)
q->first_pkt = pktlist;//If it does not exist (the end of the queue is empty), the current node will be the first node of the queue
} else {
q->last_pkt->next = pktlist;//If the tail node already exists, hang the current node on the subsequent pointer of the tail node and serve as a new tail node
}
q->last_pkt = pktlist;//Make the current node the new tail node
q->nb_packets++;//Queue length + 1
q->size += pktlist->pkt.size;//Cache length of update queue encoded data
SDL_CondSignal(q->qready);//Sends a message to the waiting thread to inform it that the queue is ready
SDL_UnlockMutex(q->qlock);//Release mutex
return 0;
}
//Extract encoded packets from the queue and take the extracted packets out of the queue
static int packet_queue_get(PacketQueue *q, AVPacket *pkt, int block) {
AVPacketList *pktlist;//Temporary linked list node object pointer
int ret;//Operation results
SDL_LockMutex(q->qlock);//Queue mutex is locked to protect queue data
for (;;) {
if (global_video_state->quit) {//Check exit process ID
ret = -1;
break;
}//end for if
pktlist = q->first_pkt;//The pointer to the first packet that will be passed to the queue
if (pktlist) {//Check whether the packet is empty (whether the queue has data)
q->first_pkt = pktlist->next;//The pointer of the first node of the queue moves back
if (!q->first_pkt) {//Check whether the successor node of the first node exists
q->last_pkt = NULL;//If it does not exist, the tail node pointer is set to null
}
q->nb_packets--;//Queue length - 1
q->size -= pktlist->pkt.size;//Cache length of update queue encoded data
*pkt = pktlist->pkt;//Return the data of the first node of the queue
av_free(pktlist);//Clear the temporary node data (clear the first node data, and the first node is out of the queue)
ret = 1;//Operation succeeded
break;
} else if (!block) {
ret = 0;
break;
} else {//The queue is not ready, at this time through SDL_ The condwait function waits for the qready ready signal and unlocks the mutex temporarily
/*---------------------
* Wait for the queue ready signal qready and unlock the mutex temporarily
* At this time, the thread is blocked and placed on the list of threads with ready waiting conditions
* This makes the thread wake up only after the resources in the critical area are ready, and the thread will not be switched frequently
* When the function returns, the mutex is locked again and subsequent operations are performed
--------------------*/
SDL_CondWait(q->qready, q->qlock);//Temporarily unlock the mutex, block itself, and wait for the critical area resources to be ready (wait for SDL_CondSignal to send the signal that the critical area resources are ready)
}
}
SDL_UnlockMutex(q->qlock);//Release mutex
return ret;
}
//The audio decoding function extracts data packets from the cache queue, decodes them, and returns the decoded data length (decode a complete packet, write the decoded data to the audio_buf cache, and return the total length of multi frame decoded data)
int audio_decode_frame(VideoState *is, double *pts_ptr) {
int coded_consumed_size,data_size=0,pcm_bytes;//The length of encoded data consumed each time [input](len1), the cache length of output original audio data [output], and the number of bytes of each group of audio sampling data
AVPacket *pkt = &is->audio_pkt;//Save packets extracted from the queue
double pts;//Audio playback timestamp
for (;;) {
/*--2,Continuously decode the audio data from the packet pkt until the remaining encoded data length < = 0---*/
while (is->audio_pkt_size>0) {//Check the remaining encoded data length in the cache (whether a complete pakcet packet has been decoded, and a packet may contain multiple audio encoded frames)
int got_frame = 0;//The success identifier of the decoding operation, and a non-zero value is returned successfully
//Decode one frame of audio data and return the consumed encoded data length
coded_consumed_size = avcodec_decode_audio4(is->audio_st->codec, &is->audio_frame, &got_frame, pkt);
if (coded_consumed_size<0) {//Check whether decoding operation is performed
// If error, skip frame.
is->audio_pkt_size=0;//Update encoded data cache length
break;
}
if (got_frame) {//Check whether the decoding operation is successful
if (is->audio_frame.format != AV_SAMPLE_FMT_S16) {//Check whether the audio data format is 16 bit sampling format
//When the audio data is not in 16 bit sampling format, decode is used_ frame_ from_ Packet calculates the length of decoded data
data_size=decode_frame_from_packet(is, is->audio_frame);
} else {//Calculate the decoded audio data length [output]
data_size=av_samples_get_buffer_size(NULL,is->audio_st->codec->channels,is->audio_frame.nb_samples,is->audio_st->codec->sample_fmt, 1);
memcpy(is->audio_buf,is->audio_frame.data[0],data_size);//Copy decoded data to output cache
}
}
is->audio_pkt_data += coded_consumed_size;//Update encoded data cache pointer position
is->audio_pkt_size -= coded_consumed_size;//Update the length of encoded data remaining in the cache
if (data_size <= 0) {//Check the output decoded data cache length
// No data yet, get more frames.
continue;
}
pts = is->audio_clock;
*pts_ptr=pts;
/*---------------------
* When a packet contains multiple audio frames
* The playback timestamp pts of other audio frames in a packet is calculated by [decoded audio original data length] and [sampling rate]
* The sampling frequency is 44.1kHz and the quantization bits are 16 bits, which means that 44.1k data are collected per second, and each data accounts for 2 bytes
--------------------*/
pcm_bytes=2*is->audio_st->codec->channels;//Calculate the number of bytes of each group of audio sampling data = the number of audio sampling bytes of each channel * the number of channels
/*----Update audio_clock---
* A pkt contains multiple audio frame s, and a pkt corresponds to a PTS (pkt - > PTS)
* Therefore, the time stamps of multiple audio frames contained in the pkt are inferred from the following formula
* bytes_per_sec=pcm_bytes*is->audio_st->codec->sample_rate
* Continuously decode from the pkt, infer the pts of each frame of data (in a pkt) and add it to the audio playback clock
--------------------*/
is->audio_clock+=(double)data_size/(double)(pcm_bytes*is->audio_st->codec->sample_rate);
// We have data, return it and come back for more later.
return data_size;//Returns the original data length of decoded data
}
/*-----------------1,Extract packets from cache queue------------------*/
if (pkt->data) {//Check that the packet is residual encoded data
av_packet_unref(pkt);//Release encoded data saved in pkt
}
if (is->quit) {//Check exit process ID
return -1;
}
// Next packet: extract packets from the queue to pkt
if (packet_queue_get(&is->audioq, pkt, 1) < 0) {
return -1;
}
if (pkt->data == flush_pkt.data) {//Check if re decoding is required
avcodec_flush_buffers(is->audio_st->codec);//The decoder needs to be reset before re decoding
continue;
}
is->audio_pkt_data = pkt->data;//Pass encoded data cache pointer
is->audio_pkt_size = pkt->size;//Transfer encoded data cache length
// If update, update the audio clock w/pts.
if (pkt->pts != AV_NOPTS_VALUE) {//Check audio playback timestamp
//When you get a new packet, update audio_clock, update audio with pts in packet_ Clock (one pkt corresponds to one pts)
is->audio_clock = av_q2d(is->audio_st->time_base)*pkt->pts;//Update the time when the audio has been played
}
}
}
/*------Audio Callback-------
* Audio output callback function, through which the system sends the decoded pcm data to the sound card for playback
* The system usually prepares a set of cached pcm data at a time (reducing the i/o times of low-speed system), and sends it to the sound card through the callback. The sound card plays pcm data in turn according to the audio pts
* After the pcm data sent into the cache is played, load a new set of pcm cache data (each time the audio output cache is empty, the system calls this function to fill the audio output cache and send it to the sound card for playback)
* The audio function callback takes the following parameters:
* stream: A pointer to the audio buffer to be filled，Output audio data to sound card cache
* len: The length (in bytes) of the audio buffer,Cache length wanted_spec.samples=SDL_AUDIO_BUFFER_SIZE(1024)
--------------------------*/
void audio_callback(void *userdata, Uint8 *stream, int len) {
VideoState *is = (VideoState *)userdata;//Transfer user data
int wt_stream_len, audio_size;//The length of data sent into the sound card each time, and the length of data after decoding
double pts;//Audio timestamp
while (len > 0) {//Check the remaining length of the audio cache
if (is->audio_buf_index >= is->audio_buf_size) {//Check whether decoding operation is required
//Extract the data packet from the cache queue, decode it, and return the decoded data length, audio_ The buf cache may contain multiple frames of decoded audio data, We have already sent all our data; get more
audio_size = audio_decode_frame(is, &pts);
if (audio_size < 0) {//Check whether the decoding operation is successful
// If error, output silence.
is->audio_buf_size = 1024;
memset(is->audio_buf, 0, is->audio_buf_size);//All zero reset buffer
} else {//The audio synchronization process is added to the callback function, that is, frame loss (or interpolation) is performed on the audio data cache to reduce the time difference between the audio clock and the main synchronization source
audio_size=synchronize_audio(is,(int16_t*)is->audio_buf,audio_size,pts);//Returns the cache length after audio synchronization
is->audio_buf_size=audio_size;//Returns the original audio data length (multiple frames) contained in the packet
}
is->audio_buf_index = 0;//Initialize cumulative write cache length
}
wt_stream_len = is->audio_buf_size - is->audio_buf_index;//Calculate the remaining length of decoding cache
if (wt_stream_len > len) {//Check whether the data length written to the cache each time exceeds the specified length (1024)
wt_stream_len = len;//Fetch data from decoded cache with specified length
}
//Each time, data is extracted from the decoded cache data with a specified length and sent to the sound card
memcpy(stream, (uint8_t *)is->audio_buf + is->audio_buf_index, wt_stream_len);
len -= wt_stream_len;//Update the remaining length of the decoded audio cache
stream += wt_stream_len;//Update cache write location
is->audio_buf_index += wt_stream_len;//Update cumulative write cache data length
}
}
//Video (image) frame rendering
void video_display(VideoState *is) {
SDL_Rect rect;//SDL rectangular object
VideoPicture *vp;//Image frame structure pointer
float aspect_ratio;//Width / height ratio
int w, h, x, y;//Window size and starting position
vp = &is->pictq[is->pictq_rindex];//Extract the image frame structure object from the image frame queue (array)
if (vp->bmp) {//Check whether the pixel data pointer is valid
if (is->video_st->codec->sample_aspect_ratio.num == 0) {
aspect_ratio = 0;
} else {
aspect_ratio = av_q2d(is->video_st->codec->sample_aspect_ratio) * is->video_st->codec->width / is->video_st->codec->height;
}
if (aspect_ratio <= 0.0) {
aspect_ratio = (float) is->video_st->codec->width / (float) is->video_st->codec->height;
}
h = screen->h;
w = ((int) rint(h * aspect_ratio)) & -3;
if (w > screen->w) {
w = screen->w;
h = ((int) rint(w / aspect_ratio)) & -3;
}
x = (screen->w - w) / 2;
y = (screen->h - h) / 2;
//Set rectangular display area
rect.x = x;
rect.y = y;
rect.w = w;
rect.h = h;
SDL_DisplayYUVOverlay(vp->bmp, &rect);//Image rendering
}
}
//Create / reset image frames and allocate memory space for image frames
void alloc_picture(void *userdata) {
VideoState *is = (VideoState *)userdata;//Transfer user data
VideoPicture *vp= &is->pictq[is->pictq_windex];//Extract the image frame structure object from the image frame queue (array)
if (vp->bmp) {//Check whether the image frame already exists
// We already have one make another, bigger/smaller.
SDL_FreeYUVOverlay(vp->bmp);//Release the current overlay cache
}
// Allocate a place to put our YUV image on that screen to recreate the pixel buffer according to the specified size and pixel format
vp->bmp = SDL_CreateYUVOverlay(is->video_st->codec->width, is->video_st->codec->height, SDL_YV12_OVERLAY, screen);
vp->width = is->video_st->codec->width;//Set image frame width
vp->height = is->video_st->codec->height;//Set image frame height
SDL_LockMutex(is->pictq_lock);//Lock the mutex to protect the pixel data of the canvas
vp->allocated = 1;//Image frame pixel buffer allocated memory
SDL_CondSignal(is->pictq_ready);//Sends a message to the waiting thread to inform it that the queue is ready
SDL_UnlockMutex(is->pictq_lock);//Release mutex
}
/*---------------------------
* queue_picture: Image frames are inserted into the queue waiting for rendering
* @is: Global state parameter set
* @pFrame: Structure for storing image decoding data
* @pts: Absolute display timestamp of the current image frame
* 1,First, check whether there is space in the image frame queue (array) to insert a new image. If there is not enough space to insert an image, make the current thread sleep and wait
* 2,Under the condition of initialization, the bmp object (YUV overlay) of VideoPicture in the queue (array) has not allocated space, and it passes FF_ ALLOC_ Method call alloc for event event_ Picture allocate space
* 3,When all the bmp objects (YUV overlay) of VideoPicture in the queue (array) have allocated space, directly skip step 2 and copy the pixel data to the bmp object. The pixel data will be copied after format conversion
---------------------------*/
int queue_picture(VideoState *is, AVFrame *pFrame, double pts) {
/*-------1,Check whether the queue has insertion space------*/
// Wait until we have space for a new pic.
SDL_LockMutex(is->pictq_lock);//Lock the mutex to protect the image frame queue
while (is->pictq_size >= VIDEO_PICTURE_QUEUE_SIZE && !is->quit) {//Check the current length of the queue
SDL_CondWait(is->pictq_ready, is->pictq_lock);//Thread sleep waiting pictq_ready signal
}
SDL_UnlockMutex(is->pictq_lock);//Release mutex
if (is->quit) {//Check the process exit ID
return -1;
}
/*------2,Initialize / reset YUV overlay------*/
// windex is set to 0 initially.
VideoPicture *vp = &is->pictq[is->pictq_windex];//Extracting image frame objects from image frame queue
// Allocate or resize the buffer to check whether YUV overlay exists. Otherwise, initialize YUV overlay and allocate pixel cache space
if (!vp->bmp || vp->width != is->video_st->codec->width || vp->height != is->video_st->codec->height) {
SDL_Event event;//SDL event object
vp->allocated = 0;//Image frame unallocated space
// We have to do it in the main thread.
event.type = FF_ALLOC_EVENT;//Specifies an event that allocates image frame memory
event.user.data1 = is;//Transfer user data
SDL_PushEvent(&event);//Send SDL event
// Wait until we have a picture allocated.
SDL_LockMutex(is->pictq_lock);//Lock the mutex to protect the image frame queue
while (!vp->allocated && !is->quit) {//Check whether the current image frame has been initialized (SDL overlay)
SDL_CondWait(is->pictq_ready, is->pictq_lock);//Thread hibernation waiting alloc_ Picturesend pictq_ The ready signal wakes up the current thread
}
SDL_UnlockMutex(is->pictq_lock);//Release mutex
if (is->quit) {//Check the process exit ID
return -1;
}
}
/*-------3,Copy video frames to YUV overlay------*/
// We have a place to put our picture on the queue.
// If we are skipping a frame, do we set this to null but still return vp->allocated = 1?
AVFrame pict;//Temporarily save the converted image frame pixels, which are associated with the elements in the queue
if (vp->bmp) {//Check whether the pixel data pointer is valid
SDL_LockYUVOverlay(vp->bmp);//locks the overlay for direct access to pixel data, atomic operation to protect pixel buffer and avoid illegal modification
// Point pict at the queue.
pict.data[0] = vp->bmp->pixels[0];//Associate the transcoded image with the pixel buffer of the canvas
pict.data[1] = vp->bmp->pixels[2];
pict.data[2] = vp->bmp->pixels[1];
pict.linesize[0] = vp->bmp->pitches[0];//Associates the scan line length of the transcoded image with the scan line length of the canvas pixel buffer
pict.linesize[1] = vp->bmp->pitches[2];//linesize-Size, in bytes, of the data for each picture/channel plane
pict.linesize[2] = vp->bmp->pitches[1];//For audio, only linesize[0] may be set
// Convert the image into YUV format that SDL uses to convert the decoded image frame into AV_PIX_FMT_YUV420P format and copy it to the image frame queue
sws_scale(is->sws_ctx, (uint8_t const * const *) pFrame->data, pFrame->linesize, 0, is->video_st->codec->height, pict.data, pict.linesize);
SDL_UnlockYUVOverlay(vp->bmp);//Unlocks a previously locked overlay. An overlay must be unlocked before it can be displayed
vp->pts = pts;//Pass the absolute display timestamp of the current image frame
// Now we inform our display thread that we have a pic ready.
if (++is->pictq_windex == VIDEO_PICTURE_QUEUE_SIZE) {//Update and check the write position of the current image frame queue
is->pictq_windex = 0;//Reset image frame queue write position
}
SDL_LockMutex(is->pictq_lock);//Lock the queue read-write lock to protect the queue data
is->pictq_size++;//Update image frame queue length
SDL_UnlockMutex(is->pictq_lock);//Release queue read / write lock
}
return 0;
}
//Modify the function corresponding to the FFmpeg internal exit callback
int decode_interrupt_cb(void *opaque) {
return (global_video_state && global_video_state->quit);
}
//Clear the queue cache and release all dynamically allocated memory in the queue
static void packet_queue_flush(PacketQueue *q) {
AVPacketList *pkt, *pkttmp;//Queue current node
SDL_LockMutex(q->qlock);//Lock mutex
for (pkt = q->first_pkt; pkt != NULL; pkt = pkttmp) {//Traverse all nodes in the queue
pkttmp = pkt->next;//Queue header node backward
av_packet_unref(&pkt->pkt);//Current node reference count - 1
av_freep(&pkt);//Release the current node cache
}
q->last_pkt = NULL;//Queue end node pointer is set to zero
q->first_pkt = NULL;//Zero the queue header node pointer
q->nb_packets = 0;//Set queue length to zero
q->size = 0;//The buffer length of queue encoded data is set to zero
SDL_UnlockMutex(q->qlock);//Mutex unlock
}
//Encoded packet parsing thread function (parses the audio and video encoded data unit from the video file. The data of an AVPacket usually corresponds to a NAL)
int parse_thread(void *arg) {
VideoState *is = (VideoState *)arg;//Pass user parameters
global_video_state = is;//Transfer global state parameter structure
AVFormatContext *pFormatCtx = NULL;//Structure for storing file container packaging information and code stream parameters
AVPacket pkt, *packet = &pkt;//Create a temporary packet object on the stack and associate a pointer
// Find the first video/audio stream.
is->videoStream = -1;//The video stream type label is initialized to - 1
is->audioStream = -1;//The audio stream type label is initialized to - 1s
int video_index = -1;//The video stream type label is initialized to - 1
int audio_index = -1;//The audio stream type label is initialized to - 1
int i;//Cyclic variable
AVDictionary *io_dict = NULL;
AVIOInterruptCB callback;
// Will interrupt blocking functions if we quit!.
callback.callback = decode_interrupt_cb;
callback.opaque = is;
//if (avio_open2(&is->io_context, is->filename, 0, &callback, &io_dict)) {
// fprintf(stderr, "Unable to open I/O for %s\n", is->filename);
// return -1;
//}
// Open video file: open the video file to obtain the packaging information and code stream parameters of the file container
if (avformat_open_input(&pFormatCtx, is->filename, NULL, NULL) != 0) {
return -1; // Couldn't open file.
}
is->pFormatCtx = pFormatCtx;//Transfer file container encapsulation information and code stream parameters
// Retrieve stream information to obtain the bitstream information saved in the file
if (avformat_find_stream_info(pFormatCtx, NULL) < 0) {
return -1; // Couldn't find stream information.
}
// Dump information about file onto standard error, print the bitstream information in pFormatCtx
av_dump_format(pFormatCtx, 0, is->filename, 0);
// Find the first video stream.
for (i=0; i<pFormatCtx->nb_streams; i++) {//Traverse all streaming media types contained in the file (video stream, audio stream, subtitle stream, etc.)
if (pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO && video_index < 0) {//If the file contains a video stream
video_index=i;//Modify the identification with the label of the video stream type so that it is not - 1
}
if (pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO && audio_index < 0) {//If the file contains an audio stream
audio_index=i;//Modify the identification with the label of the audio stream type so that it is not - 1
}
}
if (audio_index >= 0) {//Check if there is an audio stream in the file
stream_component_open(is, audio_index);//Opens the audio stream according to the specified type
}
if (video_index >= 0) {//Check whether there is a video stream in the file
stream_component_open(is, video_index);
}
if (is->videoStream < 0 || is->audioStream < 0) {//Check whether there is audio and video stream in the file
fprintf(stderr, "%s: could not open codecs\n", is->filename);
goto fail;
}
// Main decode loop.
for (;;) {
if (is->quit) {//Check exit process ID
break;
}
// Seek stuff goes here
if (is->seek_req) {//Check whether the [fast forward] / [fast reverse] operation flag is on
int stream_index= -1;//Initialize audio and video stream type label
int64_t seek_target = is->seek_pos;//Gets the reference timestamp after the [fast forward] / [fast backward] operation
if (is->videoStream >= 0) {//Check whether the video stream type label is obtained
stream_index = is->videoStream;//Get video stream type label
} else if (is->audioStream >= 0) {//Check whether the audio stream type label is obtained
stream_index = is->audioStream;//Gets the audio stream type label
}
if (stream_index >= 0){//Check whether the audio and video stream type label is obtained
//Time unit conversion, set seek_ The unit of target is determined by AV_TIME_BASE_Q to time_base
seek_target= av_rescale_q(seek_target, AV_TIME_BASE_Q, pFormatCtx->streams[stream_index]->time_base);
}
//Jump to the specified frame according to the timestamp after the [fast forward] / [fast backward] operation (this function can only jump to the key frame closest to the specified frame)
if (av_seek_frame(is->pFormatCtx, stream_index, seek_target, is->seek_flags) < 0) {
fprintf(stderr, "%s: error while seeking\n", is->pFormatCtx->filename);
} else {//After [fast forward] / [fast backward] operation, immediately empty the cache queue and reset the audio and video decoder
if (is->audioStream >= 0) {//Check whether the audio stream type label is obtained
packet_queue_flush(&is->audioq);//Clear the audio queue cache and release all dynamically allocated memory in the queue
packet_queue_put(&is->audioq, &flush_pkt);//flush_pkt inserts the audio packet queue and performs the operation of resetting the audio decoder avcodec_flush_buffers
}
if (is->videoStream >= 0) {//Get video stream type label
packet_queue_flush(&is->videoq);//Clear the video queue cache and release all dynamically allocated memory in the queue
packet_queue_put(&is->videoq, &flush_pkt);//flush_pkt inserts the video packet queue and performs the operation of resetting the video decoder avcodec_flush_buffers
}
}
is->seek_req = 0;//Turn off the [fast forward] / [fast backward] operation flag bit
}//end for if (is->seek_req)
// Seek stuff goes here to check whether the queue length of audio and video encoded packets overflows
if (is->audioq.size > MAX_AUDIOQ_SIZE || is->videoq.size > MAX_VIDEOQ_SIZE) {
SDL_Delay(10);
continue;
}
//Each image encoded data packet is sequentially read from the file and stored in the AVPacket data structure
if (av_read_frame(is->pFormatCtx, packet) < 0) {
if (is->pFormatCtx->pb->error == 0) {
SDL_Delay(100); // No error; wait for user input.
continue;
} else {
break;
}
}
// Is this a packet from the video stream?.
if (packet->stream_index == is->videoStream) {//Check whether the packet is of video type
packet_queue_put(&is->videoq, packet);//Insert packet into queue
} else if (packet->stream_index == is->audioStream) {//Check if the packet is of audio type
packet_queue_put(&is->audioq, packet);//Insert packet into queue
} else {//Check whether the packet is of caption type
av_packet_unref(packet);//Release the (subtitle) encoded data saved in the packet
}
}
// All done - wait for it.
while (!is->quit) {
SDL_Delay(100);
}
fail:
{
SDL_Event event;//SDL event object
event.type = FF_QUIT_EVENT;//Specify exit event type
event.user.data1 = is;//Transfer user data
SDL_PushEvent(&event);//Push the event object into the SDL background event queue
}
return 0;
}
//Open the stream according to the specified type, find the corresponding decoder, create the corresponding audio configuration, save key information to video state, and start audio and video threads
int stream_component_open(VideoState *is, int stream_index) {
AVFormatContext *pFormatCtx = is->pFormatCtx;//Transfer the encapsulation information and code stream parameters of the file container
AVCodecContext *codecCtx = NULL;//The decoder context object is the interface pointer of the relevant environment, state, resources and parameter set that the decoder depends on
AVCodec *codec = NULL;//The structure that stores codec information and provides a common interface between encoding and decoding. It can be regarded as a global variable between encoder and decoder
AVDictionary *optionsDict = NULL;//SDL_AudioSpec a structure that contains the audio output format, create SDL_AudioSpec structure to set audio playback data
SDL_AudioSpec wanted_spec, spec;
//Check whether the input flow type is within a reasonable range
if (stream_index < 0 || stream_index >= pFormatCtx->nb_streams) {
return -1;
}
// Get a pointer to the codec context for the video stream.
codecCtx = pFormatCtx->streams[stream_index]->codec;//Get decoder context
if (codecCtx->codec_type == AVMEDIA_TYPE_AUDIO) {//Check whether the decoder type is audio decoder
// Set audio settings from codec info,SDL_AudioSpec a structure that contains the audio output format
// Create SDL_AudioSpec structure to set audio playback parameters
wanted_spec.freq = codecCtx->sample_rate;//Sampling frequency DSP frequency -- samples per second
wanted_spec.format = AUDIO_S16SYS;//Sampling format Audio data format
wanted_spec.channels = codecCtx->channels;//Number of channels: 1 mono, 2 stereo
wanted_spec.silence = 0;//Mute when no output
//The default audio buffer size is 512 ~ 8192. ffplay uses 1024 specifications a unit of audio data references to the size of the audio buffer in sample frames
wanted_spec.samples = SDL_AUDIO_BUFFER_SIZE;
wanted_spec.callback = audio_callback;//Set the function to call when the audio device needs more data
wanted_spec.userdata = is;//Transfer user data
//Opens the audio device with the desired parameters(wanted_spec) to open the audio device with the specified parameters
if (SDL_OpenAudio(&wanted_spec, &spec) < 0) {
fprintf(stderr, "SDL_OpenAudio: %s\n", SDL_GetError());
return -1;
}
is->audio_hw_buf_size = spec.size;
}
// Find the decoder for the video stream, find the corresponding decoder according to the decoder context corresponding to the video stream, and return the corresponding decoder (information structure)
codec = avcodec_find_decoder(codecCtx->codec_id);
if (!codec || (avcodec_open2(codecCtx, codec, &optionsDict) < 0)) {
fprintf(stderr, "Unsupported codec!\n");
return -1;
}
//Check decoder type
switch (codecCtx->codec_type) {
case AVMEDIA_TYPE_AUDIO://Audio decoder
is->audioStream = stream_index;//Audio stream type label initialization
is->audio_st = pFormatCtx->streams[stream_index];
is->audio_buf_size = 0;//Length of decoded multi frame audio data
is->audio_buf_index = 0;//Cumulative write sound card cache length
// Averaging filter for audio sync.
is->audio_diff_avg_coef = exp(log(0.01 / AUDIO_DIFF_AVG_NB));//Initialization of average time difference weighting coefficient between audio clock and synchronization source
is->audio_diff_avg_count = 0;//The number of times the initialization audio clock is out of sync with the main synchronization source
// Correct audio only if larger error than this. Initialize the average time difference threshold between the audio clock and the synchronization source
is->audio_diff_threshold = 2.0 * SDL_AUDIO_BUFFER_SIZE / codecCtx->sample_rate;
is->sws_ctx_audio = (struct SwsContext *) swr_alloc();
if (!is->sws_ctx_audio) {
fprintf(stderr, "Could not allocate resampler context\n");
return -1;
}
memset(&is->audio_pkt, 0, sizeof(is->audio_pkt));
packet_queue_init(&is->audioq);//Audio packet queue initialization
SDL_PauseAudio(0);//audio callback starts running again to turn on the audio device. If no data is obtained at this time, it will be muted
break;
case AVMEDIA_TYPE_VIDEO://Video Decoder
is->videoStream = stream_index;//Video stream type label initialization
is->video_st = pFormatCtx->streams[stream_index];//Get video stream information
//Based on the system time, initialize the played time value played to the current frame, which is the real time value, dynamic time value and absolute time value
is->frame_timer = (double)av_gettime() / 1000000.0;
is->frame_last_delay = 40e-3;//Initializes the dynamic refresh delay time of the previous frame image
is->video_current_pts_time = av_gettime();//Get the current system time
packet_queue_init(&is->videoq);//Video packet queue initialization
is->video_tid = SDL_CreateThread(decode_thread,is);//Create decoding thread
// Initialize SWS context for software scaling to set the image conversion pixel format to AV_PIX_FMT_YUV420P
is->sws_ctx = sws_getContext(is->video_st->codec->width, is->video_st->codec->height,
is->video_st->codec->pix_fmt, is->video_st->codec->width, is->video_st->codec->height,
AV_PIX_FMT_YUV420P, SWS_BILINEAR, NULL, NULL, NULL);
codecCtx->get_buffer2 = our_get_buffer;
break;
default:
break;
}
return 0;
}
//Video decoding thread function
int decode_thread(void *arg) {
VideoState *is = (VideoState *)arg;//Transfer user data
AVPacket pkt, *packet = &pkt;//Create a temporary packet object on the stack and associate a pointer
int frameFinished;//Whether the decoding operation is successful, and the decoding frame end flag frameFinished
// Allocate video frame to allocate space for the decoded video information structure and complete the initialization operation (the image cache in the structure is manually installed according to the following two steps)
AVFrame *pFrame= av_frame_alloc();
double pts;//The (absolute) time position of the current frame in the entire video
for (;;) {
if (packet_queue_get(&is->videoq,packet,1)<0){//Extract packets from the queue to packets, and take the extracted packets out of the queue
// Means we quit getting packets.
break;
}
if (packet->data == flush_pkt.data) {//Check if re decoding is required
avcodec_flush_buffers(is->video_st->codec);//The decoder needs to be reset before re decoding
continue;
}
pts = 0;//(absolute) display timestamp initialization
// Save global pts to be stored in pFrame in first call.
global_video_pkt_pts = packet->pts;
/*-------------------------
* Decode video frame，Decode a complete frame of data and set frameFinished to true
* You may not be able to obtain a complete video frame by decoding only one packet. You may need to read multiple packets, avcodec_decode_video2() will set frameFinished to true when decoding to a complete frame
* avcodec_decode_video2 It will decode in the order specified by dts, and then output image frames in the correct display order, with pts of image frames
* If the decoding order is inconsistent with the display order, avcodec_decode_video2 will cache the current frame (framefinished = 0 at this time), and then output the image frames in the correct order
* The decoder bufferers a few frames for multithreaded efficiency
* It is also absolutely required to delay decoding in the case of B frames
* where the decode order may not be the same as the display order.
* Takes input raw video data from frame and writes the next output packet,
* if available, to avpkt. The output packet does not necessarily contain data for the most recent frame,
* as encoders can delay and reorder input frames internally as needed.
-------------------------*/
avcodec_decode_video2(is->video_st->codec, pFrame, &frameFinished, packet);
// if (packet->dts == AV_NOPTS_VALUE && pFrame->opaque && *(uint64_t*)pFrame->opaque != AV_NOPTS_VALUE) {
// pts = *(uint64_t *)pFrame->opaque;
// } else if (packet->dts != AV_NOPTS_VALUE) {
// pts = packet->dts;
// } else {
// pts = 0;
// }
pts=av_frame_get_best_effort_timestamp(pFrame);//Obtain the image frame display sequence number PTS(int64_t) in the encoded data packet and temporarily save it in pts(double)
/*-------------------------
* Calculate the display timestamp of the current image frame in the decoding thread function
* 1,Obtain the image frame display sequence number PTS(int64_t) in the encoded data packet and temporarily save it in pts(double)
* 2,According to PTS*time_base to calculate the display timestamp of the current frame in the whole video, that is, PTS * (1 / frame)
* av_q2d Convert AVRatioal structure into double function,
* It is used to calculate the display time (1 / frame) of each image frame of the video source, that is, return (time_base - > num / time_base - > DEN)
-------------------------*/
//According to pts=PTS*time_base={numerator=1,denominator=25} calculates the display timestamp of the current frame in the whole video
pts *= av_q2d(is->video_st->time_base);
// Did we get a video frame?
if (frameFinished) {
pts = synchronize_video(is, pFrame, pts);//Check the display timestamp pts of the current frame and update the internal video playback timer (record the video_clock)
if (queue_picture(is, pFrame, pts) < 0) {//The decoded image frame is added to the image frame queue, and the absolute display timestamp pts of the image frame is recorded
break;
}
}
av_packet_unref(packet);//Release encoded data saved in pkt
}
av_free(pFrame);//Clear memory space in pFrame
return 0;
}
/*---------------------------
* Update the internal video playback timer (record the video_clock)
* @is: Global state parameter set
* @src_frame: Current (input) (to be updated) image frame object
* @pts: Display timestamp of the current image frame
---------------------------*/
double synchronize_video(VideoState *is, AVFrame *src_frame, double pts) {
/*----------Check display timestamp----------*/
if (pts != 0) {//Check whether the display timestamp is valid
// If we have pts, set video clock to it.
is->video_clock = pts;//Update played time with display timestamp
} else {//If the display timestamp cannot be obtained
// If we aren't given a pts, set it to the clock.
pts = is->video_clock;//Update the display timestamp with the played time
}
/*--------Update video broadcast time--------*/
// Update the video clock. If the frame is to be displayed repeatedly (depending on repeat_pict), the global video playback timing video_clock shall be added with the number of repeated displays * frame rate
double frame_delay=av_q2d(is->video_st->codec->time_base);//This frame shows the time it will take to finish
// If we are repeating a frame, adjust clock appropriately, if there is a duplicate frame, arrange to render the duplicate frame between the two images before and after normal playback
frame_delay+=src_frame->repeat_pict*(frame_delay*0.5);//Calculates the time value for rendering duplicate frames (similar to note time value)
is->video_clock+=frame_delay;//Update video playback timing
return pts;//At this time, the returned value is the timestamp to be displayed at the beginning of the next frame
}
/*---------------------------
* return the wanted number of samples to get better sync if sync_type is video or external master clock
* Usually, audio or system clock will be the main synchronization source, and video will be the main synchronization source only when audio or system clock fails
* This function compares the time difference between the audio clock and the main synchronization source, and dynamically loses (or interpolates) some audio data to reduce (or increase) the audio playback time and reduce the time difference with the main synchronization source
* This function performs frame loss (or interpolation) on audio cache data and returns the length of audio data after frame loss (or interpolation)
* Because audio synchronization may bring side effects such as discontinuous output sound, this function restricts the audio synchronization process by audio_diff_avg_count and avg_diff
---------------------------*/
int synchronize_audio(VideoState *is, short *samples, int samples_size, double pts) {
double ref_clock;//Master synchronization source (reference clock)
int pcm_bytes=is->audio_st->codec->channels*2;//Number of audio data bytes per group = number of channels * number of data bytes per channel
/* if not master, then we try to remove or add samples to correct the clock */
if (is->av_sync_type != AV_SYNC_AUDIO_MASTER) {//Check the main synchronization source. If the synchronization source is not an audio clock, execute the following code
double diff, avg_diff;//diff - time difference between audio frame playback and main synchronization source, avg_diff - average value of asynchronous sampling
int wanted_size, min_size, max_size;//Cache length after frame loss (or interpolation), Max / min value of cache length
ref_clock = get_master_clock(is);//Obtain the current primary synchronization source and take the primary synchronization source as the base time (combined with the get_video_clock Implementation)
diff = get_audio_clock(is) - ref_clock;//Calculate the time difference between the audio clock and the current main synchronization source
if (diff<AV_NOSYNC_THRESHOLD) {//Check whether the audio is out of sync (limit the length of discarded audio data through AV_NOSYNC_THRESHOLD to avoid sound discontinuity)
//Accumulate the diffs, weighted accumulation of TDOA (the TDOA weight coefficient close to the current playback time is large)
is->audio_diff_cum=diff+is->audio_diff_avg_coef*is->audio_diff_cum;
if (is->audio_diff_avg_count<AUDIO_DIFF_AVG_NB) {//Compare the audio out of sync count to the threshold
//not enough measures to have a correct estimate
is->audio_diff_avg_count++;//Audio out of sync count update
} else {//When the number of audio out of synchronization exceeds the threshold limit, the audio synchronization operation is triggered
avg_diff=is->audio_diff_cum*(1.0-is->audio_diff_avg_coef);//Calculate the mean time difference (geometric mean of proportional series)
if (fabs(avg_diff)>=is->audio_diff_threshold) {//Compare TDOA mean and TDOA threshold
wanted_size=samples_size+((int)(diff*is->audio_st->codec->sample_rate)*pcm_bytes);//Convert the cache length after synchronization according to the time difference
min_size=samples_size*((100-SAMPLE_CORRECTION_PERCENT_MAX)/100);//Minimum cache length after synchronization
max_size=samples_size*((100+SAMPLE_CORRECTION_PERCENT_MAX)/100);//Maximum cache length after synchronization
if (wanted_size<min_size) {//If the cache length after synchronization is less than the minimum cache length
wanted_size=min_size;//The minimum cache length is used as the cache length after synchronization
} else if (wanted_size>max_size) {//If cache length after synchronization > minimum cache length
wanted_size=max_size;//The maximum cache length is used as the cache length after synchronization
}
if (wanted_size<samples_size) {//Compare the audio cache data length after synchronization with the original cache length
samples_size=wanted_size;//Remove samples to update the original cache length with the audio cache length after frame loss
} else if (wanted_size>samples_size) {//If the cache length after synchronization is greater than the current cache length
//Add samples by copying final sample.
//int nb= (samples_size - wanted_size);
int nb=wanted_size-samples_size;//Calculate the difference between the interpolated cache length and the original cache length (the number of audio data groups to be interpolated)
uint8_t *samples_end=(uint8_t*)samples+samples_size-pcm_bytes;//Get cache end data pointer
uint8_t *q=samples_end+pcm_bytes;//Initial interpolation position | < ----- samples ---- > | Q|
while (nb>0) {//Check the number of interpolated audio groups (each group includes pcm data of two channels)
memcpy(q, samples_end, pcm_bytes);//Append interpolation after samples original cache
q += pcm_bytes;//Update interpolation position
nb -= pcm_bytes;//Update the number of interpolation groups
}
samples_size = wanted_size;//Returns the cache length after audio synchronization
}
}
}
} else {
// Difference is too big, reset diff stuff
is->audio_diff_avg_count = 0;//Audio out of sync count reset
is->audio_diff_cum = 0;//Audio TDOA cumulative reset
}
}//end for if (is->av_sync_type != AV_SYNC_AUDIO_MASTER)
return samples_size;//Returns the cache length after audio synchronization
}
/*-----------Get audio clock-----------
* That is, the pts of the currently playing audio data is obtained, and the audio clock is used as the audio and video synchronization reference
* The principle of audio-video synchronization is to control the playback of video according to the pts of audio
* That is, whether and how long the video is displayed after decoding a frame is obtained by comparing the PTS of the frame with the PTS of the audio being played at the same time
* If the PTS of audio is large, the video display is completed and the decoding display of the next frame is prepared, otherwise wait
*
* Because pcm data adopts audio_ Playback by callback
* For audio playback, we can only get the pts of the cached audio frame before writing the callback function, but we can't get the pts of the current playback frame (we need to use the pts of the current playback audio frame as the reference clock)
* Considering that the size of the audio is proportional to the playback time (the same sampling rate), the audio frame PTS (located in the callback function cache) being played at the current time
* It can be calculated according to the length of pcm data sent into the sound card, the length of remaining pcm data in the cache, the cache length and the sampling rate
--------------------------------*/
double get_audio_clock(VideoState *is) {
double pts= is->audio_clock;//Maintained in the audio thread to obtain the timestamp when the decoding operation is completed
//The length of the remaining original audio data that has not been played (into the sound card) is equal to the length of the decoded multi frame original audio data - the length of the cumulative write to the sound card cache (into the sound card)
int hw_buf_size=is->audio_buf_size - is->audio_buf_index;
int bytes_per_sec=0;//Raw audio bytes per second
int pcm_bytes=is->audio_st->codec->channels*2;//Original audio data byte book per group = number of channels * number of data bytes per channel
if (is->audio_st) {
bytes_per_sec=is->audio_st->codec->sample_rate*pcm_bytes;//Calculates the number of raw audio bytes per second
}
if (bytes_per_sec) {//Check that the number of original audio bytes per second is valid
pts-=(double)hw_buf_size/bytes_per_sec;//According to the index position sent into the sound card cache, the audio playback timestamp pts at the current time is calculated backward
}
return pts;//Returns the audio playback timestamp
}
/*-----------Get video clock-----------
* That is, get the pts of the currently playing video frame, take the video clock pts as the audio-video synchronization benchmark, and return the current time offset of the video currently being played
* This value is the current frame timestamp pts + a slight correction value delta
* Because at the ms level, at the ms level, if the time of obtaining the video clock (i.e. the current frame pts) is delayed from the time of calling the video clock (e.g. synchronizing the audio to the video pts)
* Then, the video clock needs to be corrected when called, and the correction value delta is
* delta=[Get the time value of the video clock_ current_ pts_ Time] to [call get_video_clock time value]
* Usually, the external clock or audio clock will be selected as the main synchronization source, and the video synchronization to audio or external clock will be the preferred synchronization scheme
* The synchronization scheme with video clock as the main synchronization source belongs to three basic synchronization schemes (synchronization to audio, synchronization to video and synchronization to external clock)
* The cost is only to show the method of synchronizing to the video clock. Generally, synchronizing to the video clock is only used as an auxiliary synchronization scheme
--------------------------------*/
double get_video_clock(VideoState *is) {
double delta=(av_gettime()-is->video_current_pts_time)/1000000.0;
return is->video_current_pts+delta;//pts_of_last_frame+(Current_time-time_elapsed_since_pts_value_was_set)
}
//Obtain the system time and take the system clock as the synchronization reference
double get_external_clock(VideoState *is) {
return av_gettime()/1000000.0;//Obtain the current time of the system in 1 / 1000000 second, which is convenient for transplantation on various platforms
}
//Get master clock (reference clock)
double get_master_clock(VideoState *is) {
if (is->av_sync_type == AV_SYNC_VIDEO_MASTER) {//Check primary sync source type
return get_video_clock(is);//Return video clock
} else if (is->av_sync_type == AV_SYNC_AUDIO_MASTER) {
return get_audio_clock(is);//Return audio clock
} else {
return get_external_clock(is);//Return system clock
}
}
//Timer triggered callback function
static Uint32 sdl_refresh_timer_cb(Uint32 interval, void *opaque) {
SDL_Event event;//SDL event object
event.type = FF_REFRESH_EVENT;//Video display refresh event
event.user.data1 = opaque;//Transfer user data
SDL_PushEvent(&event);//Send event
return 0; // 0 means stop timer.
}
/*---------------------------
* Schedule a video refresh in 'delay' ms.
* Set the delay of frame playback
* Tell the system to push an FF after the specified delay_ REFRESH_ Event event, which acts like a metronome
* This event will trigger the SDL in the event queue_ refresh_ timer_ Call of CB function
* @delay It is used to adjust the rendering timing of the next frame when the decoding order of the image frame is inconsistent with the rendering order
* So as to make all image frames render and refresh at a fixed frame rate as much as possible
--------------------------*/
static void schedule_refresh(VideoState *is, int delay) {
SDL_AddTimer(delay, sdl_refresh_timer_cb, is);//Callback the user specified function after the specified time (ms)
}
/*---------------------------
* Display refresh function (FF_REFRESH_EVENT response function)
* Synchronize the video to the master clock and calculate the delay time of the next frame
* The PTS difference between the current frame and the previous frame is used to estimate the delay time of playing the next frame, and the delay time is adjusted according to the playback speed of video
---------------------------*/
void video_refresh_timer(void *userdata) {
VideoState *is = (VideoState *)userdata;//Transfer user data
VideoPicture *vp;//Image frame object
//delay - the time interval between the two frames before and after display ([picture picture] time difference), diff - the time difference between the image frame display and the audio frame playback
//sync_threshold - [picture picture] minimum time difference, actual_delay - display time interval of current frame - next frame (dynamic time, real time, absolute time)
double delay,diff,sync_threshold,actual_delay,ref_clock;
if (is->video_st) {//Check whether the video stream information structure in the global status parameter set is valid (whether the video file has been loaded)
if (is->pictq_size == 0) {//Check whether there are images waiting for display refresh in the image frame queue
schedule_refresh(is, 1);//If the queue is empty, send a display refresh event and enter video again_ refresh_ Timer function
} else {
vp = &is->pictq[is->pictq_rindex];//Obtain the image frame waiting to be displayed from the display queue
is->video_current_pts = vp->pts;//Gets the display timestamp of the current frame
is->video_current_pts_time = av_gettime();//Obtain the system time as the time reference for playing the current frame
//Calculate the interval between the current frame and the previous frame display (pts) (the difference between the display timestamp)
delay = vp->pts - is->frame_last_pts;//The pts from last time
if (delay <= 0 || delay >= 1.0) {//Check whether the time interval is within a reasonable range
// If incorrect delay, use previous one.
delay = is->frame_last_delay;//Follow the previous dynamic refresh interval
}
// Save for next time.
is->frame_last_delay = delay;//Dynamic refresh delay time for saving [previous frame image]
is->frame_last_pts = vp->pts;//Save the display timestamp of [previous frame image]
// Update delay to sync to audio to obtain the sound playback time stamp (as the reference time for video synchronization)
if (is->av_sync_type != AV_SYNC_VIDEO_MASTER) {//Check the main synchronization clock source
ref_clock = get_master_clock(is);//Judge the speed of Video playback according to the master clock, and take the master clock as the reference time
diff = vp->pts - ref_clock;//Calculate the time difference between the image frame display and the master clock
//Adjust the delay time of playing the next frame according to the time difference to synchronize Skip or repeat the frame, Take delay into account
sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD;
//Judge the audio and video synchronization conditions, i.e. [picture sound] time difference & [picture picture] time difference < 10ms threshold. If > this threshold, it is fast forward mode, and there is no audio and video synchronization problem
if (fabs(diff) < AV_NOSYNC_THRESHOLD) {
if (diff <= -sync_threshold) {//Slow down, set delay to 0 and display as soon as possible
//If the display time of the next frame is close to the current sound, speed up the display of the next frame (that is, after the video_display displays the current frame, turn on the timer to display the next frame quickly
delay=0;
} else if (diff>=sync_threshold) {//Come on, double delay
delay=2*delay;
}
}//If diff (significantly) is greater than AV_NOSYNC_THRESHOLD, i.e. fast forward mode, the picture beats too much, so there is no problem of audio and video synchronization
}
//Update the played time value when the video is played to the current frame (cumulative time value of dynamic playback of all image frames - real value), frame_timer always accumulates the delay we calculated during playback
is->frame_timer+=delay;
//Frame per calculation_ The difference between timer and system time (taking system time as the base time) will be frame_timer the purpose associated with the system time (absolute time)
actual_delay=is->frame_timer-(av_gettime()/1000000.0);//Computer the REAL delay
if (actual_delay < 0.010) {//Check absolute time range
actual_delay = 0.010;// Really it should skip the picture instead
}
schedule_refresh(is,(int)(actual_delay*1000+0.5));//Use the absolute time timer to dynamically display and refresh the next frame
video_display(is);//Refresh the image, Show the picture
// update queue for next picture!.
if (++is->pictq_rindex==VIDEO_PICTURE_QUEUE_SIZE) {//Update image frame queue read index position
is->pictq_rindex = 0;//If the read index reaches the end of the queue, reset the read index position
}
SDL_LockMutex(is->pictq_lock);//Lock the mutex to protect the pixel data of the canvas
is->pictq_size--;//Update image frame queue length
SDL_CondSignal(is->pictq_ready);//Send queue ready signal
SDL_UnlockMutex(is->pictq_lock);//Release mutex
}
} else {//If the video information acquisition fails, try to refresh the view again after the specified delay (100ms)
schedule_refresh(is,100);
}
}
//Set [fast forward] / [fast backward] status parameters
void stream_seek(VideoState *is, int64_t pos, int rel) {
if (!is->seek_req) {//Check whether the [fast forward] / [fast reverse] operation flag is on
is->seek_pos = pos;//Update reference timestamp after [fast forward] / [fast back]
is->seek_flags = rel < 0 ? AVSEEK_FLAG_BACKWARD : 0;//Determine [fast forward] or [fast backward] operation
is->seek_req = 1;//Turn on the [fast forward] / [fast backward] flag bit
}
}
//When the audio data is not in 16 bit sampling format, decode is used_ frame_ from_ Packet calculates the length of decoded data
int decode_frame_from_packet(VideoState *is, AVFrame decoded_frame) {
if (decoded_frame.channel_layout == 0) {
decoded_frame.channel_layout = av_get_default_channel_layout(decoded_frame.channels);
}
int src_nb_samples = decoded_frame.nb_samples;//Number of pcm contained in a frame of data
int src_linesize = (int) decoded_frame.linesize;//Scan line data length
uint8_t **src_data = decoded_frame.data;//Decoded raw data cache pointer
int src_rate = decoded_frame.sample_rate;//sampling rate
int dst_rate = decoded_frame.sample_rate;
int64_t src_ch_layout = decoded_frame.channel_layout;
int64_t dst_ch_layout = decoded_frame.channel_layout;
enum AVSampleFormat src_sample_fmt = decoded_frame.format;
enum AVSampleFormat dst_sample_fmt = AV_SAMPLE_FMT_S16;
av_opt_set_int(is->sws_ctx_audio, "in_channel_layout", src_ch_layout, 0);
av_opt_set_int(is->sws_ctx_audio, "out_channel_layout", dst_ch_layout, 0);
av_opt_set_int(is->sws_ctx_audio, "in_sample_rate", src_rate, 0);
av_opt_set_int(is->sws_ctx_audio, "out_sample_rate", dst_rate, 0);
av_opt_set_sample_fmt(is->sws_ctx_audio, "in_sample_fmt", src_sample_fmt, 0);
av_opt_set_sample_fmt(is->sws_ctx_audio, "out_sample_fmt", dst_sample_fmt, 0);
int ret;//Return results
// Initialize the resampling context.
if ((ret = swr_init((struct SwrContext *) is->sws_ctx_audio)) < 0) {
fprintf(stderr, "Failed to initialize the resampling context\n");
return -1;
}
// Allocate source and destination samples buffers.
int src_nb_channels=av_get_channel_layout_nb_channels(src_ch_layout);
ret=av_samples_alloc_array_and_samples(&src_data,&src_linesize,src_nb_channels,src_nb_samples,src_sample_fmt,0);
if (ret < 0) {
fprintf(stderr, "Could not allocate source samples\n");
return -1;
}
//Compute the number of converted samples: buffering is avoided ensuring that the output buffer will contain at least all the converted input samples.
int dst_nb_samples = av_rescale_rnd(src_nb_samples, dst_rate, src_rate, AV_ROUND_UP);
int max_dst_nb_samples = dst_nb_samples;
int dst_linesize;
uint8_t **dst_data = NULL;
// Buffer is going to be directly written to a rawaudio file, no alignment.
int dst_nb_channels = av_get_channel_layout_nb_channels(dst_ch_layout);
ret=av_samples_alloc_array_and_samples(&dst_data,&dst_linesize,dst_nb_channels,dst_nb_samples,dst_sample_fmt,0);
if (ret < 0) {
fprintf(stderr, "Could not allocate destination samples\n");
return -1;
}
//Compute destination number of samples.
dst_nb_samples=av_rescale_rnd(swr_get_delay((struct SwrContext*)is->sws_ctx_audio,src_rate)+src_nb_samples,dst_rate,src_rate,AV_ROUND_UP);
//Convert to destination format.
ret=swr_convert((struct SwrContext*)is->sws_ctx_audio,dst_data,dst_nb_samples,(const uint8_t **)decoded_frame.data,src_nb_samples);
if (ret<0) {
fprintf(stderr, "Error while converting\n");
return -1;
}
int dst_bufsize = av_samples_get_buffer_size(&dst_linesize, dst_nb_channels, ret, dst_sample_fmt, 1);
if (dst_bufsize < 0) {
fprintf(stderr, "Could not get sample buffer size\n");
return -1;
}
memcpy(is->audio_buf, dst_data[0], dst_bufsize);
if (src_data) {
av_freep(&src_data[0]);
}
av_freep(&src_data);
if (dst_data) {
av_freep(&dst_data[0]);
}
av_freep(&dst_data);
return dst_bufsize;
}
// These are called whenever we allocate a frame buffer. We use this to store the global_pts in a frame at the time it is allocated.
int our_get_buffer(struct AVCodecContext *c, AVFrame *pic, int flags) {
int ret = avcodec_default_get_buffer2(c, pic, 0);
uint64_t *pts = av_malloc(sizeof(uint64_t));
*pts = global_video_pkt_pts;
pic->opaque = pts;
return ret;
}
int main(int argc, char *argv[]) {
if (argc < 2) {//Check whether the number of input parameters is correct
fprintf(stderr, "Usage: test <file>\n");
exit(1);
}
av_register_all();// Register all formats and codecs to register all multimedia formats and codecs
VideoState *is=av_mallocz(sizeof(VideoState));//Create global state object
av_strlcpy(is->filename, argv[1], 1024);//Copy video file pathname
is->pictq_lock = SDL_CreateMutex();//Create encoded packet queue mutex object
is->pictq_ready = SDL_CreateCond();//Create encoded packet queue readiness condition object
//SDL_ Initialize the event handling, file I / O, and threading subsystems to initialize SDL
if (SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER)) {
fprintf(stderr, "Could not initialize SDL - %s\n", SDL_GetError());//initialize the video audio & timer subsystem
exit(1);
}
// Make a screen to put our video in sdl2 SDL in 0_ Setvideomode and SDL_Overlay has been deprecated and changed to SDL_CreateWindow and SDL_CreateRenderer creates windows and shaders
#ifndef __DARWIN__
screen = SDL_SetVideoMode(640, 480, 0, 0);//Create SDL window and drawing surface, and specify image size and pixel format
#else
screen = SDL_SetVideoMode(640, 480, 24, 0);//Create SDL window and drawing surface, and specify image size and pixel format
#endif
if (!screen) {//Check whether the SDL (drawing surface) window is created successfully (SDL operates the window with drawing surface objects)
fprintf(stderr, "SDL: could not set video mode - exiting\n");
exit(1);
}
schedule_refresh(is, 40);//After the specified time (40ms), call back the function specified by the user to update the display of image frames
is->av_sync_type = DEFAULT_AV_SYNC_TYPE;//Specify primary synchronization source
is->parse_tid = SDL_CreateThread(parse_thread ,is);//Create encoded packet parsing thread
if (!is->parse_tid) {//Check whether the thread was created successfully
av_free(is);
return -1;
}
av_init_packet(&flush_pkt);//Initialize flush_pkt
//FLUSH_ The data member of pkt is specified as "FLUSH". When the value of the data member of a package in the packet queue is "FLUSH", the decoder reset operation is performed
flush_pkt.data = (unsigned char *) "FLUSH";
/*-----------------------
* SDL Event (message) loop
* The player triggers (drives) the image frame rendering operation repeatedly through the message circulation mechanism to complete the rendering of the whole video file
* The whole process is similar to playing the piano according to the specified beat. The message circulation mechanism ensures that the video is played according to the fixed beat (6 / 8)
* Because the decoding order is inconsistent with the rendering order (decoding B frames), the video synchronization mechanism ensures that the images are played at a fixed beat after decoding
* In each message response function video_refresh_timer, recalculate the display time of the next frame
* And through schedule_refresh specifies the time (similar to the action of metronome) to trigger the next round of image frame display
-----------------------*/
SDL_Event event;//SDL event (message) object
for (;;) {
double incr, pos;
SDL_WaitEvent(&event);//Use this function to wait indefinitely for the next available event
switch (event.type) {//After the event arrives, wake up the main thread, check the event type and perform corresponding operations
case SDL_KEYDOWN://Check keyboard operation events
switch (event.key.keysym.sym) {//Check the keyboard operation type, which key get hit
case SDLK_LEFT://[left key]
incr = -10.0;//Rewind for 10s
goto do_seek;
case SDLK_RIGHT://[right click]
incr = 10.0;//Fast forward 10s
goto do_seek;
case SDLK_UP://[up]
incr = 60.0;//Fast forward 60s
goto do_seek;
case SDLK_DOWN://[down]
incr = -60.0;//Fast backward 60s
goto do_seek;
do_seek://Processing requests
if (global_video_state) {
pos = get_master_clock(global_video_state);//Gets the timestamp of the current primary synchronization source
pos += incr;//Update the main synchronization source timestamp according to the keyboard operation (AV_TIME_BASE is the timestamp reference value)
stream_seek(global_video_state,(int64_t)(pos*AV_TIME_BASE),incr);//Set the lookup location based on the primary synchronization source timestamp
}
break;
default:
break;
}
break;
case FF_QUIT_EVENT:
case SDL_QUIT://Exit process event
is->quit = 1;
// If the video has finished playing, then both the picture and audio queues are waiting for more data. Make them stop waiting and terminate normally.
SDL_CondSignal(is->audioq.qready);//Signal queue ready to avoid deadlock
SDL_CondSignal(is->videoq.qready);
SDL_Quit();
exit(0);
break;
case FF_ALLOC_EVENT://Assign overlay events
alloc_picture(event.user.data1);//Assign overlay event response function
break;
case FF_REFRESH_EVENT://Video display refresh event
video_refresh_timer(event.user.data1);//Video display refresh event response function
break;
default:
break;
}
}
return 0;
}
//The copyright belongs to the author. For commercial reprint, please contact the author for authorization, and for non-commercial reprint, please indicate the source.
/ / official account: breakpoint Laboratory
//Scan QR code and pay attention to more high-quality original works, including audio and video development, image processing, network
//Linux, Windows, Android, embedded development, etc
