Audio-Video Synchronization - Main Code Logic
That's how you chase me
Because the audio format sample channel rate is fixed, an audio time can be determined, so an audio is synchronized to the audio as the base video
General practice: after displaying a video frame, get the PTS of the next video frame, and compare it with the clock lock of the current audio. Play it immediately before, set a delay later, and set the timer to refresh the video frame when the timer times out.
av_frame_get_best_effort_timestamp(pFrame)) You can get a direct correspondence AvFrame Get a good fit pts pts To convert to seconds, multiply by time_base Then pair pts Make a calculation if it can be used directly is->video_clock = pts That's OK
Main Code Logic for Audio-Video Synchronization
Add 5 parameters to the main structure
audio_clock current audio frame time
video_clock Time of Next Frame of Current Video Frame
frame_timer The timer that timer will call next time
frame_last_pts of last frame in pts video
frame_last_delay last frame video play delay
double audio_clock; double video_clock; ///<pts of last decoded frame / predicted pts of next decoded frame double frame_timer; //Time of Video Frame Callback double frame_last_pts; double frame_last_delay;
The simple logic is!
1 Get the delay between the current true and the previous frame
delay = vp->pts - is->frame_last_pts
2 Judge delay and replace previous step
delay = vp->pts - is->frame_last_pts; /* the pts from last time */ if(delay <= 0 || delay >= 1.0) { delay = is->frame_last_delay; } is->frame_last_delay = delay; is->frame_last_pts = vp->pts;
3Get the PTS of the current audio frame and calculate the diff between them for the next frame (that is, the current processing frame) of the playing frame
ref_clock = get_audio_clock(is); diff = vp->pts - ref_clock;
4 sync_ Thresholds cannot make frames smaller than the minimum frame rate of audio
If (fabs (diff) < AV_ NOSYNC_ THRESHOLD) Synchronize if the previously obtained distance meets the minimum frame processing requirements
If diff <= -sync_ Thresholds then explain the current frame time and the delay setting 0 to play fast
If diff >= sync_ Throwold says there's still some time to play before delay grows
sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD; if(fabs(diff) < AV_NOSYNC_THRESHOLD) { if(diff <= -sync_threshold) { delay = 0; } else if(diff >= sync_threshold) { delay = 2 * delay; }
5 Add the above scheduled delay time to the current callback time
is->frame_timer += delay;
6 Calculate the delay in real time to ensure the minimum is no less than the frame rate of the audio frame
The last pass in actually takes time to refresh the video rendering + 0.5 is plus a paranoia
actual_delay = is->frame_timer - (av_gettime() / 1000000.0); if(actual_delay < 0.010) { /* Really it should skip the picture instead */ actual_delay = 0.010; } schedule_refresh(is, (int)(actual_delay * 1000 + 0.5));
void video_refresh_timer(void *userdata) { VideoState *is = (VideoState *)userdata; VideoPicture *vp; double actual_delay, delay, sync_threshold, ref_clock, diff; if(is->video_st) { if(is->pictq_size == 0) { schedule_refresh(is, 1); } else { vp = &is->pictq[is->pictq_rindex]; delay = vp->pts - is->frame_last_pts; /* the pts from last time */ if(delay <= 0 || delay >= 1.0) { /* if incorrect delay, use previous one */ delay = is->frame_last_delay; } /* save for next time */ is->frame_last_delay = delay; is->frame_last_pts = vp->pts; /* update delay to sync to audio */ ref_clock = get_audio_clock(is); diff = vp->pts - ref_clock; /* Skip or repeat the frame. Take delay into account FFPlay still doesn't "know if this is the best guess." */ sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD; if(fabs(diff) < AV_NOSYNC_THRESHOLD) { if(diff <= -sync_threshold) { delay = 0; } else if(diff >= sync_threshold) { delay = 2 * delay; } } is->frame_timer += delay; /* computer the REAL delay */ actual_delay = is->frame_timer - (av_gettime() / 1000000.0); if(actual_delay < 0.010) { /* Really it should skip the picture instead */ actual_delay = 0.010; } schedule_refresh(is, (int)(actual_delay * 1000 + 0.5)); /* show the picture! */ video_display(is); /* update queue for next picture! */ if(++is->pictq_rindex == VIDEO_PICTURE_QUEUE_SIZE) { is->pictq_rindex = 0; } SDL_LockMutex(is->pictq_mutex); is->pictq_size--; SDL_CondSignal(is->pictq_cond); SDL_UnlockMutex(is->pictq_mutex); } } else { schedule_refresh(is, 100); } }