Audio-Video Synchronization - Main Code Logic

That's how you chase me

Because the audio format sample channel rate is fixed, an audio time can be determined, so an audio is synchronized to the audio as the base video

General practice: after displaying a video frame, get the PTS of the next video frame, and compare it with the clock lock of the current audio. Play it immediately before, set a delay later, and set the timer to refresh the video frame when the timer times out.

 av_frame_get_best_effort_timestamp(pFrame)) You can get a direct correspondence AvFrame Get a good fit pts
 pts To convert to seconds, multiply by time_base 
 Then pair pts Make a calculation if it can be used directly is->video_clock = pts That's OK

Main Code Logic for Audio-Video Synchronization

Add 5 parameters to the main structure
audio_clock current audio frame time
video_clock Time of Next Frame of Current Video Frame
frame_timer The timer that timer will call next time
frame_last_pts of last frame in pts video
frame_last_delay last frame video play delay

  double          audio_clock;
  double          video_clock; ///<pts of last decoded frame / predicted pts of next decoded frame

  double          frame_timer;  //Time of Video Frame Callback
  double          frame_last_pts;
  double          frame_last_delay;

The simple logic is!
1 Get the delay between the current true and the previous frame

delay = vp->pts - is->frame_last_pts 

2 Judge delay and replace previous step

delay = vp->pts - is->frame_last_pts; /* the pts from last time */
  if(delay <= 0 || delay >= 1.0) {
	delay = is->frame_last_delay;
  }
  is->frame_last_delay = delay;
  is->frame_last_pts = vp->pts;

3Get the PTS of the current audio frame and calculate the diff between them for the next frame (that is, the current processing frame) of the playing frame

ref_clock = get_audio_clock(is);
  diff = vp->pts - ref_clock;

4 sync_ Thresholds cannot make frames smaller than the minimum frame rate of audio
If (fabs (diff) < AV_ NOSYNC_ THRESHOLD) Synchronize if the previously obtained distance meets the minimum frame processing requirements
If diff <= -sync_ Thresholds then explain the current frame time and the delay setting 0 to play fast
If diff >= sync_ Throwold says there's still some time to play before delay grows

 sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD;
  if(fabs(diff) < AV_NOSYNC_THRESHOLD) {
	if(diff <= -sync_threshold) {
  delay = 0;
	} else if(diff >= sync_threshold) {
	  delay = 2 * delay;
	}

5 Add the above scheduled delay time to the current callback time

is->frame_timer += delay;

6 Calculate the delay in real time to ensure the minimum is no less than the frame rate of the audio frame
The last pass in actually takes time to refresh the video rendering + 0.5 is plus a paranoia

actual_delay = is->frame_timer - (av_gettime() / 1000000.0);
  if(actual_delay < 0.010) {
/* Really it should skip the picture instead */
actual_delay = 0.010;
  }
  schedule_refresh(is, (int)(actual_delay * 1000 + 0.5));

void video_refresh_timer(void *userdata) {

  VideoState *is = (VideoState *)userdata;
  VideoPicture *vp;
  double actual_delay, delay, sync_threshold, ref_clock, diff;
  
  if(is->video_st) {
    if(is->pictq_size == 0) {
      schedule_refresh(is, 1);
    } else {
      vp = &is->pictq[is->pictq_rindex];

      delay = vp->pts - is->frame_last_pts; /* the pts from last time */
      if(delay <= 0 || delay >= 1.0) {
	/* if incorrect delay, use previous one */
	delay = is->frame_last_delay;
      }
      /* save for next time */
      is->frame_last_delay = delay;
      is->frame_last_pts = vp->pts;

      /* update delay to sync to audio */
      ref_clock = get_audio_clock(is);
      diff = vp->pts - ref_clock;

      /* Skip or repeat the frame. Take delay into account
	 FFPlay still doesn't "know if this is the best guess." */
      sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay : AV_SYNC_THRESHOLD;
      if(fabs(diff) < AV_NOSYNC_THRESHOLD) {
	if(diff <= -sync_threshold) {
	  delay = 0;
	} else if(diff >= sync_threshold) {
	  delay = 2 * delay;
	}
      }
      is->frame_timer += delay;
      /* computer the REAL delay */
      actual_delay = is->frame_timer - (av_gettime() / 1000000.0);
      if(actual_delay < 0.010) {
	/* Really it should skip the picture instead */
	actual_delay = 0.010;
      }
      schedule_refresh(is, (int)(actual_delay * 1000 + 0.5));
      
      /* show the picture! */
      video_display(is);
      
      /* update queue for next picture! */
      if(++is->pictq_rindex == VIDEO_PICTURE_QUEUE_SIZE) {
	is->pictq_rindex = 0;
      }
      SDL_LockMutex(is->pictq_mutex);
      is->pictq_size--;
      SDL_CondSignal(is->pictq_cond);
      SDL_UnlockMutex(is->pictq_mutex);
    }
  } else {
    schedule_refresh(is, 100);
  }
}