Android mediasoup webrtc h264 soft codec related source code analysis

Starting address of this article https://blog.csdn.net/CSqingchen/article/details/120218923
Latest update address https://gitee.com/chenjim/chenjimblog
Based on libmediasoupclient 3.2.0 And webrtc branch heads / 4147 (m84)
This article must be familiar with the relevant foundation and reference Wen 1 and Wen 2

mediasoup H264 support

Open rtc_use_h264
In webrtc.gni, you can see that Android does not support h264 by default
rtc_use_h264 = proprietary_codecs && !is_android && !is_ios && !(is_win && !is_clang)
You can change this to true or bring the compilation parameter 'rtc_use_h264=true, as follows:
./tools_webrtc/android/build_aar.py --extra-gn-args 'rtc_use_h264=true'
mediasoup-demo-android forceH264
From the code, we can see that this parameter is only used in UrlFactory.java
Normally, if forceH264=true in the url, H264 encoding should be used
Such as browser input https://v3demo.mediasoup.org/?forceH264=true&roomId=123456 The output video is H264
(Android displays a black screen by default and lacks H264 decoder, which will be solved later)
But if forceH264 is checked in the mediasoup demo Android menu, it's still VP8 coding. It's a BUG

Android webrtc video capture process source code analysis

webrtc mainly provides the videocamera interface for video capture. The implementation classes include screencaptureandroid, filevideocamera and cameracapture, which represent three different video sources: screen, file and camera. Because the android system successively provides camera1.0 and camera2.0 interfaces, Therefore, cameracapture is represented by camera1capter and camera2capter respectively.
Main class diagram

The flow chart of video acquisition and distribution is as follows:

For more details, please refer to the original text webrtc source code analysis of one of the video collection , thank you Jimmy2012

Why Android H264 doesn't use hard coding

Creation of mediasoup audio and video codec
Handler::GetNativeRtpCapabilities (libmediasoupclient/src/Device.cpp)
–> std::unique_ptr pc(new PeerConnection(privateListener.get(), peerConnectionOptions))
(libmediasoupclient/src/Handler.cpp)
–> webrtc::CreateBuiltinVideoEncoderFactory (deps/libmediasoupclient/src/PeerConnection.cpp)
–> webrtc::CreatePeerConnectionFactory
–> cricket::CreateMediaEngine(std::move(media_dependencies)) (webrtc/src/api/create_peerconnection_factory.cc)
–> CreateMediaEngine (webrtc/src/media/engine/webrtc_media_engine.cc)
–> WebRtcVideoEngine::WebRtcVideoEngine (webrtc/src/media/engine/webrtc_media_engine.cc)

From libmediasoupclient/src/PeerConnection.cpp, we can see the creation factory of audio and video codec

  this->peerConnectionFactory = webrtc::CreatePeerConnectionFactory(
    this->networkThread.get(),
    this->workerThread.get(),
    this->signalingThread.get(),
    nullptr /*default_adm*/,
    webrtc::CreateBuiltinAudioEncoderFactory(),
    webrtc::CreateBuiltinAudioDecoderFactory(),
    webrtc::CreateBuiltinVideoEncoderFactory(),
    webrtc::CreateBuiltinVideoDecoderFactory(),
    nullptr /*audio_mixer*/,
    nullptr /*audio_processing*/);

Mediasoup demo Android \ app \ SRC \ main \ Java \ ORG \ mediasoup \ droid \ lib \ peerconnectionutils.java

    mPeerConnectionFactory = builder.setAudioDeviceModule(adm)
            .setVideoEncoderFactory(encoderFactory)
            .setVideoDecoderFactory(decoderFactory)
            .createPeerConnectionFactory();
``
Not really used, so far mediasoup And Android h264 Hard coding is temporarily unavailable....


- `VideoStreamEncoder` Creation of  
SetRemoteContent_w   (webrtc/src/pc/channel.cc)  
-->  media_channel()->SetSendParameters(send_params)  
-->  WebRtcVideoChannel::SetSendParameters  (webrtc/src/media/engine/webrtc_video_engine.cc)  
-->  ApplyChangedParams(changed_params) 
-->  kv.second->SetSendParameters(changed_params)   
-->  SetCodec(*params.send_codec)  
-->  RecreateWebRtcStream(...)  
-->  stream_ = call_->CreateVideoSendStream  
-->  CreateVideoSendStream  (webrtc/src/call/call.cc)  
-->  VideoSendStream* send_stream = new VideoSendStream(...)  (webrtc/src/call/call.cc)   
-->  video_stream_encoder_ = CreateVideoStreamEncoder(...)  (webrtc/src/video/video_send_stream.cc)  
-->  VideoStreamEncoder::VideoStreamEncoder(...)  (webrtc/src/video/video_stream_encoder.cc)  

----

#### Analysis of webrtc openh264 soft coding source code  

- H264Encoder establish    
[Video frame distribution](https://www.jianshu.com/p/5902d4953ed9) to videostreamecoder:: onframe (webrtc / SRC / video / video_stream_encoder. CC)   
-->  ReconfigureEncoder()  
-->  encoder_ = settings_.encoder_factory->CreateVideoEncoder(encoder_config_.video_format)    
  (webrtc/src/video/video_stream_encoder.cc)    
-->  internal_encoder = std::make_unique<EncoderSimulcastProxy>(internal_encoder_factory_.get(), format)   (webrtc\src\api\video_codecs\builtin_video_encoder_factory.cc)   
-->  EncoderSimulcastProxy::EncoderSimulcastProxy (webrtc\src\media\engine\encoder_simulcast_proxy.cc)   
-->  InternalEncoderFactory::CreateVideoEncoder (webrtc/src/media/engine/internal_encoder_factory.cc)  
-->  H264Encoder::Create()  (webrtc/src/modules/video_coding/codecs/h264/h264.cc)   
-->  H264EncoderImpl::H264EncoderImpl  (webrtc/src/modules/video_coding/codecs/h264/h264_encoder_impl.cc)  


stay `internal_encoder_factory.cc` We can see that there are V8 V9 H264 AV1 Encoder The creation of is as follows:  
```c++
std::unique_ptr<VideoEncoder> InternalEncoderFactory::CreateVideoEncoder(const SdpVideoFormat& format) {
  if (absl::EqualsIgnoreCase(format.name, cricket::kVp8CodecName))
    return VP8Encoder::Create();
  if (absl::EqualsIgnoreCase(format.name, cricket::kVp9CodecName))
    return VP9Encoder::Create(cricket::VideoCodec(format));
  if (absl::EqualsIgnoreCase(format.name, cricket::kH264CodecName))
    return H264Encoder::Create(cricket::VideoCodec(format));
  if (kIsLibaomAv1EncoderSupported &&absl::EqualsIgnoreCase(format.name, cricket::kAv1CodecName))
    return CreateLibaomAv1Encoder();
  return nullptr;
}

H264Encoder initialization
Distribute video frames to VideoStreamEncoder::OnFrame (webrtc/src/video/video_stream_encoder.cc)
ReconfigureEncoder()
–> encoder_->InitEncode(...)
–> EncoderSimulcastProxy::InitEncode(...) (webrtc/src/media/engine/encoder_simulcast_proxy.cc)
–> H264EncoderImpl::InitEncode(...) (webrtc/src/modules/video_coding/codecs/h264/h264_encoder_impl.cc)
H264EncoderImpl::SetRates and H264EncoderImpl::CreateEncoderParams will also be called
H264Encoder code
Distribute video frames to VideoStreamEncoder::OnFrame (webrtc/src/video/video_stream_encoder.cc)
–> MaybeEncodeVideoFrame
–> VideoStreamEncoder::EncodeVideoFrame
–> encoder_->Encode(...)
–> H264EncoderImpl::Encode (webrtc/src/modules/video_coding/codecs/h264/h264_encoder_impl.cc)
–> encoded_ image_ callback_-> Onencodedimage encoding completion callback

H264Decoder initialization process

VideoReceiver2::Decode (modules/video_coding/video_receiver2.cc)
–>
VCMDecoderDataBase::GetDecoder (modules/video_coding/decoder_database.cc)
–>
H264DecoderImpl::InitDecode (modules/video_coding/codecs/h264/h264_decoder_impl.cc)
–>
avcodec_find_decoder (third_party/ffmpeg/libavcodec/allcodecs.c)
Where av_codec_iterate uses codec_ List (in libavcodec/codec_list.c)
That's why Enable h264 soft codec Need to modify here

openh264 provides codec related functions

WelsCreateDecoder;
WelsCreateSVCEncoder;
WelsDestroyDecoder;
WelsDestroySVCEncoder;
WelsGetCodecVersion;
WelsGetCodecVersionEx;

Using openh264 codec example
https://blog.csdn.net/NB_vol_1/article/details/103376649

webrtc video H264 hard coding

Due to fragmentation of Android devices, early versions did not support hard coding, and there were different chip manufacturers, such as Qualcomm, MTK, Hisilicon, Samsung, etc
In the end, not all Android devices support hard codec

Modify software in mediacodeutils.java_ IMPLEMENTATION_ PREFIXES
It is suggested to remove "OMX.SEC."
Because in HardwareVideoDecoderFactory.java and platformsoftware videodecoderfactory,
MediaCodecUtils.SOFTWARE_IMPLEMENTATION_PREFIXES contains a blacklist of chip manufacturers that do not support hard coding...
The chip code prefixes supported by H264 are as follows, refer to A peach blossom presses the Begonia blog

private static final String[] supportedH264HwCodecPrefixes = {
    "OMX.qcom.", "OMX.Intel.", "OMX.Exynos."
    ,"OMX.Nvidia.H264."     /*Nexus 7(2012), Nexus 9, Tegra 3, Tegra K1*/
    ,"OMX.ittiam.video."    /*Xiaomi Mi 1s*/
    ,"OMX.SEC.avc."         /*Exynos 3110, Nexus S*/
    ,"OMX.IMG.MSVDX."       /*Huawei Honor 6, Kirin 920*/
    ,"OMX.k3.video."        /*Huawei Honor 3C, Kirin 910*/
    ,"OMX.hisi."            /*Huawei Premium Phones, Kirin 950*/
    ,"OMX.TI.DUCATI1."      /*Galaxy Nexus, Ti OMAP4460*/
    ,"OMX.MTK.VIDEO."       /*no sense*/
    ,"OMX.LG.decoder."      /*no sense*/
    ,"OMX.rk.video_decoder."/*Youku TVBox. our service doesn't need this */
    ,"OMX.amlogic.avc"      /*MiBox1, 1s, 2. our service doesn't need this */
};

Modify the relevant codes of HardwareVideoEncoderFactory.java and HardwareVideoDecoderFactory to support more chips
For modification, please refer to https://www.pianshen.com/article/63171561802/

Programming VIP

Android mediasoup webrtc h264 soft codec related source code analysis

Android mediasoup webrtc h264 soft codec related source code analysis

mediasoup H264 support

Android webrtc video capture process source code analysis

Why Android H264 doesn't use hard coding

H264Decoder initialization process

openh264 provides codec related functions

webrtc video H264 hard coding

Popular Keywords