When using WebRTC, there are generally two ways to beautify the video: replacing the acquisition module in WebRTC and beautifying the video data.
1, Replace the acquisition module in WebRTC
It is relatively simple to replace the acquisition module in WebRTC. Use GPUImageVideoCamera to replace the video acquisition in WebRTC, get the image after adding beauty processing by GPUImage, and send it to the OnFrame method of WebRTC.
Refer to the full platform push-pull stream SDK developed based on WebRTC framework: Github
Set beauty
- (void)setBeautyFace:(BOOL)beautyFace{
if(_beautyFace == beautyFace) return;
_beautyFace = beautyFace;
[_emptyFilter removeAllTargets];
[_filter removeAllTargets];
[_videoCamera removeAllTargets];
if(_beautyFace){
_filter = [[GPUImageBeautifyFilter alloc] init];
_emptyFilter = [[GPUImageEmptyFilter alloc] init];
}else{
_filter = [[GPUImageEmptyFilter alloc] init];
}
__weak typeof(self) _self = self;
[_filter setFrameProcessingCompletionBlock:^(GPUImageOutput *output, CMTime time) {
[_self processVideo:output];
}];
[_videoCamera addTarget:_filter];
if (beautyFace) {
[_filter addTarget:_emptyFilter];
if(_gpuImageView) [_emptyFilter addTarget:_gpuImageView];
} else {
if(_gpuImageView) [_filter addTarget:_gpuImageView];
}
}
format conversion
The Pixel format after GPUImage processing is BGRA. After processing, it needs to be converted to I420 format for internal processing and rendering.
WebRTC uses the Pixel in NV12 format when encoding, so secondary format conversion will be carried out during encoding
-(void) processVideo:(GPUImageOutput *)output{
rtc::CritScope cs(&cs_capture_);
if (!_isRunning) {
return;
}
@autoreleasepool {
GPUImageFramebuffer *imageFramebuffer = output.framebufferForOutput;
size_t width = imageFramebuffer.size.width;
size_t height = imageFramebuffer.size.height;
uint32_t size = width * height * 3 / 2;
if(self.nWidth != width || self.nHeight != height)
{
self.nWidth = width;
self.nHeight = height;
if(_dst)
delete[] _dst;
_dst = NULL;
}
if(_dst == NULL)
{
_dst = new uint8_t[size];
}
uint8_t* y_pointer = (uint8_t*)_dst;
uint8_t* u_pointer = (uint8_t*)y_pointer + width*height;
uint8_t* v_pointer = (uint8_t*)u_pointer + width*height/4;
int y_pitch = width;
int u_pitch = (width + 1) >> 1;
int v_pitch = (width + 1) >> 1;
libyuv::ARGBToI420([imageFramebuffer byteBuffer], width * 4, y_pointer, y_pitch, u_pointer, u_pitch, v_pointer, v_pitch, width, height);
if(self.bVideoEnable)
libyuv::I420Rect(y_pointer, y_pitch, u_pointer, u_pitch, v_pointer, v_pitch, 0, 0, width, height, 32, 128, 128);
if(_capturer != nil)
_capturer->CaptureYUVData(_dst, width, height, size);
}
}
Send the data after beauty to the OnFrame method of WebRTC
Gpuimagevideocamera class is a camera class encapsulated by GPUImage, which is consistent with the collection class function in WebRTC. By inheriting the cricket:: videocamera class, you can insert the collected audio and video stream into WebRTC.
namespace webrtc {
// Inherit cricket:: videocamera
class GPUImageVideoCapturer : public cricket::VideoCapturer {
...
}
}
void GPUImageVideoCapturer::CaptureYUVData(const webrtc::VideoFrame& frame, int width, int height)
{
VideoCapturer::OnFrame(frame, width, height);
}
2, Beautify video data
The idea of video data beauty is the traditional third-party beauty SDK, Process the internally collected audio and video data: internally collected data (CVPixelBufferRef) - "convert to texture" - "beautify the texture of audio and video" - "convert the texture of beautify to the collected data of iOS (CVPixelBufferRef) -" return to WebRTC for internal rendering, coding and transmission.
Synchronous thread
Generally, synchronous threads are used for internal processing, which can ensure the linear flow of data. See the code snippet in GPUImage
runSynchronouslyOnVideoProcessingQueue(^{
// Beauty treatment
});
Convert CVPixelBufferRef data to texture
Conversion method of RGB format type
-
Method of CoreVideo framework: use this method to create CVOpenGLESTextureRef texture and obtain texture id through CVOpenGLESTextureGetName(texture).
- (GLuint)convertRGBPixelBufferToTexture:(CVPixelBufferRef)pixelBuffer { if (!pixelBuffer) { return 0; } CGSize textureSize = CGSizeMake(CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer)); CVOpenGLESTextureRef texture = nil; CVReturn status = CVOpenGLESTextureCacheCreateTextureFromImage(nil, [[GPUImageContext sharedImageProcessingContext] coreVideoTextureCache], pixelBuffer, nil, GL_TEXTURE_2D, GL_RGBA, textureSize.width, textureSize.height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &texture); if (status != kCVReturnSuccess) { NSLog(@"Can't create texture"); } self.renderTexture = texture; return CVOpenGLESTextureGetName(texture); } -
OpenGL method: create a texture object and upload the image data in CVPixelBufferRef to the texture object using glTexImage2D method.
glBindTexture(GL_TEXTURE_2D, [outputFramebuffer texture]); glTexImage2D(GL_TEXTURE_2D, 0, _pixelFormat==GPUPixelFormatRGB ? GL_RGB : GL_RGBA, (int)uploadedImageSize.width, (int)uploadedImageSize.height, 0, (GLint)_pixelFormat, (GLenum)_pixelType, bytesToUpload);
Conversion method of YUV format type
- (GLuint)convertYUVPixelBufferToTexture:(CVPixelBufferRef)pixelBuffer {
if (!pixelBuffer) {
return 0;
}
CGSize textureSize = CGSizeMake(CVPixelBufferGetWidth(pixelBuffer),
CVPixelBufferGetHeight(pixelBuffer));
[EAGLContext setCurrentContext:self.context];
GLuint frameBuffer;
GLuint textureID;
// FBO
glGenFramebuffers(1, &frameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
// texture
glGenTextures(1, &textureID);
glBindTexture(GL_TEXTURE_2D, textureID);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, textureSize.width, textureSize.height, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureID, 0);
glViewport(0, 0, textureSize.width, textureSize.height);
// program
glUseProgram(self.yuvConversionProgram);
// texture
CVOpenGLESTextureRef luminanceTextureRef = nil;
CVOpenGLESTextureRef chrominanceTextureRef = nil;
CVReturn status = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
self.textureCache,
pixelBuffer,
nil,
GL_TEXTURE_2D,
GL_LUMINANCE,
textureSize.width,
textureSize.height,
GL_LUMINANCE,
GL_UNSIGNED_BYTE,
0,
&luminanceTextureRef);
if (status != kCVReturnSuccess) {
NSLog(@"Can't create luminanceTexture");
}
status = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
self.textureCache,
pixelBuffer,
nil,
GL_TEXTURE_2D,
GL_LUMINANCE_ALPHA,
textureSize.width / 2,
textureSize.height / 2,
GL_LUMINANCE_ALPHA,
GL_UNSIGNED_BYTE,
1,
&chrominanceTextureRef);
if (status != kCVReturnSuccess) {
NSLog(@"Can't create chrominanceTexture");
}
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, CVOpenGLESTextureGetName(luminanceTextureRef));
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glUniform1i(glGetUniformLocation(self.yuvConversionProgram, "luminanceTexture"), 0);
glActiveTexture(GL_TEXTURE1);
glBindTexture(GL_TEXTURE_2D, CVOpenGLESTextureGetName(chrominanceTextureRef));
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glUniform1i(glGetUniformLocation(self.yuvConversionProgram, "chrominanceTexture"), 1);
GLfloat kXDXPreViewColorConversion601FullRange[] = {
1.0, 1.0, 1.0,
0.0, -0.343, 1.765,
1.4, -0.711, 0.0,
};
GLuint yuvConversionMatrixUniform = glGetUniformLocation(self.yuvConversionProgram, "colorConversionMatrix");
glUniformMatrix3fv(yuvConversionMatrixUniform, 1, GL_FALSE, kXDXPreViewColorConversion601FullRange);
// VBO
glBindBuffer(GL_ARRAY_BUFFER, self.VBO);
GLuint positionSlot = glGetAttribLocation(self.yuvConversionProgram, "position");
glEnableVertexAttribArray(positionSlot);
glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)0);
GLuint textureSlot = glGetAttribLocation(self.yuvConversionProgram, "inputTextureCoordinate");
glEnableVertexAttribArray(textureSlot);
glVertexAttribPointer(textureSlot, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)(3* sizeof(float)));
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glDeleteFramebuffers(1, &frameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
glBindBuffer(GL_ARRAY_BUFFER, 0);
glFlush();
self.luminanceTexture = luminanceTextureRef;
self.chrominanceTexture = chrominanceTextureRef;
if (luminanceTextureRef) {
CFRelease(luminanceTextureRef);
}
if (chrominanceTextureRef) {
CFRelease(chrominanceTextureRef);
}
return textureID;
}
Use GPUImageTextureInput to load filters and GPUImageTextureOutput to output data
[GPUImageContext setActiveShaderProgram:nil];
GPUImageTextureInput *textureInput = [[ARGPUImageTextureInput alloc] initWithTexture:textureID size:size];
GPUImageSmoothToonFilter *filter = [[GPUImageSmoothToonFilter alloc] init];
[textureInput addTarget:filter];
GPUImageTextureOutput *textureOutput = [[GPUImageTextureOutput alloc] init];
[filter addTarget:textureOutput];
[textureInput processTextureWithFrameTime:kCMTimeZero];
Get textureOutput, that is, get the output texture.
GPUImageTextureOutput texture converted to CVPixelBufferRef data
- (CVPixelBufferRef)convertTextureToPixelBuffer:(GLuint)texture
textureSize:(CGSize)textureSize {
[EAGLContext setCurrentContext:self.context];
CVPixelBufferRef pixelBuffer = [self createPixelBufferWithSize:textureSize];
GLuint targetTextureID = [self convertRGBPixelBufferToTexture:pixelBuffer];
GLuint frameBuffer;
// FBO
glGenFramebuffers(1, &frameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
// texture
glBindTexture(GL_TEXTURE_2D, targetTextureID);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, textureSize.width, textureSize.height, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, targetTextureID, 0);
glViewport(0, 0, textureSize.width, textureSize.height);
// program
glUseProgram(self.normalProgram);
// texture
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, texture);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glUniform1i(glGetUniformLocation(self.normalProgram, "renderTexture"), 0);
// VBO
glBindBuffer(GL_ARRAY_BUFFER, self.VBO);
GLuint positionSlot = glGetAttribLocation(self.normalProgram, "position");
glEnableVertexAttribArray(positionSlot);
glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)0);
GLuint textureSlot = glGetAttribLocation(self.normalProgram, "inputTextureCoordinate");
glEnableVertexAttribArray(textureSlot);
glVertexAttribPointer(textureSlot, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)(3* sizeof(float)));
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glDeleteFramebuffers(1, &frameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
glBindBuffer(GL_ARRAY_BUFFER, 0);
glFlush();
return pixelBuffer;
}
The CVPixelBufferRef after beauty is synchronized and returned to the SDK for rendering and transmission.
3, Summary
The beauty of audio and video has become a common function of audio and video applications. In addition to the above two methods, third-party beauty can also be used. Generally, audio and video manufacturers provide self collection function, while the third-party beauty function provides Beauty Collection camera function. The two can be seamlessly combined. If you don't have high requirements for beauty in your own application, you can use the beauty provided by the audio and video SDK (whitening, beautifying and ruddy). If you use it in entertainment scenes, you must integrate the third-party beauty SDK in addition to beauty (thin face, big eyes) and stickers (2D and 3D).