Recently the team built a WEB player with WASM + FFmpeg.We decode videos using FFmpeg by writing C and communicate with JavaScript on browsers by compiling C to WASM.The default FFmpeg decoded data is yuv, and canvas only supports rendering rgb, so there are two ways to handle this yuv. The first one uses FFmpeg exposure to convert YUV directly to RGB and render canvas. The second one uses webgl to render YUV to RGB on canvas.The first advantage is that the writing method is simple, only FFmpeg exposure method converts YUV directly to rgb. The disadvantage is that it will consume a certain amount of cpu. The second advantage is to use gpu for acceleration. The disadvantage is that the writing method is cumbersome and you need to be familiar with WEBGL.Considering that in order to reduce the CPU usage, we use gpu for parallel acceleration, using the second method.

Before we talk about YUV, let's first look at how YUV is obtained:

Since we are writing players, the steps to implement a player must go through the following steps:

1. The files of video, such as mp4, avi, flv, etc., mp4, avi, flv are equivalent to a container, which contains some information, such as compressed video, compressed audio, etc., and is demultiplexed. The compressed video and audio are extracted from the container. The compressed video is usually H265, H264 or other formats, and the compressed audio is generally aac or mp3.
2. The compressed video and compressed audio are decoded to get the original video and audio. The original audio data is usually pcm, while the original video data is usually yuv or rgb.
3. Then synchronize the audio and video.

After you can see the compressed video data decoded, you usually get a yuv.

YUV

What is YUV

For front-end developers, YUV is actually a bit unfamiliar, for audio and video development will generally come into contact with this. Simply speaking, YUV and RGB we are familiar with are the same way of color coding, but their three letters represent different meanings than RGB, YUV "Y" stands for brightness (Luminance or Luma), which is gray value; and "U" and "Luma""V" denotes chrominance (Chrominance or Chroma), which describes the color and saturation of an image and is used to specify the color of a pixel.

To give you a more intuitive sense of YUV, let's see what Y, U, V look like individually. Here we use the FFmpeg command to convert a picture of the Yahoku to YUV420P:

ffmpeg -i frame.jpg -s 352x288 -pix_fmt yuv420p test.yuv

Open test.yuv on the GLYUVPlay software to show the original image:

Y component is displayed separately:

The U component is displayed separately:

The V component is displayed separately:

As you can see from the above, when Y is displayed alone, it can display the complete image, but the picture is gray.U, V, on the other hand, represents chroma, one bluish and one red.

Benefits of using YUV

1. As you just saw, Y displays black and white images separately, so it's easy to change the YUV format from color to black and white. It's compatible with older black and white TVs. This feature is used for TV signals.
2. YUV data size is generally smaller than RGB format, which can save transmission bandwidth.(But if you use YUV444, it's 24 bits like RGB24)

YUV Sampling

Common YUV samples are YUV444, YUV422, YUV420:

Note: The black dot represents the Y component of the sample pixel point, and the hollow circle represents the UV component of the sample pixel point.

1. YUV 4:4:4 sampling, each Y corresponds to a set of UV components.
2. YUV 4:2:2 sampling, each two Ys share a set of UV components.
3. YUV 4:2:0 sampling, with one set of UV components for each four Ys.

YUV storage

YUV has two storage formats: packed and planar:

• The YUV format of packed, Y,U,V of each pixel point are continuously interlaced.
• The YUV format of planar stores the Y of all the pixel points in succession, then the U of all the pixel points, and then the V of all the pixel points.

For example, YUV can store YYYYUUVV in planar mode and YUYVYUYV in packed mode.

There are many formats for YUV, such as YUV420SP, YUV420P, YUV422P, YUV422SP, etc. Let's take a look at the more common formats:

• YUV420P (a set of UV components is shared for every four Ys):

• YUV420SP (packed, four Ys share a set of UV components. Unlike YUV420P, when YUV420SP is stored, U, V is staggered):

• YUV422P (planar, where each Y shares a set of UV components, so U and V add one more line than each YUV420P U and V):

• YUV422SP (packed, one set of UV components for each two Ys):

YUV420P and YUV420SP can be divided into two formats according to the order of U and V:

• YUV420P:U before and after YUV420P, also known as I420, before and after V, called YV12.
• Y U V420SP: Before and after V is called NV12, and before and after V is called NV21.

The data is arranged as follows:

I420: YYYYYYYY UU VV =>YUV420P

YV12: YYYYYYYY VV UU =>YUV420P

NV12: YYYYYYYY UV UV =>YUV420SP

NV21: YYYYYYYY VU VU =>YUV420SP

As to why there are so many formats, a lot of searches have found that the reason is to adapt to different TV broadcast systems and device systems, such as NV12, which is the only mode under ios. Android mode is NV21, such as YUV411, YUV420, which are more common in digital camera data, the former for NTSC and the latter for PAL.As for the introduction of the TV broadcast system, we can read this article Introduction of NTSC, PAL and SECAM

YUV calculation method

Take YUV420P for example to store a 1080 x 1280 picture. Its storage size is ((1080 x 1280 x 3) > 1) bytes. How does this work out?Let's look at this picture:

With Y420P storage, the size of Y is W x H = 1080x1280, U is (W/2) * (H/2)= (W*H)/4 = (1080x1280)/4, the same as V is
(W*H)/4 = (1080x1280)/4, so a graph is Y+U+V = (1080x1280)*3/2.
Since all three parts have row-first storage inside and Y,U,V sequential storage between them, the storage location of YUV is as follows (PS: used later):

Y:0 to 1080*1280
 U:1080*1280 to (1080*1280)*5/4
 V: (1080*1280)*5/4 to (1080*1280)*3/2

## WEBGL

What is WEBGL

Simply put, WebGL is a technology used to draw and render complex 3D graphics on a Web page and allow users to interact with them.

WEBGL Composition

In the world of webgl, the basic graphic elements that can be drawn are points, lines, triangles. Each image is made up of triangles of all sizes. The following figure shows that, no matter how complex a graphic is, its basic components are made up of triangles.

Shader

The shader is a program that runs on a GPU and is written in the OpenGL ES shading language, a bit like the c language:

Specific grammars can be referenced Introduction to Shader Language GLSL That's not much to say here.

In order to draw a graphic in WEBGL, there must be two shaders:

• Vertex Shader
• Fragment Shader

The main function of the vertex shader is to process vertices, while the pixel shader is to process each pixel generated by the rasterization phase (PS: a pixel can be understood as a pixel) and calculate the color of each pixel.

WEBGL Drawing Process

1. Provide vertex coordinates
Because the program is silly and does not know the vertices of a graph, we need to provide them by ourselves. Vertex coordinates can be written manually by ourselves or exported by software:

In this diagram, we write the vertices into a buffer, which is a memory area in the WebGL system. We can fill the buffer object with a large amount of vertex data at once, then save the data in it for use by the vertex shader.Next, we create and compile the vertex and chip shaders, connect the two shaders with program, and use them.To give an example of why this is done, we can think of it as creating a Fragment element: let f = document.createDocumentFragment(),
All shaders are created and compiled in a free state and need to be linked and used (which can be understood as document.body.appendChild(f), added to the body so that the dom element can be seen, that is, connected and used).
Next, we need to connect the buffer to the vertex shader for it to take effect.

2. Meta-assembly
After we provide vertices, the GPU will execute the vertex shader program one by one to generate the final coordinates of the vertices and assemble the graphics, depending on the number of vertices we provide.It can be understood that to make a kite, the kite skeleton needs to be built first, and the meta-assembly is at this stage.

3. Rasterization
This stage is like making a kite and building the kite skeleton, but at this time it cannot fly because it is empty and needs to be clothed.Rasterization is the phase in which the assembled geometry of primitives is converted into primitives (PS: primitives understand imagery).

4. Coloring and Rendering

Coloring is like kite cloth building, but there are no patterns at this time. You need to draw a pattern to make the kite look better, that is, the rasterized graphics do not have color at this time. It needs to be processed by the element colorizer, colored one by one and written into the color buffer, and finally the geometry with the image can be displayed in the browser.

summary
The WEBGL drawing process can be summarized as follows:

1. Provide vertex coordinates (we need them)
2. Meta-assembly (assembly into graphics by meta-type)
3. Rasterization (assembled graphics with primitives to generate pixel points)
4. Provide color values (can be calculated dynamically, pixel coloring)
5. Draw on the browser using canvas.

Image Drawing by WEBGL YUV

Since the images of each video frame are different, it is certainly impossible to know so many vertices, so how can we draw the images of video frames using webgl?A technique used here is texture mapping.Simply put, an image is pasted on a geometric surface to make the geometry look like an image-like geometry, that is, to match the texture coordinates with the webgl system coordinates one-to-one:

As shown in the figure above, the texture coordinate is divided into s and t coordinates (or uv coordinates). The range of values is between [0,1], independent of image size and resolution.The image below is a webgl coordinate system, a three-dimensional coordinate system, in which four vertices are declared, two triangles are assembled into a rectangle, and then the vertices of texture coordinates are corresponded to the webgl coordinate system one by one. The vertices of texture coordinates are eventually passed to the element shader, which extracts each of the striation colors of the picture, outputs them in a color buffer, and finally draws them in liu.Inside the viewer (PS: Texin you can understand as the pixels that make up the texture image).But if you do a one-to-one correspondence on the graph, the image will be inverted, because the image coordinates of canvas, by default (0, 0), are in the upper left corner:

Texture coordinates are in the lower left corner, so the image will be inverted when drawing. There are two solutions:

• For texture images with Y-axis flip, webgl provides api:

// 1 represents y-axis inversion of texture image
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, 1);

• Texture coordinates and webgl coordinate mapping are inverted, hold a chestnut. As shown in the figure above, the original texture coordinates (0.0, 1.0) correspond to webgl coordinates (-1.0, 1.0, 0.0), (0.0, 0.0) correspond to (-1.0, -1.0, 0.0), then we invert, (0.0, 1.0) correspond to (-1.0, -1.0, 0.0), and (0.0) correspond to (-1.0, 1.0, 1.0)., 0.0), so imaging in the browser is not the opposite.

Detailed steps

• Shader section

// Vertex shader
attribute lowp vec4 a_vertexPosition; // Transfer vertex coordinates through js
attribute vec2 a_texturePosition; // Transfer texture coordinates through js
varying vec2 v_texCoord; // Transfer texture coordinates to the cell shader
void main(){
    gl_Position=a_vertexPosition;// Set vertex coordinates
    v_texCoord=a_texturePosition;// Setting texture coordinates
}


// fragmentShader
precision lowp float;// lowp stands for calculation accuracy, and the lowest precision is used for performance savings
uniform sampler2D samplerY;// sampler2D is the sampler type where the texture of the picture is ultimately stored
uniform sampler2D samplerU;// sampler2D is the sampler type where the texture of the picture is ultimately stored
uniform sampler2D samplerV;// sampler2D is the sampler type where the texture of the picture is ultimately stored
varying vec2 v_texCoord; // Accept texture coordinates from vertex shader
void main(){
  float r,g,b,y,u,v,fYmul;
  y = texture2D(samplerY, v_texCoord).r;
  u = texture2D(samplerU, v_texCoord).r;
  v = texture2D(samplerV, v_texCoord).r;
    
    // YUV420P to RGB    
  fYmul = y * 1.1643828125;
  r = fYmul + 1.59602734375 * v - 0.870787598;
  g = fYmul - 0.39176171875 * u - 0.81296875 * v + 0.52959375;
  b = fYmul + 2.01723046875 * u - 1.081389160375;
  gl_FragColor = vec4(r, g, b, 1.0);
}

• Create and compile shaders, connect vertex and segment shaders to the program, and use:

let vertexShader=this._compileShader(vertexShaderSource,gl.VERTEX_SHADER);// Create and compile vertex shaders
let fragmentShader=this._compileShader(fragmentShaderSource,gl.FRAGMENT_SHADER);// Create and compile a meta shader

let program=this._createProgram(vertexShader,fragmentShader);// Create program and connect shaders

• Create a buffer to hold vertices and texture coordinates (PS: Buffer object is a memory area in the WebGL system where we can fill in a large amount of vertex data at once to the buffer object and then save it for use by the vertex shader).

let vertexBuffer = gl.createBuffer();
let vertexRectangle = new Float32Array([
    1.0,
    1.0,
    0.0,
    -1.0,
    1.0,
    0.0,
    1.0,
    -1.0,
    0.0,
    -1.0,
    -1.0,
    0.0
]);
gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
// Write data to buffer
gl.bufferData(gl.ARRAY_BUFFER, vertexRectangle, gl.STATIC_DRAW);
// Location of vertices found
let vertexPositionAttribute = gl.getAttribLocation(program, 'a_vertexPosition');
// Tell the graphics card to read vertex data from the currently bound buffer
gl.vertexAttribPointer(vertexPositionAttribute, 3, gl.FLOAT, false, 0, 0);
// Connect the vertexPosition variable to the buffer object assigned to it
gl.enableVertexAttribArray(vertexPositionAttribute);

// Declare texture coordinates
let textureRectangle = new Float32Array([1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0]);
let textureBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, textureBuffer);
gl.bufferData(gl.ARRAY_BUFFER, textureRectangle, gl.STATIC_DRAW);
let textureCoord = gl.getAttribLocation(program, 'a_texturePosition');
gl.vertexAttribPointer(textureCoord, 2, gl.FLOAT, false, 0, 0);
gl.enableVertexAttribArray(textureCoord); 

• Initialize and activate texture unit (YUV)

//Activates the specified texture unit
gl.activeTexture(gl.TEXTURE0);
gl.y=this._createTexture(); // Create Texture
gl.uniform1i(gl.getUniformLocation(program,'samplerY'),0);//Gets the storage location of the samplerY variable, specifies texture unit number 0 to pass the texture object to samplerY

gl.activeTexture(gl.TEXTURE1);
gl.u=this._createTexture();
gl.uniform1i(gl.getUniformLocation(program,'samplerU'),1);//Gets the storage location of the samplerU variable, specifies texture unit number 1 to pass the texture object to samplerU

gl.activeTexture(gl.TEXTURE2);
gl.v=this._createTexture();
gl.uniform1i(gl.getUniformLocation(program,'samplerV'),2);//Gets the storage location of the samplerV variable, specifies texture unit number 2 to pass the texture object to samplerV

• Render Drawing (PS: Since the data we get is YUV420P, the calculation method can refer to the calculation method just mentioned).

 // Set the color value when emptying the color buffer
 gl.clearColor(0, 0, 0, 0);
 // Empty Buffer
 gl.clear(gl.COLOR_BUFFER_BIT);

let uOffset = width * height;
let vOffset = (width >> 1) * (height >> 1);

gl.bindTexture(gl.TEXTURE_2D, gl.y);
// Fill in the Y texture, and the width and height of Y are width and height, stored in data.subarray(0, width * height)
gl.texImage2D(
    gl.TEXTURE_2D,
    0,
    gl.LUMINANCE,
    width,
    height,
    0,
    gl.LUMINANCE,
    gl.UNSIGNED_BYTE,
    data.subarray(0, uOffset)
);

gl.bindTexture(gl.TEXTURE_2D, gl.u);
// Fill in the U texture, the width and height of Y are width/2 and height/2, and the storage location is data.subarray(width * height, width/2 * height/2 + width * height)
gl.texImage2D(
    gl.TEXTURE_2D,
    0,
    gl.LUMINANCE,
    width >> 1,
    height >> 1,
    0,
    gl.LUMINANCE,
    gl.UNSIGNED_BYTE,
    data.subarray(uOffset, uOffset + vOffset)
);

gl.bindTexture(gl.TEXTURE_2D, gl.v);
// Fill in the U texture, the width and height of Y are width/2 and height/2, and the storage location is data.subarray(width/2 * height/2 + width * height, data.length)
gl.texImage2D(
    gl.TEXTURE_2D,
    0,
    gl.LUMINANCE,
    width >> 1,
    height >> 1,
    0,
    gl.LUMINANCE,
    gl.UNSIGNED_BYTE,
    data.subarray(uOffset + vOffset, data.length)
);

gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4); // Draw four points, or rectangles

These steps will ultimately make this diagram:

Full code:

export default class WebglScreen {
    constructor(canvas) {
        this.canvas = canvas;
        this.gl = canvas.getContext('webgl') || canvas.getContext('experimental-webgl');
        this._init();
    }

    _init() {
        let gl = this.gl;
        if (!gl) {
            console.log('gl not support!');
            return;
        }
        // Image preprocessing
        gl.pixelStorei(gl.UNPACK_ALIGNMENT, 1);
        // Vertex shader code in GLSL format
        let vertexShaderSource = `
            attribute lowp vec4 a_vertexPosition;
            attribute vec2 a_texturePosition;
            varying vec2 v_texCoord;
            void main() {
                gl_Position = a_vertexPosition;
                v_texCoord = a_texturePosition;
            }
        `;

        let fragmentShaderSource = `
            precision lowp float;
            uniform sampler2D samplerY;
            uniform sampler2D samplerU;
            uniform sampler2D samplerV;
            varying vec2 v_texCoord;
            void main() {
                float r,g,b,y,u,v,fYmul;
                y = texture2D(samplerY, v_texCoord).r;
                u = texture2D(samplerU, v_texCoord).r;
                v = texture2D(samplerV, v_texCoord).r;

                fYmul = y * 1.1643828125;
                r = fYmul + 1.59602734375 * v - 0.870787598;
                g = fYmul - 0.39176171875 * u - 0.81296875 * v + 0.52959375;
                b = fYmul + 2.01723046875 * u - 1.081389160375;
                gl_FragColor = vec4(r, g, b, 1.0);
            }
        `;

        let vertexShader = this._compileShader(vertexShaderSource, gl.VERTEX_SHADER);
        let fragmentShader = this._compileShader(fragmentShaderSource, gl.FRAGMENT_SHADER);

        let program = this._createProgram(vertexShader, fragmentShader);

        this._initVertexBuffers(program);

        // Activates the specified texture unit
        gl.activeTexture(gl.TEXTURE0);
        gl.y = this._createTexture();
        gl.uniform1i(gl.getUniformLocation(program, 'samplerY'), 0);

        gl.activeTexture(gl.TEXTURE1);
        gl.u = this._createTexture();
        gl.uniform1i(gl.getUniformLocation(program, 'samplerU'), 1);

        gl.activeTexture(gl.TEXTURE2);
        gl.v = this._createTexture();
        gl.uniform1i(gl.getUniformLocation(program, 'samplerV'), 2);
    }
    /**
     * Initialize vertex buffer
     * @param {glProgram} program program
     */

    _initVertexBuffers(program) {
        let gl = this.gl;
        let vertexBuffer = gl.createBuffer();
        let vertexRectangle = new Float32Array([
            1.0,
            1.0,
            0.0,
            -1.0,
            1.0,
            0.0,
            1.0,
            -1.0,
            0.0,
            -1.0,
            -1.0,
            0.0
        ]);
        gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
        // Write data to buffer
        gl.bufferData(gl.ARRAY_BUFFER, vertexRectangle, gl.STATIC_DRAW);
        // Location of vertices found
        let vertexPositionAttribute = gl.getAttribLocation(program, 'a_vertexPosition');
        // Tell the graphics card to read vertex data from the currently bound buffer
        gl.vertexAttribPointer(vertexPositionAttribute, 3, gl.FLOAT, false, 0, 0);
        // Connect the vertexPosition variable to the buffer object assigned to it
        gl.enableVertexAttribArray(vertexPositionAttribute);

        let textureRectangle = new Float32Array([1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0]);
        let textureBuffer = gl.createBuffer();
        gl.bindBuffer(gl.ARRAY_BUFFER, textureBuffer);
        gl.bufferData(gl.ARRAY_BUFFER, textureRectangle, gl.STATIC_DRAW);
        let textureCoord = gl.getAttribLocation(program, 'a_texturePosition');
        gl.vertexAttribPointer(textureCoord, 2, gl.FLOAT, false, 0, 0);
        gl.enableVertexAttribArray(textureCoord);
    }

    /**
     * Create and compile a shader
     * @param {string} shaderSource GLSL Shader code for format
     * @param {number} shaderType Shader type, VERTEX_SHADER or FRAGMENT_SHADER.
     * @return {glShader} Shader.
     */
    _compileShader(shaderSource, shaderType) {
        // Create Shader Program
        let shader = this.gl.createShader(shaderType);
        // Set source code for shader
        this.gl.shaderSource(shader, shaderSource);
        // Compile Shader
        this.gl.compileShader(shader);
        const success = this.gl.getShaderParameter(shader, this.gl.COMPILE_STATUS);
        if (!success) {
            let err = this.gl.getShaderInfoLog(shader);
            this.gl.deleteShader(shader);
            console.error('could not compile shader', err);
            return;
        }

        return shader;
    }

    /**
     * Create a program from two shaders
     * @param {glShader} vertexShader Vertex shader.
     * @param {glShader} fragmentShader Fragment shader.
     * @return {glProgram} program
     */
    _createProgram(vertexShader, fragmentShader) {
        const gl = this.gl;
        let program = gl.createProgram();

        // Attach shader
        gl.attachShader(program, vertexShader);
        gl.attachShader(program, fragmentShader);

        gl.linkProgram(program);
        // Add WebGLProgram object to current rendering state
        gl.useProgram(program);
        const success = this.gl.getProgramParameter(program, this.gl.LINK_STATUS);

        if (!success) {
            console.err('program fail to link' + this.gl.getShaderInfoLog(program));
            return;
        }

        return program;
    }

    /**
     * Set Texture
     */
    _createTexture(filter = this.gl.LINEAR) {
        let gl = this.gl;
        let t = gl.createTexture();
        // Bind the given glTexture to the target (binding point)
        gl.bindTexture(gl.TEXTURE_2D, t);
        // Texture packaging refers to https://github.com/fem-d/webGL/blob/master/blog/WebGL Basic Learning Paper (Lesson%207). MD -> Texture wrapping
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
        // Set Texture Filtering
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, filter);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, filter);
        return t;
    }

    /**
     * Render the picture
     * @param {number} width width
     * @param {number} height height
     */
    renderImg(width, height, data) {
        let gl = this.gl;
        // Set the viewport, that is, specify the x, y affine transformation from the standard device to the window coordinates
        gl.viewport(0, 0, gl.canvas.width, gl.canvas.height);
        // Set the color value when emptying the color buffer
        gl.clearColor(0, 0, 0, 0);
        // Empty Buffer
        gl.clear(gl.COLOR_BUFFER_BIT);

        let uOffset = width * height;
        let vOffset = (width >> 1) * (height >> 1);

        gl.bindTexture(gl.TEXTURE_2D, gl.y);
        // Fill Texture
        gl.texImage2D(
            gl.TEXTURE_2D,
            0,
            gl.LUMINANCE,
            width,
            height,
            0,
            gl.LUMINANCE,
            gl.UNSIGNED_BYTE,
            data.subarray(0, uOffset)
        );

        gl.bindTexture(gl.TEXTURE_2D, gl.u);
        gl.texImage2D(
            gl.TEXTURE_2D,
            0,
            gl.LUMINANCE,
            width >> 1,
            height >> 1,
            0,
            gl.LUMINANCE,
            gl.UNSIGNED_BYTE,
            data.subarray(uOffset, uOffset + vOffset)
        );

        gl.bindTexture(gl.TEXTURE_2D, gl.v);
        gl.texImage2D(
            gl.TEXTURE_2D,
            0,
            gl.LUMINANCE,
            width >> 1,
            height >> 1,
            0,
            gl.LUMINANCE,
            gl.UNSIGNED_BYTE,
            data.subarray(uOffset + vOffset, data.length)
        );

        gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);
    }

    /**
     * Based on resetting canvas size
     * @param {number} width width
     * @param {number} height height
     * @param {number} maxWidth Maximum width
     */
    setSize(width, height, maxWidth) {
        let canvasWidth = Math.min(maxWidth, width);
        this.canvas.width = canvasWidth;
        this.canvas.height = canvasWidth * height / width;
    }

    destroy() {
        const {
            gl
        } = this;

        gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT | gl.STENCIL_BUFFER_BIT);
    }
}

Finally, let's look at the effects:

Problems encountered

In the actual development process, we test some live streams. Sometimes when rendering, the image display is normal, but the color will be green. It is found that the video widths of different hosts of live streams will be different, such as 368 when the host is in pk, 720 when the host is in pk, 540 when the host is small, 540 when the width is 540.The body reason is that webgl is preprocessed and the following values are set to 4 by default:

// Image preprocessing
gl.pixelStorei(gl.UNPACK_ALIGNMENT, 4);

The default setting is 4 bytes per line, and the width of Y component is 540, which is a multiple of 4, and the bytes are aligned, so the image can be displayed normally, while the width of U and V component is 540/2 = 270, 270 is not a multiple of 4, and the bytes are not aligned, so the pigments will appear green.There are two ways to solve this problem:

• The first is to have webgl handle 1 byte per line directly (which has an impact on performance):

// Image preprocessing
gl.pixelStorei(gl.UNPACK_ALIGNMENT, 1);

• The second is to get an image with a width of 8 times, so that YUV byte alignment can be achieved without displaying a green screen, but it is not recommended to do so. CPU occupies a lot when rotating, so the first scheme is recommended.

Reference Article

Image Video Coding and FFmpeg(2) - Introduction and Application of YUV Format - eustoma - Blog Park
YUV pixel formats
https://wiki.videolan.org/YUV/
Video rendering using 8-bit YUV format| Microsoft Docs?redirectedfrom=MSDN)
YUV for IOS Video Format - Short Book
Illustrate how WebGL&Three.js works - cnwander - Blog Park