"android Multimedia api" series is a series of media-related APIs which are often used in the integration of android development. The main contents of multimedia development include audio, video recording and playing, camera operation, recording operation, streaming media, live broadcasting, push-flow, pull-flow and so on. In recent years, Mobile Live Broadcasting and video applications have developed like bamboo shoots after a spring rain. Well, this metaphor can not be read!! Anyway, the rise of the industry has certainly spawned a lot of multimedia-related application developers. So how to become a multimedia developer, first of all, we must be proficient in using and understanding android's own multimedia api, and also master many file formats and streaming media protocols such as pcm, yuv, rgb, h264, aac, flv, mpegts, mp4, udp, rtp, rtmp and so on. So here is the collation of android-related multimedia api, to provide students who want to work in streaming media as a reference, but also to thank those who share the spirit of God on the network!!

Basic concepts:

1. Video playback: demuxer - > separated audio stream and video stream - > decoder - > playback original data (e.g. pcm yuv)
2. Video recording: acquisition of raw data (e.g. PCM yuv) - > encoder (encoding) - > Muxer (encapsulation format, e.g. mp43 gp)
3. Streaming Media Protocol: udp, rtp, rtmp, rtcp, rtsp, etc.
4. Audio and video encapsulation formats: mp4, 3gp, flv, etc.
5. Audio and video coding formats: aac, amr, h264, h265, etc.
6. Original audio and video data formats: pcm, yuv, rgb, etc.

Flow chart:

Catalogue of articles:

1. VideoView Video Player Control
2. Camera cooperates with surface to preview camera pictures and take photos
3. MediaPlayer Custom Video Player
4. Media Recorder Audio and Video Recording api
5. AudioTrack original audio pcm playback api
6. AudioRecord original audio pcm acquisition api
7. camera captures raw video data

How to collect nv21 raw data through camera camera?
In the development of live broadcasting, it is often necessary to acquire the original frame data of video, then pre-process such as beauty, watermarking, special effects, etc. and then push it out by coding in the way of rtmp or rtsp protocol to complete real-time image transmission. So how to get the original video frame data? Installing camera api has a method of setting callback, which can be used to obtain raw video data such as nv21, nv12, yv12, etc. The following demo is built to get the configuration camera to open the preview when the surfaceview control is built, so that real-time data can be obtained in the callback method setPreview Callback and in the public void onPreview Frame (byte [] data, Camera camera).

Since the camera is the user's privacy right, it is necessary to apply for the camera right:

    <uses-permission android:name="android.permission.CAMERA" />
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

    <uses-feature android:name="android.hardware.camera" />
    <uses-feature android:name="android.hardware.camera.autofocus" />

Acquisition of original video frame data based on camera:

xml layout file:

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity" >

    <SurfaceView 
        android:id="@+id/show"
        android:layout_width="match_parent"
        android:layout_height="match_parent"/>

</RelativeLayout>

java code:

package com.example.getyuvandpush;

import java.io.IOException;
import java.util.List;

import android.app.Activity;
import android.graphics.ImageFormat;
import android.hardware.Camera;
import android.hardware.Camera.Parameters;
import android.hardware.Camera.PreviewCallback;
import android.hardware.Camera.Size;
import android.os.Bundle;
import android.view.SurfaceHolder;
import android.view.SurfaceHolder.Callback;
import android.view.SurfaceView;

@SuppressWarnings("deprecation")
public class MainActivity extends Activity implements Callback, PreviewCallback {

    private SurfaceView show;
    private Camera camera;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        // TODO Auto-generated method stub
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        show=(SurfaceView) findViewById(R.id.show);
        show.getHolder().addCallback(this);

    }

    @Override
    public void surfaceCreated(SurfaceHolder holder) {
        // TODO Auto-generated method stub

    }

    @Override
    public void surfaceChanged(SurfaceHolder holder, int format, int width,
            int height) {
        // Open the camera when surfaceview is built and effective width is obtained
        if (camera!=null) {
            return;
        }
        camera= Camera.open();
        Parameters parms = camera.getParameters();
        List<Size> listsize = parms.getSupportedPictureSizes();
        Size sizeOut=null;
        for (Size size : listsize) {
            if (size.width>=width) {
                break;
            }
            sizeOut=size;
        }
        parms.setPreviewSize(sizeOut.width, sizeOut.height);
        parms.setPictureSize(sizeOut.width, sizeOut.height);
        parms.setPreviewFormat(ImageFormat.NV21);
        camera.setParameters(parms);
        camera.setPreviewCallback(this);
        try {
            camera.setPreviewDisplay(holder);
            camera.startPreview();//Start Preview
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

    @Override
    public void surfaceDestroyed(SurfaceHolder holder) {
        // surfaceview destructs to shut down the camera
        if (camera==null) {
            return;
        }
        camera.setPreviewCallback(null);
        camera.stopPreview();// Stop Preview
        camera.release();
        camera=null;
    }

    @Override
    public void onPreviewFrame(byte[] data, Camera camera) {
        //nv21 raw data can be obtained

    }

}