Technical background
Before realizing the front-end device access of GB28181 on Android platform, we had very mature RTMP push, RTSP push and lightweight RTSP service modules a few years ago, especially RTMP push, which is widely used in the industry. Many developers may ask, since we have the above modules, why do we need to realize the front-end access of GB28181?
First, let's understand GB/T28181: The full name of the national standard GB/T28181 protocol is "technical requirements for information transmission, exchange and control of security video surveillance networking system". It is a white paper defining the standards for video networking transmission and equipment control. It is proposed by the science and Technology Information Bureau of the Ministry of public security. The standard specifies the interconnection structure, communication protocol structure, transmission Basic and security requirements for exchange and control, as well as technical requirements for control, transmission process and protocol interface. It solves the problems of video interconnection, data sharing and equipment control. This problem solves the problem of video information fighting on its own from the top, and opens up the information island of video networking.
Technical features
The implementation of GB28181 protocol is divided into two parts, one is signaling and the other is streaming media data transmission. Compared with RTMP, GB28181 supports TCP and UDP modes. Signaling flow is responsible for session interaction and data flow is responsible for data transmission. It is suitable for platform level product docking of standard protocol specifications.
In addition to supporting conventional audio and video data access, Android terminal can also support Subscribe, mobile position subscription, real-time directory query, etc., and support standard 28181 service docking.
In addition, in the product design, the media stream supports UDP and TCP passive modes of the latest GB28181-2016, parameter configuration, registration validity period, heartbeat interval, heartbeat interval times, TCP/UDP signaling settings, RTP Sender IP address type, RTP socket local port, SSRC, RTP socket sending Buffer size, RTP timestamp clock frequency settings, successful registration Registration timeout, INVIT, ACK, BYE status callback.
functional design
GB28181 front end device module on Android side supports conventional video acquisition and coding settings. The functional design is as follows:
- [local preview] support local front and rear camera preview;
- [video format] H.264/H.265(Android H.265 hard coding);
- [audio format] AAC;
- [volume adjustment] the collection terminal of Android platform supports real-time volume adjustment;
- [H.264 hard coding] support hard coding of H.264 specific models;
- [H.265 hard coding] support the hard coding of H.265 specific models;
- [soft and hard coding parameter configuration] support gop interval, frame rate and bit rate settings;
- [soft coding parameter configuration] support soft coding profile, soft coding speed and variable bit rate settings;
- [horizontal and vertical screen streaming] Android platform supports horizontal and vertical screen streaming;
- [multi resolution support] support multiple resolution settings of camera or screen;
- [mobile terminal push screen] Android platform supports background service push screen (push screen requires version 5.0 +);
- [mode support] media streaming supports UDP and TCP passive modes of the latest GB28181-2016;
- [parameter setting] support the setting of registration validity period, heartbeat interval, number of heartbeat intervals and TCP/UDP signaling;
- [parameter setting] support RTP Sender IP address type, RTP socket local port, SSRC, RTP socket sending Buffer size, RTP timestamp clock frequency setting;
- [status callback] supports successful registration, registration timeout, INVIT, ACK and BYE status callback;
- [watermark] support text watermark and png watermark;
- [image] Android platform supports real-time image function of front camera;
- [real time switching between front and rear cameras] Android platform supports switching between front and rear cameras during acquisition;
- [complex network processing] support automatic adaptation of various network environments such as network disconnection and reconnection;
- [dynamic bit rate] supports automatic adjustment of streaming bit rate according to network conditions;
- [real time mute] support real-time mute / unmute during push;
- [real time snapshot] support real-time snapshot during streaming;
- [noise reduction] support noise reduction treatment, automatic gain and VAD detection caused by ambient sound and mobile phone interference;
- [video data docking before external coding] support YUV data docking;
- [external coding front audio data docking] support PCM docking;
- [video data docking after external coding] support external H.264 data docking;
- [audio data docking after external coding] external AAC data docking;
- [extended video recording function] support the combination of video recording module and video recording related functions;
- [server compatible] supports standard GB28181 services.
interface design
Interface design is divided into two parts: RTP Sender interface and GB28181 interface. You can also pay attention to detailed examples github；
RTP Sender Interface Description:
1. Create an RTP Sender instance and return the instance handle:
/* * Create RTP Sender instance * * @param reserve: Reserved parameter 0 * * @return RTP Sender Handle, 0 means failure */ public native long CreateRTPSender(int reserve);
2. Set the RTP Sender transmission protocol, 0:UDP, 1:TCP. The default is UDP
/**
*Set RTP Sender transport protocol
*
* @param rtp_sender_handle, CreateRTPSender Return value
* @param transport_protocol, 0:UDP, 1:TCP, The default is UDP
*
* @return {0} if successful
*/
public native int SetRTPSenderTransportProtocol(long rtp_sender_handle, int transport_protocol);3. Set the IP address type of RTP sender, such as IPv4 and IPv6. Currently, only IPv4 is supported
/**
*Set RTP Sender IP address type
*
* @param rtp_sender_handle, CreateRTPSender Return value
* @param ip_address_type, 0:IPV4, 1:IPV6, The default is IPV4. Currently, only IPV4 is supported
*
* @return {0} if successful
*/
public native int SetRTPSenderIPAddressType(long rtp_sender_handle, int ip_address_type);4. Set the local port of RTP Sender RTP Socket, which must be an even number. If it is set to 0, the SDK will automatically allocate it. The default value is 0
/**
*Set RTP Sender RTP Socket local port
*
* @param rtp_sender_handle, CreateRTPSender Return value
* @param port, It must be an even number. If 0 is set, the SDK will be automatically allocated. The default value is 0
*
* @return {0} if successful
*/
public native int SetRTPSenderLocalPort(long rtp_sender_handle, int port);5. Set RTP Sender SSRC
/**
*Set RTP Sender SSRC
*
* @param rtp_sender_handle, CreateRTPSender Return value
* @param ssrc, If set, this string must be able to be converted to uint32 type, otherwise the setting fails
*
* @return {0} if successful
*/
public native int SetRTPSenderSSRC(long rtp_sender_handle, String ssrc);6. Set the sending Buffer size of RTP Sender RTP socket
/**
*Set RTP Sender RTP socket send Buffer size
*
* @param rtp_sender_handle, CreateRTPSender Return value
* @param buffer_size, It must be greater than 0. The default value is 512 * 1024. Currently, it is only valid for UDP socket s. Set an appropriate value according to the video bit rate
*
* @return {0} if successful
*/
public native int SetRTPSenderSocketSendBuffer(long rtp_sender_handle, int buffer_size);7. Set RTP Sender RTP timestamp clock frequency
/**
*Set RTP Sender RTP timestamp clock frequency
*
* @param rtp_sender_handle, CreateRTPSender Return value
* @param clock_rate, It must be greater than 0. For GB28181 PS, it is 90kHz, that is 90000
*
* @return {0} if successful
*/
public native int SetRTPSenderClockRate(long rtp_sender_handle, int clock_rate);8. Set the destination IP address of RTP Sender. Note that it is currently used in GB2818 push, and only one address is set. If it is used in other places in the future, multiple destination addresses may be set, and the interface may be adjusted at that time
/**
*Set the destination IP address of RTP Sender. Note that it is currently used in GB2818 push, and only one address is set. If it is used in other places in the future, multiple destination addresses may be set, and the interface may be adjusted at that time
*
* @param rtp_sender_handle, CreateRTPSender Return value
* @param address, IP address
* @param port, port
*
* @return {0} if successful
*/
public native int SetRTPSenderDestination(long rtp_sender_handle, String address, int port);9. Initialize RTP Sender. Call the above interface to configure relevant parameters before initialization
/**
*Initialize RTP Sender. Call the above interface to configure relevant parameters before initialization
*
* @param rtp_sender_handle, CreateRTPSender Return value
*
* @return {0} if successful
*/
public native int InitRTPSender(long rtp_sender_handle);10. Get RTP Sender RTP Socket local port
/** *Get RTP Sender RTP Socket local port * * @param rtp_sender_handle, CreateRTPSender Return value * * @return Failed to return 0. If successful, return the port of response. Please call after InitRTPSender returns successfully. */ public native int GetRTPSenderLocalPort(long rtp_sender_handle);
11. UnInit RTP Sender
/**
* UnInit RTP Sender
*
* @param rtp_sender_handle, CreateRTPSender Return value
*
* @return {0} if successful
*/
public native int UnInitRTPSender(long rtp_sender_handle);12. Release RTP Sender, and then release RTP_ sender_ The handle is invalid. Please don't use it again
/**
* Release RTP Sender. After releasing RTP_ sender_ The handle is invalid. Please don't use it again
*
* @param rtp_sender_handle, CreateRTPSender Return value
*
* @return {0} if successful
*/
public native int DestoryRTPSender(long rtp_sender_handle);GB28181 related interfaces
1. Set GB28181 RTP Sender
/**
* Setting GB28181 RTP Sender
*
* @param rtp_sender_handle, CreateRTPSender Return value
* @param rtp_payload_type, For GB28181 PS, the protocol definition is 96, and the SDP shall prevail
*
* @return {0} if successful
*/
public native int SetGB28181RTPSender(long handle, long rtp_sender_handle, int rtp_payload_type);2. Start GB28181 media streaming
/**
* Start GB28181 media streaming
*
* @return {0} if successful
*/
public native int StartGB28181MediaStream(long handle);3. Stop GB28181 media streaming
/**
* Stop GB28181 media streaming
*
* @return {0} if successful
*/
public native int StopGB28181MediaStream(long handle);Interface call instance
1. Initialization of relevant parameters
/*** GB28181 Relevant parameters can be tested after modifying relevant parameters***/
GBSIPAgent gb28181_agent_ = null;
private int gb28181_sip_local_port_ = 12070;
private String gb28181_sip_server_id_ = "34020000002000000001";
private String gb28181_sip_server_domain_ = "3402000000";
private String gb28181_sip_server_addr_ = "192.168.0.105";
private int gb28181_sip_server_port_ = 15060;
private String gb28181_sip_user_agent_filed_ = "NT GB28181 User Agent V1.0";
private String gb28181_sip_username_ = "31011500991320000069";
private String gb28181_sip_password_ = "12345678";
private int gb28181_reg_expired_ = 3600; // The minimum registration validity time is 3600 seconds
private int gb28181_heartbeat_interval_ = 20; // The heartbeat interval GB28181 is 60 by default, and now adjusted to 20 seconds.
private int gb28181_heartbeat_count_ = 3; // Failure in heartbeat interval of 3 times indicates disconnection from the server
private int gb28181_sip_trans_protocol_ = 0; // 0 indicates that signaling is transmitted by UDP and 1 indicates that signaling is transmitted by TCP
private long gb28181_rtp_sender_handle_ = 0;
private int gb28181_rtp_payload_type_ = 96;
/*** GB28181 Relevant parameters can be tested after modifying relevant parameters***/2. Start or stop GB28181 operation
class ButtonGB28181AgentListener implements OnClickListener {
public void onClick(View v) {
stopGB28181Stream();
destoryRTPSender();
if (null == gb28181_agent_ ) {
if( !initGB28181Agent() )
return;
}
if (gb28181_agent_.isRunning()) {
gb28181_agent_.terminateAllPlays(true);// At present, after sending BYE, some servers will send INVITE immediately. Whether to send BYE depends on the actual situation
gb28181_agent_.stop();
btnGB28181Agent.setText("start-up GB28181");
}
else {
if ( gb28181_agent_.start() ) {
btnGB28181Agent.setText("stop it GB28181");
}
}
}
}3. Implementation of initgb28181agent
private boolean initGB28181Agent()
{
if ( gb28181_agent_ != null )
return true;
String local_ip_addr = IPAddrUtils.getIpAddress(myContext);
Log.i(TAG, "initGB28181Agent local ip addr: " + local_ip_addr);
if ( local_ip_addr == null || local_ip_addr.isEmpty() ) {
Log.e(TAG, "initGB28181Agent local ip is empty");
return false;
}
gb28181_agent_ = GBSIPAgentFactory.getInstance().create();
if ( gb28181_agent_ == null ) {
Log.e(TAG, "initGB28181Agent create agent failed");
return false;
}
gb28181_agent_.addListener(this);
// Required information
gb28181_agent_.setLocalAddressInfo(local_ip_addr, gb28181_sip_local_port_);
gb28181_agent_.setServerParameter(gb28181_sip_server_addr_, gb28181_sip_server_port_, gb28181_sip_server_id_, gb28181_sip_server_domain_);
gb28181_agent_.setUserInfo(gb28181_sip_username_, gb28181_sip_password_);
// Optional parameters
gb28181_agent_.setUserAgent(gb28181_sip_user_agent_filed_);
gb28181_agent_.setTransportProtocol(gb28181_sip_trans_protocol_==0?"UDP":"TCP");
// GB28181 configuration
gb28181_agent_.config(gb28181_reg_expired_, gb28181_heartbeat_interval_, gb28181_heartbeat_count_);
com.gb28181.ntsignalling.Device gb_device = new com.gb28181.ntsignalling.Device("34020000001380000001", "Android test equipment", Build.MANUFACTURER, Build.MODEL,
"universe","Mars 1","Mars", true);
getLocation(this);
gb_device.setLongitude(mLongitude);
gb_device.setLatitude(mLatitude);
gb28181_agent_.addDevice(gb_device);
if (!gb28181_agent_.initialize()) {
gb28181_agent_ = null;
Log.e(TAG, "initGB28181Agent gb28181_agent_.initialize failed.");
return false;
}
return true;
}4. After successful registration, return the time
@Override
public void ntsRegisterOK(String dateString) {
Log.i(TAG, "ntsRegisterOK Date: " + (dateString!= null? dateString : ""));
}5. Registration timeout callback
@Override
public void ntsRegisterTimeout() {
Log.e(TAG, "ntsRegisterTimeout");
}6. Register transport exception callback
@Override
public void ntsRegisterTransportError(String errorInfo) {
Log.e(TAG, "ntsRegisterTransportError error:" + (errorInfo != null?errorInfo :""));
}7. Abnormal heartbeat callback
@Override
public void ntsOnHeartBeatException(int exceptionCount, String lastExceptionInfo) {
Log.e(TAG, "ntsOnHeartBeatException heart beat timeout count reached, count:" + exceptionCount+
", exception info:" + (lastExceptionInfo!=null?lastExceptionInfo:""));
// After 10 milliseconds, stop signaling and restart
handler.postDelayed(new Runnable() {
@Override
public void run() {
Log.i(TAG, "gb28281_heart_beart_timeout");
stopGB28181Stream();
destoryRTPSender();
if (gb28181_agent_ != null) {
Log.i(TAG, "gb28281_heart_beart_timeout sip stop");
gb28181_agent_.stop();
Log.i(TAG, "gb28281_heart_beart_timeout sip start");
gb28181_agent_.start();
}
}
},10);
}8. After invite returns OK, create RTP Sender and set relevant parameters according to the returned information
@Override
public void ntsOnInvitePlay(String deviceId, InvitePlaySessionDescription session_des) {
handler.postDelayed(new Runnable() {
@Override
public void run() {
Log.i(TAG,"ntsInviteReceived, device_id:" +device_id_+", is_tcp:" + session_des_.isRTPOverTCP()
+ " rtp_port:" + session_des_.getMediaPort() + " ssrc:" + session_des_.getSSRC()
+ " address_type:" + session_des_.getAddressType() + " address:" + session_des_.getAddress());
// You can send a temporary ringing response to the signaling server first
//sip_stack_android.respondPlayInvite(180, device_id_);
long rtp_sender_handle = libPublisher.CreateRTPSender(0);
if ( rtp_sender_handle == 0 ) {
gb28181_agent_.respondPlayInvite(488, device_id_);
Log.i(TAG, "ntsInviteReceived CreateRTPSender failed, response 488, device_id:" + device_id_);
return;
}
gb28181_rtp_payload_type_ = session_des_.getPSRtpMapAttribute().getPayloadType();
libPublisher.SetRTPSenderTransportProtocol(rtp_sender_handle, session_des_.isRTPOverUDP()?0:1);
libPublisher.SetRTPSenderIPAddressType(rtp_sender_handle, session_des_.isIPv4()?0:1);
libPublisher.SetRTPSenderLocalPort(rtp_sender_handle, 0);
libPublisher.SetRTPSenderSSRC(rtp_sender_handle, session_des_.getSSRC());
libPublisher.SetRTPSenderSocketSendBuffer(rtp_sender_handle, 2*1024*1024); // Set to 2M
libPublisher.SetRTPSenderClockRate(rtp_sender_handle, session_des_.getPSRtpMapAttribute().getClockRate());
libPublisher.SetRTPSenderDestination(rtp_sender_handle, session_des_.getAddress(), session_des_.getMediaPort());
if ( libPublisher.InitRTPSender(rtp_sender_handle) != 0 ) {
gb28181_agent_.respondPlayInvite(488, device_id_);
libPublisher.DestoryRTPSender(rtp_sender_handle);
return;
}
int local_port = libPublisher.GetRTPSenderLocalPort(rtp_sender_handle);
if (local_port == 0) {
gb28181_agent_.respondPlayInvite(488, device_id_);
libPublisher.DestoryRTPSender(rtp_sender_handle);
return;
}
Log.i(TAG,"get local_port:" + local_port);
String local_ip_addr = IPAddrUtils.getIpAddress(myContext);
gb28181_agent_.respondPlayInviteOK(device_id_,local_ip_addr, local_port);
gb28181_rtp_sender_handle_ = rtp_sender_handle;
}
private String device_id_;
private InvitePlaySessionDescription session_des_;
public Runnable set(String device_id, InvitePlaySessionDescription session_des) {
this.device_id_ = device_id;
this.session_des_ = session_des;
return this;
}
}.set(deviceId, session_des),0);
}9. Cancel playback
@Override
public void ntsOnCancelPlay(String deviceId) {
// Cancel the Play session here
handler.postDelayed(new Runnable() {
@Override
public void run() {
Log.i(TAG, "ntsOnCancelPlay, deviceId=" + device_id_);
destoryRTPSender();
}
private String device_id_;
public Runnable set(String device_id) {
this.device_id_ = device_id;
return this;
}
}.set(deviceId),0);
}10. After receiving the ACK, start sending audio and video data
@Override
public void ntsOnAckPlay(String deviceId) {
handler.postDelayed(new Runnable() {
@Override
public void run() {
Log.i(TAG,"ntsOnACKPlay, device_id:" +device_id_);
if (!isRecording && !isRTSPPublisherRunning && !isPushingRtsp && !isPushingRtmp) {
InitAndSetConfig();
}
libPublisher.SetGB28181RTPSender(publisherHandle, gb28181_rtp_sender_handle_, gb28181_rtp_payload_type_);
int startRet = libPublisher.StartGB28181MediaStream(publisherHandle);
if (startRet != 0) {
if (!isRecording && !isRTSPPublisherRunning && !isPushingRtmp && !isPushingRtsp) {
if (publisherHandle != 0) {
libPublisher.SmartPublisherClose(publisherHandle);
publisherHandle = 0;
}
}
destoryRTPSender();
Log.e(TAG, "Failed to start GB28181 service..");
return;
}
if (!isRecording && !isRTSPPublisherRunning && !isPushingRtsp && !isPushingRtmp) {
if (pushType == 0 || pushType == 1) {
CheckInitAudioRecorder(); //enable pure video publisher..
}
}
isGB28181StreamRunning = true;
}
private String device_id_;
public Runnable set(String device_id) {
this.device_id_ = device_id;
return this;
}
}.set(deviceId),0);
}11. Invite exception handling
@Override
public void ntsOnPlayInviteResponseException(String deviceId, int statusCode, String errorInfo) {
// Here we need to release the response resources
Log.i(TAG, "ntsOnPlayInviteResponseException, deviceId=" + deviceId + " statusCode=" +statusCode
+ " errorInfo:" + errorInfo);
handler.postDelayed(new Runnable() {
@Override
public void run() {
Log.i(TAG, "ntsOnPlayInviteResponseException, deviceId=" + device_id_);
destoryRTPSender();
}
private String device_id_;
public Runnable set(String device_id) {
this.device_id_ = device_id;
return this;
}
}.set(deviceId),0);
}12. Receive Bye and stop sending data
@Override
public void ntsOnByePlay(String deviceId)
{
handler.postDelayed(new Runnable() {
@Override
public void run() {
Log.i(TAG, "ntsOnByePlay, stop GB28181 media stream, deviceId=" + device_id_);
stopGB28181Stream();
destoryRTPSender();
}
private String device_id_;
public Runnable set(String device_id) {
this.device_id_ = device_id;
return this;
}
}.set(deviceId),0);
}13. Play Dialog terminates processing
@Override
public void ntsOnPlayDialogTerminated(String deviceId) {
/*
Play When the conversation corresponding to the session is terminated, the callback will not start. At present, it may start only after the response is 200K but the ACK has not been received after 64*T1 time
Please clean up after receiving this
*/
handler.postDelayed(new Runnable() {
@Override
public void run() {
Log.i(TAG, "ntsOnPlayDialogTerminated, deviceId=" + device_id_);
stopGB28181Stream();
destoryRTPSender();
}
private String device_id_;
public Runnable set(String device_id) {
this.device_id_ = device_id;
return this;
}
}.set(deviceId),0);
}summary
GB28181 design, in addition to supporting TCP and UDP transmission, supports the separation of signaling and data transmission, and can realize the on-demand playback and processing of other terminals for front-end devices without separate signaling support. The disadvantage is that there are not many external servers supporting GB28181. For example, the support of open source SRS server for GB28181 is not commercial enough. We look forward to the subsequent version upgrade.