Source Framework Analysis

There are only two parts in the main function to initialize the rtsp service and video encoding program

RtspServer_init

Our idea before writing code is to have the development board as a server and windows as a client. The server must then run, then block listening after initialization like a socket, and wait for the client to connect.

S(server)C(client) mode, basically this structure.

The initialization code is as follows

void RtspServer_init(void)
{
	int i;
	pthread_t threadId = 0;

	memset(&g_rtp_playload,0,sizeof(g_rtp_playload));
	strcpy(&g_rtp_playload,"G726-32");
	g_audio_rate = 8000;
	pthread_mutex_init(&g_sendmutex,NULL);
	pthread_mutex_init(&g_mutex,NULL);
	pthread_cond_init(&g_cond,NULL);
	memset(&g_rtspClients,0,sizeof(RTSP_CLIENT)*MAX_RTSP_CLIENT);
	
	//pthread_create(&g_SendDataThreadId, NULL, SendDataThread, NULL);
	
	struct sched_param thdsched;
	thdsched.sched_priority = 2;
	//to listen visiting
	pthread_create(&threadId, NULL, RtspServerListen, NULL);
	//pthread_setschedparam(threadId,SCHED_RR,&thdsched);
	printf("RTSP:-----Init Rtsp server\n");

	pthread_create(&gs_RtpPid, 0, vdRTPSendThread, NULL);
	
	//exitok++;
}

First, apply for a thread, blind guess is used to listen

Next, the storage request space for payload, starting with RTP, should be a connection between rtsp and rtp. rtsp is actually sent as a RTP package, so it starts with RTP
g_rtp_playload is a 20-byte array followed by G726-32, which is an audio standard payload

G_below Audio_ Rate = 8000; It is the sampling rate of the audio, which is not used in the future. It can be used if you do something related to audio.

Then initialize the global mutex and cond

Then g_rtspClients requests space to store information about connected clients with the following data structure

typedef struct
{
	int index;
	int socket;
	int reqchn;
	int seqnum;
	int seqnum2;
	unsigned int tsvid;
	unsigned int tsaud;
	int status;
	int sessionid;
	int rtpport[2];
	int rtcpport;
	char IP[20];
	char urlPre[PARAM_STRING_MAX];
}RTSP_CLIENT;

Then configure sched_ The param structure appears to be related to the priority of thread scheduling. If there are more threads, when competing for CPU resources, settings are required and the system API can be invoked for thread scheduling.
This is reserved for use in actual code
Pthread_ Setschedparam (threadId, SCHED_RR, &thdsched); Commented out

RtspServer_ The real function called in init is RtspServerListen, which opens a thread to listen on. The socket's to-network byte order, servaddr's configuration, bind, listen are all placed in this thread.

The vdRTPSendThread is used for sending and also has a thread open for doing so

SAMPLE_VENC_720P_CLASSIC

This function is the standard Hays sample, but it adds rtsp content

We changed the sample to one use only and the rate control mode to fixed fixQP

Nothing else has changed, specific reference column Hess 3518E Development Notes Contents beginning with middle 2

The modified part is the sixth step, get the video stream from the venc encoding channel (vb) and save it. Instead, get the video stream without saving it and transfer it directly to the rtsp channel

Instead of storing the 264 stream, send it via rtsp

HI_S32 SAMPLE_COMM_VENC_Sentjin(VENC_STREAM_S *pstStream)
{
    HI_S32 i,flag=0;

    for(i=0;i<MAX_RTSP_CLIENT;i++)//have atleast a connect
    {
		if(g_rtspClients[i].status == RTSP_SENDING)
		{
		    flag = 1;
		    break;
		}
    }
    if(flag)
    {
	    for (i = 0; i < pstStream->u32PackCount; i++)
	    {
		HI_S32 lens=0,j,lastadd=0,newadd=0,showflap=0;
		char sendbuf[320*1024];
		//char tmp[640*1024];
		lens = pstStream->pstPack[i].u32Len-pstStream->pstPack[i].u32Offset;
		memcpy(&sendbuf[0],pstStream->pstPack[i].pu8Addr+pstStream->pstPack[i].u32Offset,lens);
		//printf("lens = %d, count= %d\n",lens,count++);
		VENC_Sent(sendbuf,lens);		
		lens = 0;
	    }

	
    }
    return HI_SUCCESS;
}

In a practical scenario, there may be many rtsp streams, where we use only one macro definition

The following determines the status of the rtsp and sends it if it is in the sending State

Then call VENC_Sent sends the packages ready

Detailed analysis

RtspServer_init

RtspServerListen

Here is the main content to configure port number and modify network byte order in network programming
The port number is 554, which is the default port number for RTSP

setsockopt sets the socket property and SO_REUSEADDR can be used for multiplexing

And then bind and listen

Next to accept
If no one connects, it blocks while's judgment and someone connects and enters while
And print out the connected ip address

Then set up the socket with the option SO_SNDBUF, which sets the size of the send buffer

The status of the newly connected rtspClient is then determined if it is equal to RTSP_IDLE handles this. Because g_rtspClients is a global variable that is automatically initialized to zero after definition, the first in the status enumeration.
In this way, as long as the new ip address is available, it must be idle. After entering if, change its status to connected to avoid repetitive operation

After entering if, the attributes of the accessed client are populated one by one.

memset(&g_rtspClients[i],0,sizeof(RTSP_CLIENT));
g_rtspClients[i].index = i;
g_rtspClients[i].socket = s32CSocket;
g_rtspClients[i].status = RTSP_CONNECTED ;//RTSP_SENDING;
g_rtspClients[i].sessionid = nSessionId++;
strcpy(g_rtspClients[i].IP,inet_ntoa(addrAccept.sin_addr));

After populating the properties, a thread is created to process the current access client.

If the maximum limit is exceeded, then the excess will be covered to the original 0, so that the cycle will be covered (design ideas can also exceed direct disconnection)

The function handled is RtspClientMsg, serving only currently accessed clients
When you look at this code, you need to have some understanding of the RTSP transport protocol. ParseRequestString parses the RTSP header in the function

The RtspClientMsg function goes in and makes an RTSP_CLIENT * pClient accepts incoming structures, not directly to avoid modifying the contents of the original structure
When the connecting client is not in an empty state, receive the data, read-only RTSP_at a time RECV_ SIZE is so large that the unfinished part is left for the next reading. If the number of bytes read is less than 0, it is wrong. Change the state to empty again

Next, a series of local variables are defined as output parameters for ParseRequestString.
The function prototype is as follows

int ParseRequestString(char const* reqStr,
		       unsigned reqStrSize,
		       char* resultCmdName,
		       unsigned resultCmdNameMaxSize,
		       char* resultURLPreSuffix,
		       unsigned resultURLPreSuffixMaxSize,
		       char* resultURLSuffix,
		       unsigned resultURLSuffixMaxSize,
		       char* resultCSeq,
		       unsigned resultCSeqMaxSize) 

The first parameter is the string received by the previous socket, which is the target of the analysis
The second parameter is to accept the size of the string
The third parameter holds the parsed RTSP method
The fourth parameter is the size of the array in which the third parameter is applied before the function executes
The fifth parameter is the resolved URL prefix
The sixth parameter is the size of the array used to hold the fifth parameter
The seventh parameter is the resolved URL suffix
The eighth parameter is the size of the array holding the seventh parameter
The ninth parameter is the parsed session sequence number
The tenth parameter is the size of the array holding the ninth parameter

The above parsed parameters correspond to the RTSP session content, referring in detail to what I wrote earlier RTSP protocol details

Later, we'll take action on the parsed RTSP methods

Next, you'll take a closer look at the ParseRequestString function

ParseRequestString

The purpose of this function is to parse CS interaction information in RTSP protocol, mostly string operations.

First paragraph

// Read everything up to the first space as the command name:
 int parseSucceeded = FALSE;
 unsigned i;
 for (i = 0; i < resultCmdNameMaxSize-1 && i < reqStrSize; ++i) 
 {
   char c = reqStr[i];
   if (c == ' ' || c == '\t') 
   {
     parseSucceeded = TRUE;
     break;
   }

   resultCmdName[i] = c;
 }
 resultCmdName[i] = '\0';
 if (!parseSucceeded) return FALSE;

First, we did a flag to show if the analysis was successful
The obtained data is then segmented, marked by spaces or tabs
This loop gets the method of the RTSP
Since it resolves a string, you need to replace the last character with the end flag of the string

The second paragraph

// Skip over the prefix of any "rtsp://" or "rtsp:/" URL that follows:
  unsigned j = i+1;
  while (j < reqStrSize && (reqStr[j] == ' ' || reqStr[j] == '\t')) ++j; // skip over any additional white space
  for (j = i+1; j < reqStrSize-8; ++j) {
    if ((reqStr[j] == 'r' || reqStr[j] == 'R')
	&& (reqStr[j+1] == 't' || reqStr[j+1] == 'T')
	&& (reqStr[j+2] == 's' || reqStr[j+2] == 'S')
	&& (reqStr[j+3] == 'p' || reqStr[j+3] == 'P')
	&& reqStr[j+4] == ':' && reqStr[j+5] == '/') {
      j += 6;
      if (reqStr[j] == '/') {
	// This is a "rtsp://" URL; skip over the host:port part that follows:
	++j;
	while (j < reqStrSize && reqStr[j] != '/' && reqStr[j] != ' ') ++j;
      } else {
	// This is a "rtsp:/" URL; back up to the "/":
	--j;
      }
      i = j;
      break;
    }
  }

 // Look for the URL suffix (before the following "RTSP/"):
  parseSucceeded = FALSE;
  unsigned k;
  for (k = i+1; k < reqStrSize-5; ++k) {
    if (reqStr[k] == 'R' && reqStr[k+1] == 'T' &&
	reqStr[k+2] == 'S' && reqStr[k+3] == 'P' && reqStr[k+4] == '/') {
      while (--k >= i && reqStr[k] == ' ') {} // go back over all spaces before "RTSP/"
      unsigned k1 = k;
      while (k1 > i && reqStr[k1] != '/' && reqStr[k1] != ' ') --k1;
      // the URL suffix comes from [k1+1,k]

      // Copy "resultURLSuffix":
      if (k - k1 + 1 > resultURLSuffixMaxSize) return FALSE; // there's no room
      unsigned n = 0, k2 = k1+1;
      while (k2 <= k) resultURLSuffix[n++] = reqStr[k2++];
      resultURLSuffix[n] = '\0';

      // Also look for the URL 'pre-suffix' before this:
      unsigned k3 = --k1;
      while (k3 > i && reqStr[k3] != '/' && reqStr[k3] != ' ') --k3;
      // the URL pre-suffix comes from [k3+1,k1]

      // Copy "resultURLPreSuffix":
      if (k1 - k3 + 1 > resultURLPreSuffixMaxSize) return FALSE; // there's no room
      n = 0; k2 = k3+1;
      while (k2 <= k1) resultURLPreSuffix[n++] = reqStr[k2++];
      resultURLPreSuffix[n] = '\0';

      i = k + 7; // to go past " RTSP/"
      parseSucceeded = TRUE;
      break;
    }
  }
  if (!parseSucceeded) return FALSE;

Get the second key piece of information with j first, by offsetting one address back with last last address i
It is then case-compatible to skip rtsp://or rtsp:/
Then find/follow the address that is not/or is the first address of the resolved URL

Paragraph 3

  // Look for "CSeq:", skip whitespace,
  // then read everything up to the next \r or \n as 'CSeq':
  parseSucceeded = FALSE;
  for (j = i; j < reqStrSize-5; ++j) {
    if (reqStr[j] == 'C' && reqStr[j+1] == 'S' && reqStr[j+2] == 'e' &&
	reqStr[j+3] == 'q' && reqStr[j+4] == ':') {
      j += 5;
      unsigned n;
      while (j < reqStrSize && (reqStr[j] ==  ' ' || reqStr[j] == '\t')) ++j;
      for (n = 0; n < resultCSeqMaxSize-1 && j < reqStrSize; ++n,++j) {
	char c = reqStr[j];
	if (c == '\r' || c == '\n') {
	  parseSucceeded = TRUE;
	  break;
	}

	resultCSeq[n] = c;
      }
      resultCSeq[n] = '\0';
      break;
    }
  }
  if (!parseSucceeded) return FALSE;

Used to resolve CSeq fields

Above is the parsing of the request information sent by the client by the development board as a server, which will be processed accordingly after parsing.

OPTIONS

Take OptionAnswer as an example

int OptionAnswer(char *cseq, int sock)
{
	if (sock != 0)
	{
		char buf[1024];
		memset(buf,0,1024);
		char *pTemp = buf;
		pTemp += sprintf(pTemp,"RTSP/1.0 200 OK\r\nCSeq: %s\r\n%sPublic: %s\r\n\r\n",
			cseq,dateHeader(),"OPTIONS,DESCRIBE,SETUP,PLAY,PAUSE,TEARDOWN");
	
		int reg = send(sock, buf,strlen(buf),0);
		if(reg <= 0)
		{
			return FALSE;
		}
		else
		{
			printf(">>>>>%s\n",buf);
		}
		return TRUE;
	}
	return FALSE;
}

When the server receives the client's request, it returns information to the client
The returned information is placed in the buf and sent over the socket channel
Returned information follows RTSP rules

DESCRIBE

int DescribeAnswer(char *cseq,int sock,char * urlSuffix,char* recvbuf)
{
	if (sock != 0)
	{
		char sdpMsg[1024];
		char buf[2048];
		memset(buf,0,2048);
		memset(sdpMsg,0,1024);
		char*localip;
		localip = GetLocalIP(sock);
		
		char *pTemp = buf;
		pTemp += sprintf(pTemp,"RTSP/1.0 200 OK\r\nCSeq: %s\r\n",cseq);
		pTemp += sprintf(pTemp,"%s",dateHeader());
		pTemp += sprintf(pTemp,"Content-Type: application/sdp\r\n");
		

		char *pTemp2 = sdpMsg;
		pTemp2 += sprintf(pTemp2,"v=0\r\n");
		pTemp2 += sprintf(pTemp2,"o=StreamingServer 3331435948 1116907222000 IN IP4 %s\r\n",localip);
		pTemp2 += sprintf(pTemp2,"s=H.264\r\n");
		pTemp2 += sprintf(pTemp2,"c=IN IP4 0.0.0.0\r\n");
		pTemp2 += sprintf(pTemp2,"t=0 0\r\n");
		pTemp2 += sprintf(pTemp2,"a=control:*\r\n");
		

		/*H264 TrackID=0 RTP_PT 96*/
		pTemp2 += sprintf(pTemp2,"m=video 0 RTP/AVP 96\r\n");
		pTemp2 += sprintf(pTemp2,"a=control:trackID=0\r\n");
		pTemp2 += sprintf(pTemp2,"a=rtpmap:96 H264/90000\r\n");
		pTemp2 += sprintf(pTemp2,"a=fmtp:96 packetization-mode=1; sprop-parameter-sets=%s\r\n", "AAABBCCC");
#if 1
		
		/*G726*/
		
		pTemp2 += sprintf(pTemp2,"m=audio 0 RTP/AVP 97\r\n");
		pTemp2 += sprintf(pTemp2,"a=control:trackID=1\r\n");
		if(strcmp(g_rtp_playload,"AAC")==0)
		{
			pTemp2 += sprintf(pTemp2,"a=rtpmap:97 MPEG4-GENERIC/%d/2\r\n",16000);
			pTemp2 += sprintf(pTemp2,"a=fmtp:97 streamtype=5;profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1410\r\n");
		}
		else
		{
			pTemp2 += sprintf(pTemp2,"a=rtpmap:97 G726-32/%d/1\r\n",8000);
			pTemp2 += sprintf(pTemp2,"a=fmtp:97 packetization-mode=1\r\n");
		}	
#endif
		pTemp += sprintf(pTemp,"Content-length: %d\r\n", strlen(sdpMsg));     
		pTemp += sprintf(pTemp,"Content-Base: rtsp://%s/%s/\r\n\r\n",localip,urlSuffix);
		
		//printf("mem ready\n");
		strcat(pTemp, sdpMsg);
		free(localip);
		//printf("Describe ready sent\n");
		int re = send(sock, buf, strlen(buf),0);
		if(re <= 0)
		{
			return FALSE;
		}
		else
		{
			printf(">>>>>%s\n",buf);
		}
	}

	return TRUE;
}

Incoming parameters

• cseq - Sequence number of the session
• sock - network channel established by socket
• urlSuffix - Resolved URL String
• recvbuf - The entire message the server receives

Function, first get the local IP address of the server
Next, reply according to the DESCRIBE standard of RTSP

In RTP protocol, using VLC for playback requires an SDP file to be parsed for playback. RTP transmits bare streams without control information and requires local configuration.
In RTSP protocol, SDP is included in the interactive information. No need to open SDP file with VLC. By parsing the interactive information, the client can know how to parse the video stream. The code is shown below.

char *pTemp2 = sdpMsg;
pTemp2 += sprintf(pTemp2,"v=0\r\n");
pTemp2 += sprintf(pTemp2,"o=StreamingServer 3331435948 1116907222000 IN IP4 %s\r\n",localip);
pTemp2 += sprintf(pTemp2,"s=H.264\r\n");
pTemp2 += sprintf(pTemp2,"c=IN IP4 0.0.0.0\r\n");
pTemp2 += sprintf(pTemp2,"t=0 0\r\n");
pTemp2 += sprintf(pTemp2,"a=control:*\r\n");

PLAY

int PlayAnswer(char *cseq, int sock,int SessionId,char* urlPre,char* recvbuf)
{
	if (sock != 0)
	{
		char buf[1024];
		memset(buf,0,1024);
		char *pTemp = buf;
		char*localip;
		localip = GetLocalIP(sock);
		pTemp += sprintf(pTemp,"RTSP/1.0 200 OK\r\nCSeq: %s\r\n%sRange: npt=0.000-\r\nSession: %d\r\nRTP-Info: url=rtsp://%s/%s;seq=0\r\n\r\n",
			cseq,dateHeader(),SessionId,localip,urlPre);

		free(localip);
		
		int reg = send(sock, buf,strlen(buf),0);
		if(reg <= 0)
		{
			return FALSE;
		}
		else
		{
			printf(">>>>>%s",buf);
			udpfd = socket(AF_INET,SOCK_DGRAM,0);//UDP
			struct sockaddr_in server;
			server.sin_family=AF_INET;
		   	server.sin_port=htons(g_rtspClients[0].rtpport[0]);          
		   	server.sin_addr.s_addr=inet_addr(g_rtspClients[0].IP);
			connect(udpfd,(struct sockaddr *)&server,sizeof(server));
    		printf("udp up\n");
		}
		return TRUE;
	}
	return FALSE;
}

After the client play requests, the server opens a UDP SOCKET channel for bare-stream transmission, and the command goes through the TCP channel. This is the design of the RTSP

UDP is used because real-time is considered

vdRTPSendThread

HI_VOID* vdRTPSendThread(HI_VOID *p)
{
	while(1)
	{
		if(!list_empty(&RTPbuf_head))
		{
			
			RTPbuf_s *p = get_first_item(&RTPbuf_head,RTPbuf_s,list);
			VENC_Sent(p->buf,p->len);
			list_del(&(p->list));
			free(p->buf);
			free(p);
			p = NULL;
			count--;
			//printf("count = %d\n",count);
		
		}
		usleep(5000);
	}
}

After the client connects, the server receives the play instruction to create a bare-flow UDP channel

Sending occurs in the vdRTPSendThread thread. Every 5 subtle in RTPbuf_ Look for data in the headlist and send it whenever there is data

get_first_item is used to remove the first non-empty node from the list
Then pass the contents of the node through VENC_Sent sent out, the main purpose of the function is to add headers to H264 RTP packets
Release it after sending
Here send is the consumer, constantly sending data from the list

Where there are consumers there are producers, where production is coded

There are two ways to send, one is to send directly, the other is to send in a ring buffer

The direct send method is to encode one frame and send one. The direct send method is not in the vdRTPSendThread thread, but in the encoded place, sending one frame at the end of a frame

HI_S32 SAMPLE_COMM_VENC_Sentjin(VENC_STREAM_S *pstStream)
{
    HI_S32 i,flag=0;

    for(i=0;i<MAX_RTSP_CLIENT;i++)//have atleast a connect
    {
		if(g_rtspClients[i].status == RTSP_SENDING)
		{
		    flag = 1;
		    break;
		}
    }
    if(flag)
    {
	    for (i = 0; i < pstStream->u32PackCount; i++)
	    {
		HI_S32 lens=0,j,lastadd=0,newadd=0,showflap=0;
		char sendbuf[320*1024];
		//char tmp[640*1024];
		lens = pstStream->pstPack[i].u32Len-pstStream->pstPack[i].u32Offset;
		memcpy(&sendbuf[0],pstStream->pstPack[i].pu8Addr+pstStream->pstPack[i].u32Offset,lens);
		//printf("lens = %d, count= %d\n",lens,count++);
		VENC_Sent(sendbuf,lens);		
		lens = 0;
	    }

	
    }
    return HI_SUCCESS;
}

The advantage of sending directly is that it is simple, but not in practical use. In reality, it is more complex, and encoding and transmission speeds may not necessarily match.
If the sending speed is faster than the encoding speed block, then the sending has to wait for encoding. Time will be redundant

Ring buffer Can have a certain buffer. If the sending is faster than encoding, it becomes blocked, like a video buffer; If the encoding is faster than sending, or even sending a loop, you can increase the ring buffer, but if there is a crash in the evening, consider the design of the system if the encoding is faster.
It is uncertain who is fast or slow, for example, network fluctuations can affect the sending speed. Sometimes the transmission is slow, so the encoded frames are ready for alignment; Sometimes the sending is faster, then the number of frames waiting in the queue after encoding is less.

The code sent by the ring buffer is as follows

HI_S32 saveStream(VENC_STREAM_S *pstStream)
{
    HI_S32 i,j,lens=0;

    for(j=0;j<MAX_RTSP_CLIENT;j++)//have atleast a connect
    {
		if(g_rtspClients[j].status == RTSP_SENDING)
		{
		    for (i = 0; i < pstStream->u32PackCount; i++)
		    {
				RTPbuf_s *p = (RTPbuf_s *)malloc(sizeof(RTPbuf_s));
				INIT_LIST_HEAD(&(p->list));

				lens = pstStream->pstPack[i].u32Len-pstStream->pstPack[i].u32Offset;
				p->buf = (char *)malloc(lens);
				p->len = lens;
				memcpy(p->buf,pstStream->pstPack[i].pu8Addr+pstStream->pstPack[i].u32Offset,lens);

				list_add_tail(&(p->list),&RTPbuf_head);
				count++;
				//printf("count = %d\n",count);
		    }
    	}
    }

    return HI_SUCCESS;
}

Ring buffer s are implemented through a chain table
Counts are used to record the number of frames remaining encoded and then subtract them if sent. A value of 0 indicates that the encoded frames are exhausted.
This is the producer mentioned above, sending the encoded frames to the global list of chains sent by vdRTPSendThread
After sending a node, remove it and free up memory