Chapter 3: protocol transmission between Netty server and client

Based on the content of the previous chapter, we learned how to send string data to the server and output it. In this chapter, we will talk about how to make protocol, transform data content into corresponding format for communication, and realize simple login function.
The link to the previous chapter is here at https://blog.csdn.net/sinat 18538231/article/details/101565251

Agreement making

Generally speaking, protocol is a data format that can be understood by both sides of communication. Before the development, the two sides should unify the protocol format, so that when one side receives the data from the other side, it can analyze according to the protocol format to get the correct data content.
To think about how to make an agreement, we usually need to consider three aspects
Application layer, security layer, transport layer
The application layer controls how data content is packaged into the format we need. There are three main types: text protocol (json, etc.), binary protocol (byte stream), data format protocol (protobuf, etc.);
The security layer controls the encryption of packed data to reduce the possibility of being cracked;
The transmission layer controls the data transmission mode (Tcp or Udp).
In this case, we will not expand the security layer and the transmission layer. We will not encrypt for the moment. The transmission is still tcp. In the application layer, we will expand the use of the most basic binary protocol.

1. Binary protocol

Here we develop a binary protocol format

As shown in the figure, in the format we developed, a binary data package consists of a package header and a package body. The header contains a 16 bit integer data length and a 16 bit integer protocol number cmd. Length helps us to solve the problem of package sticking and unpacking, and ensures the integrity of data. Protocol number cmd helps us distribute the data content in the body to the corresponding parser for data analysis and logical processing.

2. Transform the server

In the previous chapter, we dealt with the data interaction between the client and the server, but the codes and decoders used are the StringEncoder and StringDecoder provided by netty, which are specially used for the data transmission of string type. Since our protocol is self-defined, we also need to customize the coding and decoder.

The following codes can be found at https://github.com/gaaolengyan/asimplegameserver
BinaryDecoder.java protocol decoder

public class BinaryDecoder extends ByteToMessageDecoder {
    @Override
    protected void decode(ChannelHandlerContext channelHandlerContext, ByteBuf byteBuf, List<Object> list) throws Exception {
        if (byteBuf.readableBytes() < 8) {  // Because the protocol header is length:int16+cmd:int16, a total of 32-bit 8 bytes
            return;                         // So when the readable data is less than 8 bits, we do not process it
        }
        short bodyLength = byteBuf.readShortLE();   // Message body length
        int cmd = byteBuf.readShortLE();            // Agreement number
        byte[] body = new byte[bodyLength];         //
        byteBuf.readBytes(body);                    // Read message body
        Pt pt = BinaryRouting.routing(cmd);         // Distribute message body to corresponding protocol parsing class according to protocol number
        if (pt != null){
            HashMap argsMap = pt.decode(cmd, body); // Parse to get the parameter map
            Request request = new Request(cmd, argsMap, channelHandlerContext.channel()); // Package as Request object); / / package as Request object
            list.add(request);                      // Put the parsed data into the result
        }
    }
}

Binaryencoder. Java protocol encoder

public class BinaryEncoder extends MessageToByteEncoder<Response> {
    @Override
    protected void encode(ChannelHandlerContext channelHandlerContext, Response response, ByteBuf byteBuf) throws Exception {
        int cmd = response.getCmd();
        Pt pt = BinaryRouting.routing(cmd);
        System.out.println(response.toString());
        if (pt != null){
            byte[] body = pt.encode(cmd, response.getHashMap());
            byte[] cmdBytes = Protocol.writeInt16(response.getCmd());
            byte[] length = Protocol.writeInt16(body.length);
            byte[] result = Protocol.byteMergerAll(length, cmdBytes, body);
            byteBuf.writeBytes(result);
        }
    }
}

BinaryRouting.java protocol parsing distributor

public class BinaryRouting {
    private static HashMap hashMap = new HashMap<Short, Object>(); // map from protocol number to resolution class instance
    private static BinaryRouting binaryRouting = new BinaryRouting();

    private BinaryRouting(){init();}

    private void init() {
        hashMap.put(100, new Pt100());
    }

    public static Pt routing(int cmd){
        return (Pt) hashMap.get(cmd / 100); // Only the first three bits of cmd are used for mapping
    }
}

Pt.java is a protocol to analyze the interface of packing class, and implement it according to the function

public interface Pt {
    HashMap decode(int cmd, byte[] bytes);
    byte[] encode(int cmd, HashMap hashMap);
}

What we need to pay attention to is the problem of size and end order

public class Protocol {
    private static final boolean LITTLE_ENDIAN = true; // Is it a small end sequence
    // Packing string
    public static byte[] writeString(String str){
        byte[] strBody = str.getBytes(StandardCharsets.UTF_8);
        short length = (short) strBody.length;
        byte[] strLength = writeInt16(length);
        return byteMerger(strLength, strBody);
    }
    // Read string
    public static Pair<String, byte[]> readString(byte[] bytes){
        Pair pair = readInt16(bytes); // String length
        int strLength = (int) pair.getValue0();
        byte[] leftBytes = (byte[]) pair.getValue1();
        byte[] strBytes = new byte[strLength];
        System.arraycopy(leftBytes, 0, strBytes, 0, strLength);
        String str = new String(strBytes);
        byte[] lastBytes = new byte[leftBytes.length - strLength];
        System.arraycopy(leftBytes, strLength, lastBytes, 0, lastBytes.length);
        return Pair.with(str, lastBytes);
    }
    // Packed 16 bit unsigned integer
    public static byte[] writeInt16(int num){
        byte[] bytes = new byte[2];
        for (int i = 0; i < 2; i++) {
            int offset = LITTLE_ENDIAN ? i * 8 : (bytes.length - 1 - i) * 8;
            bytes[i] = (byte) ((num >>> offset) & 0xff);
        }
        return bytes;
    }
    public static Pair readInt16(byte[] bytes){
        if (bytes.length < 2) {
            return Pair.with(0, new byte[0]);
        }else{
            int num;
            if (LITTLE_ENDIAN){
                num = (bytes[0] & 0xFF) | ((bytes[1] & 0xFF) << 8);
            }else{
                num = (bytes[1] & 0xFF) | ((bytes[0] & 0xFF) << 8);
            }
            byte[] leftBytes = new byte[bytes.length - 2];
            System.arraycopy(bytes, 2, leftBytes, 0, bytes.length - 2);
            return Pair.with(num, leftBytes);
        }
    }
    // Packed 32-bit unsigned integer
    public static byte[] writeInt32(long num){
        byte[] bytes = new byte[4];
        for (int i = 0; i < 4; i++) {
            int offset = LITTLE_ENDIAN ? i * 8 : (bytes.length - 1 - i) * 8;
            bytes[i] = (byte) ((num >>> offset) & 0xff);
        }
        return bytes;
    }
    public static Pair readInt32(byte[] bytes){
        if (bytes.length < 4) {
            return Pair.with(0, new byte[0]);
        }else{
            int num;
            if (LITTLE_ENDIAN) {
                num = (int) ((bytes[0] & 0xFF)
                    | ((bytes[1] & 0xFF) << 8)
                    | ((bytes[2] & 0xFF) << 16)
                    | ((bytes[3] & 0xFF) << 24));
            }else{
                num = (int) ((bytes[3] & 0xFF)
                    | ((bytes[2] & 0xFF) << 8)
                    | ((bytes[1] & 0xFF) << 16)
                    | ((bytes[0] & 0xFF) << 24));
            }
            byte[] leftBytes = new byte[bytes.length - 4];
            System.arraycopy(bytes, 4, leftBytes, 0, bytes.length - 4);
            return Pair.with(num, leftBytes);
        }
    }
    // Packed 64 bit unsigned integer
    public static byte[] writeInt64(long num){
        byte[] bytes = new byte[8];
        for (int i = 0; i < 8; i++) {
            int offset = LITTLE_ENDIAN ? i * 8 : (bytes.length - 1 - i) * 8;
            bytes[i] = (byte) ((num >>> offset) & 0xFF);
        }
        return bytes;
    }
    // Read 64 bit unsigned integer
    public static Pair readInt64(byte[] bytes){
        if (bytes.length < 8) {
            return Pair.with(0, new byte[0]);
        }else{
            long num = 0;
            for(int i = 0; i < 8; i ++){
                int offset=(LITTLE_ENDIAN ? i : (7 - i)) << 3;
                num |=((long)0xff<< offset) & ((long)bytes[i] << offset);
            }
            byte[] leftBytes = new byte[bytes.length - 8];
            System.arraycopy(bytes, 8, leftBytes, 0, bytes.length - 8);
            return Pair.with(num, leftBytes);
        }
    }
    // Concatenate 2 byte arrays
    public static byte[] byteMerger(byte[] header, byte[] body){
        byte[] result = new byte[header.length + body.length];
        System.arraycopy(header, 0, result, 0, header.length);
        System.arraycopy(body, 0, result, header.length, body.length);
        return result;
    }
    // Concatenate multiple byte arrays
    public static byte[] byteMergerAll(byte[]... values) {
        int length_byte = 0;
        for (byte[] value : values) {
            length_byte += value.length;
        }
        byte[] all_byte = new byte[length_byte];
        int countLength = 0;
        for (byte[] b : values) {
            System.arraycopy(b, 0, all_byte, countLength, b.length);
            countLength += b.length;
        }
        return all_byte;
    }
}

Pair is a class under the javatuples package. To use it, we need to introduce the javatuples package, which can easily provide tuple like functions in java programs.
Finally, don't forget to modify our netty startup class.

public class Main {
    private static Properties properties = new Properties();
    public static void main(String[] args) throws Exception {
        loadProperties(properties); // Loading configuration
        start();    // Start server
    }
    private static void start() throws InterruptedException {
        EventLoopGroup bossGroup = new NioEventLoopGroup();    // boss object, used to listen for socket connection
        EventLoopGroup workerGroup = new NioEventLoopGroup();  // worker object for data reading, writing and processing logic
        ServerBootstrap bootstrap = new ServerBootstrap();  // Boot class, boot server startup
        bootstrap
                .group(bossGroup, workerGroup)          // Two thread binding
                .channel(NioServerSocketChannel.class)  // Specify IO model as NIO
                .childHandler(new MyChannelInitializer(properties.getProperty("pt_type")))
                .bind(properties.getProperty("ip"), Integer.parseInt(properties.getProperty("port"))).sync();
        System.out.println("Server start!");
        test();
    }
    // load configuration
    private static void loadProperties(Properties p){
        InputStream inputStream = Main.class.getClassLoader().getResourceAsStream("Config.properties");
        try {
            p.load(inputStream);
        } catch (IOException e1) {
            e1.printStackTrace();
        }
    }
//    Protocol test method
    private static void test(){
        int a = 25535;
        byte[] bytes = Protocol.writeInt16(a);
        Pair pair = Protocol.readInt16(bytes);
        System.out.println(pair.getValue0());
    }
}

Here I put some things that were originally written in the code into a properties configuration file, which will not be discussed for the moment.
As you can see, after the netty server is turned on, a test method is added. First call writeInt16 to write the test data, and then call readInt16 to read the data and print it.

3. Operation test

Start the server.
Server output:

You can see that there is no problem with the method of packing and unpacking int16 bit data. Other methods can also be tested as before.

4. Transform the client

Here we make a login agreement

cmd:10001{
	c2s{ // Client to server
		username:string  // User name
		password:string  // Password
	}	
	s2c{  // Server returns to client
		res:int32		// Result code 0 account not registered 1 login succeeded 2 password error
	}
}

Such a protocol means that the client requests to log in through the 10001 protocol number, and the message body is a string type user name and password.
After processing by the server, a 32-bit integer result code is returned to represent the login result.

Here we write a Pt100.java to analyze the data of the server

public class Pt100 implements Pt {
	public HashMap decode(int cmd, byte[] bytes){
		HashMap hashMap = new HashMap();
		if (cmd == 10001){
			Pair args1 = Protocol.readString(bytes);
			String accName = (String) args1.getValue0();
			byte[] leftBytes = (byte[]) args1.getValue1();
			Pair args2 = Protocol.readString(leftBytes);
			String passWord = (String) args2.getValue0();
			hashMap.put(1, accName);
			hashMap.put(2, passWord);
		}
		return hashMap;
	}

	public byte[] encode(int cmd, HashMap hashMap){
        if (cmd == 10001){
			int res = (int) hashMap.get(1);
			return Protocol.writeInt32(res);
		}
		return new byte[0];
	}
}

Pt100.java implements Pt interface. When receiving data, it calls decode method to decode; when returning data, it calls encode method to encode.

In order to make the project structure clearer, the Action interface is implemented by logic processing
Action.java

public interface Action {
    void action(int cmd, Request request);
}

Use an ActionManager to distribute commands
ActionManager.java

public class ActionManager {
    private static ActionManager actionManager = new ActionManager();
    private HashMap<Integer, Action> actionMap = new HashMap<>();
    
    private void init(){
        actionMap.put(100, (Action) new LoginAction()); // Initialize action mapping here
    }

    public void dispatch(Request request) {
        int cmd = request.getCmd();
        Action action = actionMap.get(cmd);
        if(action != null) {
            action.action(cmd, request);
        }
    }

    private ActionManager(){
        init();
    }

    public static ActionManager getInstance() {
        return actionManager;
    }
}

Finally, let's look at the login action
LoginAction.java

public class LoginAction implements Action {
    public void action(int cmd, Request request) {
        Channel channel = request.getChannel();
        HashMap argsMap = request.getHashMap();
        String accName = (String) argsMap.get(1);
        String passWord = (String) argsMap.get(2);
        HashMap responseArgsMap = new HashMap();
        int res;
        // Write the test data here first
        String testName = "GaolengYan";
        String testPassWord = "123456";
        // Determine whether the user name and password are correct
        if (testName.equals(accName) && testPassWord.equals(passWord )){
                res = 1;
                System.out.println("Login succeeded!");
        } else {
            res = 2;
            System.out.println("Wrong password!");
        }
        responseArgsMap.put(1, res);
        Response response = new Response(cmd, responseArgsMap);
        channel.writeAndFlush(response);
    }
}

Open the client project and copy Protocol.java to the client project project. Then modify the code

public class Main {
    public static void main(String[] args) {
        try {
            InetAddress addr;
            Socket socket = new Socket("127.0.0.1", 9999);
            addr = socket.getInetAddress();
            OutputStream outPutStream = socket.getOutputStream();
            System.out.println("connection to" + addr);
            String accName = "GaolengYan";
            String passWord = "123456";
            byte[] accNameBytes = Protocol.writeString(accName);
            byte[] passWordBytes = Protocol.writeString(passWord);
            byte[] body = Protocol.byteMerger(accNameBytes, passWordBytes);
            int bodyLength = body.length;
            int requestId = 10001;
            byte[] length = Protocol.writeInt16(bodyLength);
            byte[] cmd = Protocol.writeInt16(requestId);
            byte[] result = Protocol.byteMergerAll(length, cmd, body);
            outPutStream.write(result);
        } catch (IOException e) {
            System.out.println("cannot connect");
        }
    }

The startup server starts the client. Get the result

The client receives the code of the server in the same way, so it will not be repeated. Interested parties can leave messages or private letters to discuss their progress together.