background

At the beginning of the year, HttpRunner3 framework was implemented in the team. A brief introduction: HttpRunner is an open-source general testing framework for HTTP(S) protocol developed by python. The use case script is in YAML/JSON format, and version 3.0 supports py format.

HttpRunner relies on open source libraries requests ，pytest ，pydantic ，allure And locust , it can realize various test requirements such as automatic test, performance test, online monitoring, continuous integration and so on.

PB refers to Protocol Buffers, or protobuffer for short. It is a binary data exchange format launched by Google (similar to json and xml, but lighter).

Protobuf has its own compiler, which is called protoc in Linux and can be interpreted proto file and claim the source file of the corresponding language. At present, it supports multiple languages, githab link , there are many tutorials on how to access PB online, which will not be discussed here.

problem

At present, the front and rear interfaces of the company have been connected to PB, the json input parameters of the old interface remain unchanged temporarily, and the new interface adopts PB format, but HttpRunner does not support PB format input parameters, so the new interface test cannot be carried out.

Solution

If it's the first time, I must be confused, so I collect information:

1. pb is a Google serialization protocol, which can be simply understood as base64 encoding and decoding. The difference is that the parsing rules are defined by ourselves, and the rules are proto file (in fact, a. porto file is more appropriate as an interface document).
2. A proto file is a parsing rule, and a message in the file is structured data, which corresponds to the structure of an interface request or the structure of an output parameter. It can also be abstractly understood as defining a class, enum is enumeration, with a digital number as the primary key, and so on.
3. The proto file has nothing to do with the language. To parse other languages, you need the compiler tool of the corresponding language to compile the proto file into the target language, and python will compile it into something similar to xxx_pb2.py file.
4. Using the example, you can install proto (pip install protobuf, import google.protobuf for verification, ok if no error is reported), Call method instance , the json format is as follows

google.protobuf.json_format 
Include for JSON Routine for printing protocol messages in format.
Simple use example:
# Create a proto Object and serialize it as json Format string.
message_object= my_proto_pb2.MyMessage(foo='bar')    
json_string = json_format.MessageToJson(message_object)
# Parse a json Format string to proto Object.
message = json_format.Parse(json_string, my_proto_pb2.MyMessage())   --my_proto_pb2 After compilation pb Documents, MyMessage()This is the corresponding message Method name

The conclusions are as follows:

1. Front - end and back - end developers write interface documents, that is proto file (it has nothing to do with the front and back languages here, and uses PB format syntax). A project can be divided according to services, and each service may contain multiple proto files, each proto files may also correspond to multiple interfaces.
2. The front and back end uses the official compiler to compile each proto file to generate the language package you use (for example, python is xxx_pb2.py).
3. With xxx_pb2.py file, just reference Google protobuf. json_ Format package, call the corresponding methods, and for a specific message, you can parse the JSON string into PB binary format.

So my solution: first, pass in the interface to call pb; 2, Find the corresponding interface pb2 file according to the input parameters; 3, Analyze the input parameter data of the interface; 4, Return the input parameter of the replacement request;

First step

1. The interface request of HttpRunner has pre-processing, so you only need to write a common method to serialize json into PB in the debugtalk file.
2. In order to make debugtalk simpler, you can write a single conversion class to handle PB serialization, and the above method refers to this conversion class.
3. This transformation class needs to provide at least two methods for parsing json into PB and serializing PB into json.
4. Other logging, filtering and encryption methods may be required (depending on the actual situation).

Step 2

Question 1: as mentioned earlier, don't care proto file, that XXX_ pb2. How did py get here?

Fortunately, our front-end staff did the conversion project and updated the project every time proto files will be uploaded to gitlab and can be pulled according to their own language.

Question 2: now_ The pb2 file is available, but proto must have multiple interfaces (corresponding to multiple messages), and the request input parameters of each interface in HttpRunner need to be resolved. How to find the corresponding message according to the interface name?

The front end also makes a json file (index.json), which contains the message information associated with the interface, so you can find this file. The file content style is as follows.

So:

1. The conversion class needs to be imported_ pb2.py files and json files are not the same project, so you need to use the sub module function of git.
2. The transformation class needs to implement a lookup method, input the request url of the interface, and find the corresponding interface message in the json file.

Step 3

Focusing on this step, we need to implement two methods: parsing json into PB and serializing PB into json

1. According to the above call example, both methods require three input parameters: json string, pb2 file name and message name.
2. Since each interface has a corresponding pb2 file, this is also a variable and can be spliced with the path of the interface.
3. The two methods should return the corresponding string. Note that PB is in byte stream bytes format.

Step 4

Basically, the third step has completed the conversion of classes. This step is mainly for the debugtalk method

1. When the common method of serializing json into PB backfill data in debugtalk, you need to select from_ Data format (only the from_data format of post supports byte stream bytes format).
2. If interface signature is involved, methods can be added according to the actual situation.

code implementation

The above idea is also thinking slowly in the process of writing. At present, the implementation project has been running for some time, mainly because the second step of internal conversion project has been realized, which saves a lot of things.

The following is the implementation code, which may have defects. Students are welcome to correct.

debugtalk call method

Request is the built-in request object of HttpRunner, which can preprocess the request

def json_proto(request):
    """
    serialize request Input parameter of json
    :param request: Interface request object
    :return: Serialized interface request object
    """
    if request["method"] == "POST":
        if 'data' in request and request["data"]:
            origin_json = json.dumps(request["data"]).replace("'", "\"")
            print("INF: original json by", origin_json)
            request["data"] = ProtoDataFormat().get_proto_data(request["url"], origin_json, "request")
            print("INF: last json by", request["data"])
    if request['method'] == 'GET':
        if 'params' in request and request["params"]:
            origin_json = json.dumps(request["params"]).replace("'", "\"")
            print("INF: original json by", origin_json)
            params = ProtoDataFormat().get_proto_data(request["url"], origin_json, "request")
            request['params'] =  "bbValue=" + params
            print("INF: last json by", request["params"])
    return request　

Conversion class

# coding: utf-8
import base64
import importlib
import google.protobuf.json_format as json_format
import json
import re
import sys

sys.path.append("./subModuleForPB/b-python")

class ProtoDataFormat:
    def __init__(self):
        self.pwd = r"./subModuleForPB/b-js/mock/index.json"
        try:
            self.read_json(self.pwd)
        except:
            print("Current directory none index.json，Try to change pwd Path properties")


    def get_proto_data(self, url, paramer, type):
        """
        The main functions are json Format data conversion pb format
        :param url: Incoming interface Url，Remove domain name IP address
        :param paramer: Incoming json Original string value in format
        :return: json Data conversion pb Protocol format and Base64 Data
        """
        print("INF: json String start preprocessing", paramer)
        if type =="request":
            type = "requestMessage"
        elif type =="response":
            type = "responseMessage"
        else:
            return "type wrongful"
        # Compatible url / path characters
        # if url[0] =="/":
        #     url = url[1:]
        # Find the message of the corresponding url
        pattern1 = ".*{(.*?), \"url\": \"" + url + "\""
        # Read index json file, converted to json string
        load_str = self.read_json(self.pwd)
        # print("INF: json string start preprocessing 2")
        # Extract message information from json
        relt = self.re_str(pattern1, load_str)
        if relt:
            proto_message = relt[type]
        else:
            print("Error: extract json Medium message Information failure ", relt)
        # Find the path of pb2 file
        pattern2 = ".*{(.*?), \"name\": \"" + proto_message + "\""
        # Extract the path information of pb2 file in json
        path = self.re_str(pattern2, load_str)
        if path:
            # Modify the path of pb2 file to import module format
            mod_path = self.path_module(path["path"])
        else:
            print("Error: extract json Medium pb2 Document path Information failure ", path)
        print("INF: json String preprocessing completed")
        return self.json_proto_base64(paramer, mod_path, proto_message[17:])


    def get_json_data(self, url, paramer, type):
        """
        The main functions are json Format data conversion pb format
        :param url: Incoming interface Url，Remove domain name IP address
        :param paramer: Incoming pb Original string value in format
        :return: pb Protocol format data conversion json Data
        """
        if type =="request":
            type = "requestMessage"
        elif type =="response":
            type = "responseMessage"
        else:
            return "type wrongful"
        # Find the message of the corresponding url
        pattern1 = ".*{(.*?), \"url\": \"" + url + "\""
        # Read index json file, converted to json string
        load_str = self.read_json(self.pwd)
        # Extract message information from json
        relt = self.re_str(pattern1, load_str)
        if relt:
            proto_message = relt[type]
        else:
            print("Error: extract json Medium message Information failure ", relt)
        # Find the path of pb2 file
        pattern2 = ".*{(.*?), \"name\": \"" + proto_message + "\""
        # Extract the path information of pb2 file in json
        path = self.re_str(pattern2, load_str)
        if path:
            # Modify the path of pb2 file to import module format
            mod_path = self.path_module(path["path"])
        else:
            print("Error: extract json Medium pb2 Document path Information failure ", path)
        paramer = base64.b64decode(paramer)

        return self.proto_json(paramer, mod_path, proto_message[17:])

    def proto_json(self, orginjson, path, message):
        """
        The main functions are pb Format data conversion json format
        :param orginjson: Incoming json Original string value in format
        :param path: To import model Path, especially xxx_py2 file
        :param message: message，Specifically xxx_py2 Corresponding interface in the file message
        :return: pb Protocol format converted to json data
        For example: message = my_proto_pb2.MyMessage(foo='bar') json_string = json_format.MessageToJson(message)
        """
        try:
            foo = importlib.import_module(path)
        except:
            raise ModuleNotFoundError("error: Module import failed, trying to modify the source file sys.path of b-python Folder path")
        fun = eval("foo." + message)
        mes = fun()

        mes.ParseFromString(orginjson)
        return json_format.MessageToJson(mes)

    def json_proto_base64(self, orginjson, path, message):
        """
        The main functions are json Format data conversion pb format
        :param orginjson: Incoming json Original string value in format
        :param path: To import model Path, especially xxx_py2 file
        :return: json Data conversion pb Protocol format and Base64 Data
        """
        try:
            foo = importlib.import_module(path)
        except:
            print("Error: Module import failed, trying to modify the source file sys.path of b-python Folder path")
        fun = eval("foo."+ message)
        print("INF: json String start conversion PB")
        try:
            mes = json_format.Parse(orginjson, fun())
            buffer = mes.SerializeToString()
        except:
             raise Exception("Error: json Formatting failed, please check the input parameter format")
        else:
            print("INF: json_proto_base64 Main function execution passed")
            return base64.b64encode(buffer).decode(encoding = "utf-8")

    def read_json(self, pwd):
        """
        read index.json File and return the corresponding json character string
        :param pwd: Incoming index.json File path
        :return: index.json content json All data of string
        """
        with open(pwd, "r") as load_f:
            load_dict = json.load(load_f)
        return json.dumps(load_dict)

    def re_str(self, pattern, str):
        """
        json Get the corresponding string of the interface according to the regular string message information
        :param pattern: Regular expression passed in
        :param str: Original string to find
        :return: Interface corresponding message Dictionary, such as{"name": "DecrVirtual","requestMessage": "billion.protobuf.BDecrVirtualRequest","responseMessage": ".google.protobuf.Empty","method": "POST"}
        """
        search_obj = re.search(pattern, str)
        lis = []
        if search_obj:
            for i in search_obj.groups():
                i = "{"+ i +"}"
                lis.append(i)
        if lis:
            dic = json.loads(lis[0])
            return dic
        else:
            return {}

    def path_module(self, path):
        """
        modify path For module path, replace"\"by"."，add"_pb2"
        :param path: index.json Interface file path in
        :return: Can be imported xxx_pb2 Document module route
        """
        path = str(path).strip("/")
        new_path = path.replace("/",".")
        new_path = new_path[:-5]
        return new_path + "_pb2"


if __name__ == '__main__':
    """
    call get_json_data() Method completion pb Protocol data conversion json format
    call get_proto_data() Method completion json Convert data in format to pb Protocol data
    """
    orginjson = '{"pageInfo": {"pageNo": 1, "pageSize": 10}, "topicInfo": {}, "searchType": "B_SEARCH_TYPE_NEW"}'
    apiUrl = "/fleaTopic/topic/v1/releaseTopic"
    # pbvalue = "Ci0QATABUicKFBIM57u/6Imy6aOf5ZOBIAEwAzgDEgcoATDuBUAKIgYKBG51bGw="
    # jp.pwd = r"D:\b-js\mock\index.json"
    type = "request"
    bbvalue = ProtoDataFormat().get_proto_data(apiUrl, orginjson, type)
    print(bbvalue)
    # bbjson = ProtoDataFormat().get_json_data(apiUrl, pbvalue, type)
    # print(bbjson)