Flatbuffers for serialization and deserialization (I): preliminary use

1: Foreword

In MNN, a trained static model is serialized by Flatbuffers and stored in the hard disk This brings two problems: 1 Why should model information be serialized and cannot be saved directly 2 Other frameworks such as caffe and onnx are serialized with Protobuf. Why does MNN use Flatbuffers? What are the advantages? Before answering these two questions, we need to understand what serialization and deserialization are

2: What are serialization and deserialization

What is serialization and deserialization:

Serialization refers to turning an instance object into binary content, which is essentially a byte [] array. Why serialize instance objects? After serialization, you can save byte [] to a file or transfer byte [] to a remote via the network. In this way, it is equivalent to storing the instance object in a file or transmitting it through the network. In order to serialize, there is deserialization, that is, to change a binary content (that is, byte [] array) back to the instance object. With deserialization, the byte [] array saved in the file can be "changed back" to the instance object, or read byte [] from the network and "changed back" to the instance object

• Serialization: the process of converting an object into a sequence of bytes.
• Deserialization: the process of restoring a sequence of bytes to an object.

Object serialization has two main purposes:

• Save the byte sequence of the object permanently to the hard disk, usually in a file; (persistent object)
• A sequence of bytes that transfers objects over a network. (network transmission object)

For example, I trained a model in C + +, and then used a class to describe the model in the code:

class net{
private:
    string name;
    vector<layer> layers;
    ...
}

Can I save the whole memory block pointed to by the pointer of this class to the hard disk? When you want to restore, load it directly into memory. Isn't this the fastest? But this will introduce several problems

• The files I saved from the 32-bit machine cannot be restored when I open them on the 64 bit machine, because some types of sizeof are different
• The file size saved directly will be relatively large. In fact, some information can be compressed to reduce the hard disk space occupied
• TODO:

In short, no one will do this. We all use some kind of serialization protocol to convert the objects to be saved into different forms containing the same information, save them to the hard disk or transfer them

3: Why use Flatbuffers

After understanding the necessity of serialization when saving files, we mainly consider the following points when selecting serialization protocol:

• Does the protocol support cross platform
• Speed of serialization
• Serialized size

And Flatbuffers Official website The introduction to myself is as follows:

• Access to serialized data without parsing/unpacking
• Memory efficiency and speed
• Flexible
• Tiny code footprint
• Strongly typed
• Convenient to use
• Cross platform code with no dependencies

In fact, Flatbuffers have the following advantages over Protobuf:

• The biggest feature is faster serialization and deserialization This is because Flatbuffers serializes the data into binary buffers, and then directly reads the buffer according to some offset information when deserializing the data, which is the perfect version of "can I save the whole memory block pointed by the pointer of this class to the hard disk? When you want to restore, load it directly to the memory, which is not the fastest?" Therefore, Flatbuffers are often used for frequent communication with the server in the game, but it feels that when it is used to save and load the neural network model, it should have no obvious advantage over Protobuf, because the access to the weight of the model's load process occupies the main time, while the reduction of the structure time of deserialization model should not significantly accelerate the load process Next time I have time to get some models to test mark
• The utility model has the advantages of small occupation space and simple use, and is suitable for mobile terminals The header file and library file of Protobuf add up to more than ten megabytes, while Flatbuffers only need to include one header file when using, which saves more space At the same time, simplicity is a blessing for beginners
• Add the advantage of both Automatic generation of code Write an fbs or proto file to describe the object structure to be managed, and you can generate all the corresponding cpp class codes with one line of command, which is very easy to manage and modify It can save a lot of hair I think this is the most important reason why these open source neural network frameworks use Protobuf and Flatbuffers

4: How to use Flatbuffers

The following focuses on how to use Flatbuffers to describe a neural network model (actually MNN scheme) and how to serialize, save and deserialize data with C + + code The relevant code of this article has been uploaded to Warehouse Welcome to and star
Details are highly recommended Official documents

4.1 installation

git clone https://github.com/google/flatbuffers
cd flatbuffers
mkdir build
cd build
cmake .. && cmake --build . --target flatc -- -j4

Finally, you can get an executable file of flatc under the directory flatbuffers/build The installation process is done at one go. Compared with the version problems of Protobuf and a bunch of dependency library problems, it's not very comfortable

4.2 writing fbs

Using Flatbuffers is very similar to Protobuf. We will first define a schema file to define the organization relationship of the data structure we want to serialize Let's take a minimalist neural network model PiNet as an example (yes, it's a condensed version of MNN) to introduce the use of the common structures int, string, enum, union, vector and table of Flatbuffers

namespace PiNet;//The namespace must not have the same name as the internal object!!!!!!

table Pool {
    padX:int;
    padY:int;
    // ...
}
table Conv {
    kernelX:int = 1;
    kernelY:int = 1;
    // ...
}
union OpParameter {
    Conv,
    Pool,
}
enum OpType : int {
    Conv,
    Pool,
}
table Op {
    type: OpType;
    parameter: OpParameter;
    name: string;
    inputIndexes: [int];
    outputIndexes: [int];
}
table Net {
    oplists: [Op];
    tensorName: [string];
}
root_type Net;

Our root type is net, which represents a neural network model. Net is a table type. This type should be the most commonly used type, similar to the dictionary in Python. The left side of the colon is the name key and the right side is the data type value [] represents the array vector It can be seen that a net contains multiple layers of OPS and multiple tensor s Then we focus on OP, using an enum to represent the type of OP and a union to represent the parameter of the Op The concept of enum also exists in C/C + +, which is the same here, that is, memory space reuse In fact, it's an int enum instead of an enum A table is defined to describe the parameters of each op. the Conv layer contains only two parameters, kernelX and kernelY Everything else is easy to understand. A little attention should be paid here to the understanding of the concept of union

4.3 generated H file

/flatbuffers/build/flatc net.fbs --cpp --binary --reflect-names --gen-object-api --gen-compare

Convert the prepared fbs file into usable one h documents In the command – Gen object API h file will produce a convenient xxT class-- Gen compare indicates Each class in the h file will generate an Operator = = method to compare whether each object is equal
Let's take a general look at the resulting h what is in the document

enum OpType : int32_t {
  OpType_Conv = 0,
  OpType_Pool = 1,
  OpType_MIN = OpType_Conv,
  OpType_MAX = OpType_Pool
};

The enum in fbs is converted to the enum in C/C + +, which is easy to understand and needless to say

struct Op;
struct OpBuilder;
struct OpT;

The table structures defined in fbs are all changed into structs. Taking Op as an example, three structures are generated, in which Op is used to describe the serialized Op object and OpT is used to describe the non serialized Op object. This XXT is generated only when we add the option - gen object API during compilation

struct OpT : public flatbuffers::NativeTable {
  typedef Op TableType;
  PiNet::OpType type = PiNet::OpType_Conv;
  PiNet::OpParameterUnion parameter{};
  std::string name{};
  std::vector<int32_t> inputIndexes{};
  std::vector<int32_t> outputIndexes{};
};

The structure of OpT is actually what we want to describe Op It can be seen that one of the advantages of using Flatbuffers is convenience. You can automatically generate the corresponding class object code by describing it in a few lines of fbs Aside from serialization, automatic code generation alone is exciting enough. It's a gospel for lazy people

struct Op FLATBUFFERS_FINAL_CLASS : private flatbuffers::Table {
  enum FlatBuffersVTableOffset FLATBUFFERS_VTABLE_UNDERLYING_TYPE {
    VT_TYPE = 4,
    VT_PARAMETER_TYPE = 6,
    VT_PARAMETER = 8,
    VT_NAME = 10,
    VT_INPUTINDEXES = 12,
    VT_OUTPUTINDEXES = 14
  };
  PiNet::OpType type() const ;
  PiNet::OpParameter parameter_type() const ;
  const void *parameter() const ;
  template<typename T> const T *parameter_as() const;
  const PiNet::Conv *parameter_as_Conv() const;
  const PiNet::Pool *parameter_as_Pool() const;
  const flatbuffers::String *name() const ;
  const flatbuffers::Vector<int32_t> *inputIndexes() const ;
  const flatbuffers::Vector<int32_t> *outputIndexes() const ;
  bool Verify(flatbuffers::Verifier &verifier) const ;
  OpT *UnPack() const;
  void UnPackTo() const;
  static flatbuffers::Offset<Op> Pack();
};

Let's look at the structure of op. OP describes the serialized object. The member function mainly includes the method of directly accessing member variables from op (actually deserialization), and an enum of VTableOffset, which will be used in the next detailed explanation. It is not shown here for the time being In addition, it also includes two Pack and UnPack methods, as the name suggests, which are serialization and deserialization methods UnPack can convert the serialized object OP to the non serialized object OpT, and Pack can convert the OpT to Op

struct OpParameterUnion {
  OpParameter type;
  void *value;

  static void *UnPack();
  flatbuffers::Offset<void> Pack() const;

  PiNet::ConvT *AsConv() {
    return type == OpParameter_Conv ?
      reinterpret_cast<PiNet::ConvT *>(value) : nullptr;
  }
  const PiNet::ConvT *AsConv() const {
    return type == OpParameter_Conv ?
      reinterpret_cast<const PiNet::ConvT *>(value) : nullptr;
  }
  PiNet::PoolT *AsPool() {
    return type == OpParameter_Pool ?
      reinterpret_cast<PiNet::PoolT *>(value) : nullptr;
  }
  const PiNet::PoolT *AsPool() const {
    return type == OpParameter_Pool ?
      reinterpret_cast<const PiNet::PoolT *>(value) : nullptr;
  }
};

Let's look at Union, which contains two member variables, a description type and a data pointer For an instantiated Union, if you want to get the data it represents, you need to manually execute the corresponding AsXXX function to cast according to the type Several function parameters contained in the structure of Op just now_ type(), parameter(), parameter_ as_ Conv(), parameter_ as_ Pool () encapsulates the member function of the union

bool operator==(const PoolT &lhs, const PoolT &rhs);
bool operator!=(const PoolT &lhs, const PoolT &rhs);
bool operator==(const ConvT &lhs, const ConvT &rhs);
bool operator!=(const ConvT &lhs, const ConvT &rhs);
bool operator==(const OpT &lhs, const OpT &rhs);
bool operator!=(const OpT &lhs, const OpT &rhs);
bool operator==(const NetT &lhs, const NetT &rhs);
bool operator!=(const NetT &lhs, const NetT &rhs);

After - gen compare is turned on in the compilation option, these comparison operator overloading codes will be generated automatically. Here again shows the convenience of Flatbuffers. Imagine that if I have 100 Op parameters and have to write comparison operator overloading manually, I'm tired to death

4.4 serialization code

#include <fstream>
#include <iostream>

#include "net_generated.h"
using namespace PiNet;

int main() {
    flatbuffers::FlatBufferBuilder builder(1024);

    // table ConvT
    auto ConvT = new PiNet::ConvT;
    ConvT->kernelX = 3;
    ConvT->kernelY = 3;
    // union ConvUnionOpParameter
    OpParameterUnion ConvUnionOpParameter;
    ConvUnionOpParameter.type = OpParameter_Conv;
    ConvUnionOpParameter.value = ConvT;
    // table OpT
    auto ConvTableOpt = new PiNet::OpT;
    ConvTableOpt->name = "Conv";
    ConvTableOpt->inputIndexes = {0};
    ConvTableOpt->outputIndexes = {1};
    ConvTableOpt->type = OpType_Conv;
    ConvTableOpt->parameter = ConvUnionOpParameter;

    // table PoolT
    auto PoolT = new PiNet::PoolT;
    PoolT->padX = 3;
    PoolT->padY = 3;
    // union OpParameterUnion
    OpParameterUnion PoolUnionOpParameter;
    PoolUnionOpParameter.type = OpParameter_Pool;
    PoolUnionOpParameter.value = PoolT;
    // table Opt
    auto PoolTableOpt = new PiNet::OpT;
    PoolTableOpt->name = "Pool";
    PoolTableOpt->inputIndexes = {1};
    PoolTableOpt->outputIndexes = {2};
    PoolTableOpt->type = OpType_Pool;
    PoolTableOpt->parameter = PoolUnionOpParameter;

    // table NetT
    auto netT = new PiNet::NetT;
    netT->oplists.emplace_back(ConvTableOpt);
    netT->oplists.emplace_back(PoolTableOpt);
    netT->tensorName = {"conv_in", "conv_out", "pool_out"};
    netT->outputName = {"pool_out"};
    // table Net
    auto net = CreateNet(builder, netT);
    builder.Finish(net);

    // This must be called after `Finish()`.
    uint8_t* buf = builder.GetBufferPointer();
    int size = builder.GetSize();  // Returns the size of the buffer that
                                   //`GetBufferPointer()` points to.
    std::ofstream output("net.mnn", std::ofstream::binary);
    output.write((const char*)buf, size);

    return 0;
}

Since the – Gen object API option is enabled, the XXT structure will be generated. We only need to assign values to the data structures at all levels. Finally, we only need to Create the root node once to complete the serialization, which is very simple and convenient Compared with the creation of monster on the official website, the data of each level should be created and serialized, and the code structure can be much simplified

4.5 deserialization

#include <fstream>
#include <iostream>
#include <vector>

#include "net_generated.h"
using namespace PiNet;

int main() {
    std::ifstream infile;
    infile.open("net.mnn", std::ios::binary | std::ios::in);
    infile.seekg(0, std::ios::end);
    int length = infile.tellg();
    infile.seekg(0, std::ios::beg);
    char* buffer_pointer = new char[length];
    infile.read(buffer_pointer, length);
    infile.close();

    auto net = GetNet(buffer_pointer);

    auto ConvOp = net->oplists()->Get(0);
    auto ConvOpT = ConvOp->UnPack();

    auto PoolOp = net->oplists()->Get(1);
    auto PoolOpT = PoolOp->UnPack();

    auto inputIndexes = ConvOpT->inputIndexes;
    auto outputIndexes = ConvOpT->outputIndexes;
    auto type = ConvOpT->type;
    std::cout << "inputIndexes: " << inputIndexes[0] << std::endl;
    std::cout << "outputIndexes: " << outputIndexes[0] << std::endl;

    PiNet::OpParameterUnion OpParameterUnion = ConvOpT->parameter;
    switch (OpParameterUnion.type) {
        case OpParameter_Conv: {
            auto ConvOpParameterUnion = OpParameterUnion.AsConv();
            auto k = ConvOpParameterUnion->kernelX;
            std::cout << "ConvOpParameterUnion, k: " << k << std::endl;
            break;
        }
        case OpParameter_Pool: {
            auto PoolOpParameterUnion = OpParameterUnion.AsPool();
            auto k = PoolOpParameterUnion->padX;
            std::cout << "PoolOpParameterUnion, k: " << k << std::endl;
            break;
        }
        default:
            break;
    }
    return 0;
}

5: Summary

The use of Flatbuffers is quite simple. It's good to understand several common data types and read the monster example on the official website several times This article is only a preliminary use, and we will make an in - depth analysis of the next one