brief introduction
Protobuf fer is the abbreviation of Protocol Buffers. It is a data description language developed by Google. It is a portable and efficient structured data storage format, which can be used for structured data serialization or serialization. It is very suitable for data storage or RPC data exchange format. It can be used for language independent, platform independent and extensible serialization structure data format in communication protocol, data storage and other fields.
- protobuf is a data description language (data format) similar to json
- protobuf is very suitable for RPC data exchange format
Advantages and disadvantages
- Advantages:
1: Compared with Json and XML, the volume after serialization is very small, which is suitable for network transmission
2: Support cross platform and multi language
3: The message format upgrade and compatibility are good
4: Serialization and deserialization are fast, faster than Jason's processing speed - Disadvantages:
1: Not widely used (compared with xml and json)
2: Poor readability due to binary format
3: Lack of self description
The overall features are shown in the figure below:
performance comparison
Reference beep beep video data:
install
Method 1
The mac notebook can directly brew install protobuf
In the command protocol -- go_ Out =. / *. proto when compiling the. proto file to go, if an error occurs:
protoc-gen-go: program not found or is not executable Please specify a program using absolute path or make sure the program is available in your PATH system variable --go_out: protoc-gen-go: Plugin failed with status code 1.
It means that the protocol Gen go is not installed, and then install it through the following command
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
Execute the command compilation again if an error occurs
Please specify either:
• a "go_package" option in the .proto source file, or
• a "M" argument on the command line.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.
--go_out: protoc-gen-go: Plugin failed with status code 1.
Cause: because the path cannot be found
terms of settlement:
Don't forget to add option go in the. Proto file_ package =“go/src/microtest/proto”;
go/src/microtest/proto is the absolute path of the project. If it is not added, the above error is reported because the path cannot be found. If you want to output to the current proto directory, you only need to add option go_package = "./"; Just.
Method 2
-
Download protobuf
Method 1:===> git clone https://github.com/protocolbuffers/protobuf.git Method 2:===> Or drag the prepared compressed package into the Unzip to $GOPATH/src/github.com/protocolbuffers/below Unzip protobuf.zip
-
Installation (ubuntu)
(1)Install dependent tools (Networking) $ sudo apt-get install autoconf automake libtool curl make g++ unzip libffi-dev -y (2)get into protobuf file cd protobuf/ (3)Perform installation detection and generate automatic installation scripts ./autogen.sh ./configure (4)Compile C code make (5)Install sudo make install (6)Refresh linux Shared library relationship sudo ldconfig
-
Test protobuf compiler
protoc -h
If relevant instructions are output normally and no error is reported, the installation is successful
-
Install the go language plug-in for protobuf
Since protobuf does not directly support go language, we need to install relevant plug-ins manually
(1)download Method 1:===> go get -v -u github.com/golang/protobuf/proto Method 2:===>Or will github.com-golang-protobuf.zip Drag in to unzip to $GOPATH/src/github.com/golang (2)Enter the folder to compile $ cd $GOPATH/src/github.com/golang/protobuf/protoc-gen-go $ go build (3)To be generated protoc-gen-go Executable file, placed in/bin Directory $ sudo cp protoc-gen-go /bin/ (4)Try to make up protoc-gen-go If you can make up, it means success. If you do not report an error, it means success
protobuf simple syntax
Reference documents (wall climbing required): https://developers.google.com/protocol-buffers/docs/proto3
First, let's look at a very simple example.
syntax = "proto3"; //Specify the version information. If it is not specified, an error will be reported
package pb; //Package name of post generated go file
//Message is a keyword used to define a message type
message Person{
// name
string name = 1;
// Age
int32 age = 2 ;
}
enum test{
int32 age = 0;
}
- The definition (or description) of protobuf messages is usually written in a file ending in. proto.
- The first line of the file specifies that proto3 syntax is being used: if not, the protocol buffer compiler assumes that proto2 is being used. This must also be the first non empty non comment line of the file.
- Package in the second line indicates that it is currently a pb package (the package name of go is consistent with that of go after the go file is generated)
- Finally, the message keyword defines a Person message body, which is similar to the structure in go language. It is a collection containing a series of type data. Many standard simple data types can be used as field types, including bool, int32, float, double, and string. You can also use other message types as field types.
- There is a value member of string type in message, which is encoded with 1 instead of the name. We know that in json, the corresponding data is bound by the name of the member, but Protobuf coding binds the corresponding data by the unique number of the member. Therefore, the volume of Protobuf coded data will be relatively small and can be transmitted quickly. The disadvantage is that it is not conducive to reading.
**message Format description of** The message is composed of at least one field, similar to Go Structure in language, each field has a certain format: ```message Format description //Note format notes should also be written above the content as far as possible (Field modifier) data type field name = Unique number label value;
- Unique number tag: a unique number tag representing each field. It cannot be repeated in the same message. These number tags are used to identify your field in the message binary format, and once the message is defined, it cannot be changed. It should be noted that tags in the range of 1 to 15 are encoded by one byte, so tags 1 to 15 are usually used for frequently occurring message fields. The number tag size ranges from 1 to 2 29 times. 19000-19999 are officially reserved values and cannot be used.
- Comment format: to add comments to the. proto file, you can use the C/C++/java/Go style double slash (/ /) syntax format or / **/
Comparison between common data types of message and those in go
| . proto type | Go type | introduce |
|---|---|---|
| double | float64 | 64 bit floating point number |
| float | float32 | 32-bit floating point number |
| int32 | int32 | Use variable length encoding. Encoding negative numbers is inefficient - if your field may have negative values, use sint32 instead. |
| int64 | int64 | Use variable length encoding. Encoding negative numbers is inefficient - if your field may have negative values, use sint64 instead. |
| uint32 | uint32 | Use variable length encoding. |
| uint64 | uint64 | Use variable length encoding. |
| sint32 | int32 | Use variable length encoding. Symbolic integer value. These are more effective than conventional int32s encoding negative numbers. |
| sint64 | int64 | Use variable length encoding. Symbolic integer value. These are more effective than conventional int64s encoding negative numbers. |
| fixed32 | uint32 | Always four bytes. If the value is usually greater than 228, it is more effective than uint 32 |
| fixed64 | uint64 | Always eight character knots. If the value is usually greater than 256, it is more effective than uint64 |
| sfixed32 | int32 | Always four bytes. |
| sfixed64 | int64 | Always eight character knots. |
| bool | bool | Boolean type |
| string | string | The string must always contain UTF - 8 encoding or 7-Bit ASCII text |
| bytes | []byte | Can contain any sequence of bytes |
protobuf advanced usage
In addition to the above simple types, protobuf also has some complex uses, as follows:
message nesting
In addition to simple data types, messsage can also store other message types, as follows:
syntax = "proto3"; //Specify the version information. If it is not specified, an error will be reported
package pb; //Package name of post generated go file
//Message is a keyword used to define a message type
message Person{
// name
string name = 1;
// Age
int32 age = 2 ;
//Define a message
message PhoneNumber {
string number = 1;
int64 type = 2;
}
PhoneNumber phone = 3;
}
repeated keyword
The repeated keyword is similar to the slice in go. After compilation, the corresponding slice is also the slice of go. The usage is as follows:
syntax = "proto3"; //Specify the version information. If it is not specified, an error will be reported
package pb; //Package name of post generated go file
//Message is a keyword used to define a message type
message Person{
// name
string name = 1;
// Age
int32 age = 2 ;
//Define a message
message PhoneNumber {
string number = 1;
int64 type = 2;
}
repeated PhoneNumber phone = 3;
}
Default value
When parsing data, if the encoded message does not contain a specific singular element, the corresponding field in the parsing object will be set as the default value of the field. Different types have different default values, as follows:
- For strings, the default is an empty string.
- For bytes, the default value is empty bytes.
- For bools, the default value is false.
- For numeric types, the default value is zero.
- For enumerations, the default value is the first defined enumeration value, which must be 0.
- The default value of the repeated field is an empty list
- The default value of the message field is an empty object
enum keyword
When defining a message type, you might want one of the fields to have a predefined list of values. For example, the phone number field has a type, which can be home, work and mobile. We can do this very simply by enum adding constants for each possible value in the message definition. Examples are as follows:
syntax = "proto3"; //Specify the version information. If it is not specified, an error will be reported
package pb; //Package name of post generated go file
//Message is a keyword used to define a message type
message Person{
// name
string name = 1;
// Age
int32 age = 2 ;
//Define a message
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
repeated PhoneNumber phone = 3;
}
//enum is the keyword used to define an enumeration type
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
As mentioned above, the first constant of enum is mapped to 0, and each enumeration definition must contain a constant mapped to zero as its first element. This is because:
- There must be a zero value so that we can use 0 as the numeric default.
- The zero value must be the first element to be compatible with proto2 semantics, where the first enumeration value is always the default value.
enum can also define aliases by specifying the same value for different enumeration constants. If you want to use this function, you must say allow_ If the alias option is set to true, the compiler will report an error. Examples are as follows:
syntax = "proto3"; //Specify the version information. If it is not specified, an error will be reported
package pb; //Package name of post generated go file
//Message is a keyword used to define a message type
message Person{
// name
string name = 1;
// Age
int32 age = 2 ;
//Define a message
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
repeated PhoneNumber phone = 3;
}
//enum is the keyword used to define an enumeration type
enum PhoneType {
//If it is not set, an error will be reported
option allow_alias = true;
MOBILE = 0;
HOME = 1;
WORK = 2;
Personal = 2;
}
oneof keyword
If there is a message containing many fields, and at most one of them can be set at the same time, you can use the oneof function. An example is as follows:
message Person{
// name
string name = 1;
// Age
int32 age = 2 ;
//Define a message
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
repeated PhoneNumber phone = 3;
oneof data{
string school = 5;
int32 score = 6;
}
}
Define RPC services
If you need to use message with RPC, you can define the RPC service interface in the. proto file, and the protobuf compiler will generate the RPC interface code according to the language you choose. Examples are as follows:
//Define RPC services
service HelloService {
rpc Hello (Person)returns (Person);
}
This is the syntax of most protobuf. Others who want to learn can refer to the official documents. After writing the syntax, let's compile it and try it through the code.
protobuf basic compilation
protobuf is compiled through the compiler protoc. Through this compiler, we can generate. proto files into go,Java,Python,C++, Ruby, JavaNano, Objective-C, or c# codes. The generated commands are as follows:
protoc --proto_path=IMPORT_PATH --go_out=DST_DIR path/to/file.proto
- –proto_path=IMPORT_PATH，IMPORT_ Path is the path where the. Proto file is located. If omitted, the current directory will be the default. If there are multiple directories, you can call – proto multiple times_ Path, they will be accessed and imported in sequence.
- –go_out=DST_DIR, which specifies the folder where the generated go language code file is placed
- Allow protocol -- go_ Out =. / *. proto compiles multiple. proto files at one time
- When the go language is compiled, the protobuf compiler will compile the. proto file into a. pd.go file
Generally, we use the following simple commands:
protoc --go_out=./ *.proto
Compile all. proto files in the current folder and place the generated go file in the current folder.
experiment
. proto file
syntax = "proto3";
package pb;
option go_package = "./";
// Message body --- in a package, it is not allowed to define a message body with the same name
message Teacher {
int32 age = 1;
string name = 2;
}
// Define service
service SayName {
rpc SayHello (Teacher) returns (Teacher);
}
Compiled go file
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.27.1-devel
// protoc v3.17.3
// source: person.proto
package __
import (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
// Message body --- in a package, it is not allowed to define a message body with the same name
type Teacher struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Age int32 `protobuf:"varint,1,opt,name=age,proto3" json:"age,omitempty"`
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
}
func (x *Teacher) Reset() {
*x = Teacher{}
if protoimpl.UnsafeEnabled {
mi := &file_person_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *Teacher) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*Teacher) ProtoMessage() {}
func (x *Teacher) ProtoReflect() protoreflect.Message {
mi := &file_person_proto_msgTypes[0]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use Teacher.ProtoReflect.Descriptor instead.
func (*Teacher) Descriptor() ([]byte, []int) {
return file_person_proto_rawDescGZIP(), []int{0}
}
func (x *Teacher) GetAge() int32 {
if x != nil {
return x.Age
}
return 0
}
func (x *Teacher) GetName() string {
if x != nil {
return x.Name
}
return ""
}
var File_person_proto protoreflect.FileDescriptor
var file_person_proto_rawDesc = []byte{
0x0a, 0x0c, 0x70, 0x65, 0x72, 0x73, 0x6f, 0x6e, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x02,
0x70, 0x62, 0x22, 0x2f, 0x0a, 0x07, 0x54, 0x65, 0x61, 0x63, 0x68, 0x65, 0x72, 0x12, 0x10, 0x0a,
0x03, 0x61, 0x67, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x05, 0x52, 0x03, 0x61, 0x67, 0x65, 0x12,
0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e,
0x61, 0x6d, 0x65, 0x32, 0x2f, 0x0a, 0x07, 0x53, 0x61, 0x79, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x24,
0x0a, 0x08, 0x53, 0x61, 0x79, 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x12, 0x0b, 0x2e, 0x70, 0x62, 0x2e,
0x54, 0x65, 0x61, 0x63, 0x68, 0x65, 0x72, 0x1a, 0x0b, 0x2e, 0x70, 0x62, 0x2e, 0x54, 0x65, 0x61,
0x63, 0x68, 0x65, 0x72, 0x42, 0x04, 0x5a, 0x02, 0x2e, 0x2f, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74,
0x6f, 0x33,
}
var (
file_person_proto_rawDescOnce sync.Once
file_person_proto_rawDescData = file_person_proto_rawDesc
)
func file_person_proto_rawDescGZIP() []byte {
file_person_proto_rawDescOnce.Do(func() {
file_person_proto_rawDescData = protoimpl.X.CompressGZIP(file_person_proto_rawDescData)
})
return file_person_proto_rawDescData
}
var file_person_proto_msgTypes = make([]protoimpl.MessageInfo, 1)
var file_person_proto_goTypes = []interface{}{
(*Teacher)(nil), // 0: pb.Teacher
}
var file_person_proto_depIdxs = []int32{
0, // 0: pb.SayName.SayHello:input_type -> pb.Teacher
0, // 1: pb.SayName.SayHello:output_type -> pb.Teacher
1, // [1:2] is the sub-list for method output_type
0, // [0:1] is the sub-list for method input_type
0, // [0:0] is the sub-list for extension type_name
0, // [0:0] is the sub-list for extension extendee
0, // [0:0] is the sub-list for field type_name
}
func init() { file_person_proto_init() }
func file_person_proto_init() {
if File_person_proto != nil {
return
}
if !protoimpl.UnsafeEnabled {
file_person_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*Teacher); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_person_proto_rawDesc,
NumEnums: 0,
NumMessages: 1,
NumExtensions: 0,
NumServices: 1,
},
GoTypes: file_person_proto_goTypes,
DependencyIndexes: file_person_proto_depIdxs,
MessageInfos: file_person_proto_msgTypes,
}.Build()
File_person_proto = out.File
file_person_proto_rawDesc = nil
file_person_proto_goTypes = nil
file_person_proto_depIdxs = nil
}
reference material:
https://developers.google.com/protocol-buffers/docs/reference/go-generated
https://www.bilibili.com/video/BV1po4y1X7hH?p=79&spm_id_from=pageDriver
https://blog.csdn.net/qq_44033530/article/details/115418377