Using git kit micro service framework to realize an application that supports http and grpc services at the same time. Take one of the most common article services as an example to start the tutorial!
Project shelf
Introduction to go kit three-tier model
Go kit is an open source collection of golang microservice tools. Go kit provides a three-tier model from top to bottom, including Transport layer, Endpoint layer and Service layer.
- Transport layer: handles the logic related to HTTP, gRPC, Thrift and other protocols, mainly decoding the request and encoding the response;
- Endpoint layer: as the business middleware in the upper layer of Service, it can use current limiting, fusing, monitoring and other capabilities;
- Service layer: used to process business logic;
Project initialization
thank FengGeSe/demo The project provides a good project demo. This tutorial is based on the transformation of this warehouse. The tutorial code is here git-kit-demo.
FengGeSe/demo project adds Server and Router layers in the go kit three-tier model. The former is started as a service and the latter is forwarded as a route.
The data Model is used as a "neutral" data format and is compatible with multi protocol requests.
eg:
When an http request arrives, the json data will be converted to model, and the model will be converted to json when responding.
When a grpc request arrives, the protobuf data will be converted to model. When the response is received, the model will be converted to protobuf.
Project directory structure
. ├── README.md ├── cmd // Provide access to client and server │ ├── client │ └── server ├── conf // Configuration related ├── endpoint // endpoint layer │ └── article ├── errors // error handling ├── go.mod ├── go.sum ├── params // model layer (represented by params in code) │ └── article ├── pb // pb layer │ └── article ├── router // Routing layer. Where grpc and http register routes │ ├── grpc │ └── http ├── server // server layer, where the service is started │ ├── grpc │ └── http ├── service // service layer, where business logic is processed │ └── article ├── static // Document, document picture related │ └── img ├── transport // transport, where data is converted │ ├── grpc │ └── http ├── util // Tool method └── vendor // Tripartite dependence
Let's go to development!
Service development
Define interface
The Service layer is used to process business logic. A Service consists of many functional methods. Take the most familiar article Service for example, it will provide the function of adding, deleting, modifying and checking.
Define the interface. Define the ArticleService interface and specify the methods to be provided by the Service.
// service/article.go
package service
import (
"context"
"demo/params/article_param"
"fmt"
)
type ArticleService interface {
Create (ctx context.Context, req *article_param.CreateReq) (*article_param.CreateResp, error)
Detail (ctx context.Context, req *article_param.DetailReq) (*article_param.DetailResp, error)
}Define data model
To implement a method, we must first think about its data model and clarify its input and output parameters. In order to distinguish the model layer of orm, in the code implementation, the data model is called params, which mainly defines the incoming and outgoing parameters of a request.
// params/article_param/article.go
package article_param
type CreateReq struct {
Title string `json:"title"`
Content string `json:"content"`
CateId int64 `json:"cate_id"`
}
type CreateResp struct {
Id int64 `json:"id"`
}
type DetailReq struct {
Id int64 `json:"id"`
}
type DetailResp struct {
Id int64 `json:"id"`
Title string `json:"title"`
Content string `json:"content"`
CateId int64 `json:"cate_id"`
UserId int64 `json:"user_id"`
}Service implementation
All methods of articleservice interface planning are implemented by defining an articleservice structure. And expose the implemented Service through the NewArticleService method.
package service
import (
"context"
"demo/params/article_param"
"fmt"
)
// ArticleService defines the article service interface and specifies the methods to be provided by this service
type ArticleService interface {
Create (ctx context.Context, req *article_param.CreateReq) (*article_param.CreateResp, error)
Detail (ctx context.Context, req *article_param.DetailReq) (*article_param.DetailResp, error)
}
// NewArticleService new service
func NewArticleService() ArticleService {
var svc = &articleService{}
{
// middleware
}
return svc
}
// Define the article service structure and implement all methods of the article service interface
type articleService struct {}
func (s *articleService) Create (ctx context.Context, req *article_param.CreateReq) (*article_param.CreateResp, error) {
fmt.Printf("req:%#v\n", req)
// mock: insert generates Id based on the input parameter req
id := 1
return &article_param.CreateResp{
Id: int64(id),
}, nil
}
func (s *articleService) Detail (ctx context.Context, req *article_param.DetailReq) (*article_param.DetailResp, error) {
......
}Taking the method of creating an article as an example, a method obtains the input parameters defined by the model layer, then executes the specific logic, and finally returns the output parameters defined by the model layer.
It is easy for a web developer to think that the parameters returned by an http response should be in json format, which is different here. The response here is struct defined by mdoel layer. It is a neutral data structure, which is only related to the development language. The json format is coupled with http services. Not all services use json to transfer data. Like grpc services, protobuf is generally used as the data format. Therefore, the input and output parameters of the methods implemented by the Service layer are neither json nor protobuf message.
As for data conversion, that's what the transport layer should care about (later).
Endpoint development
The endpoint layer calls the NewArticleService method exposed by the Service layer. Before the Service layer, the endpoint layer can be used as the business middleware. It also doesn't care whether the request is http or grpc.
// endpoint/article/article.go
package article
import (
"context"
"demo/errors"
"demo/params/article_param"
service "demo/service"
"github.com/go-kit/kit/endpoint"
)
// make endpoint service -> endpoint
func MakeCreateEndpoint(svc service.ArticleService) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (interface{}, error) {
req, ok := request.(*article_param.CreateReq)
if !ok {
return nil, errors.EndpointTypeError
}
resp, err := svc.Create(ctx, req)
if err != nil {
return nil, err
}
return resp, nil
}
}
// make endpoint service -> endpoint
func MakeDetailEndpoint(svc service.ArticleService) endpoint.Endpoint {
......
}HTTP + Json development
Transport layer
The transport layer is before the endpoint layer. Its main function is to parse the requested data and encode the response data. In this layer, it is necessary to distinguish whether the request is http or grpc. After all, the data formats of the two service requests are different and need to be processed separately.
In the transport layer, decodeRequest, encodeResponse and HTTP Handler.
- decodeRequest parses the parameters from the http request and turns them into a "neutral" model structure;
- encodeResponse converts the "neutral" model structure into json, or adds operations such as response headers;
- The handler uses the transport/http fusion coderequest and encodeResponse provided by git kit, and calls the endpoint layer.
// transport/http/article/create.go
package article
import (
"context"
endpoint "demo/endpoint/article"
"demo/params/article_param"
"demo/service"
transport "demo/transport/http"
"encoding/json"
"fmt"
httptransport "github.com/go-kit/kit/transport/http"
"net/http"
)
// Server
// 1. decode request http.request -> model.request
func decodeCreateRequest(_ context.Context, r *http.Request) (interface{}, error) {
if err := transport.FormCheckAccess(r); err != nil {
return nil, err
}
if err := r.ParseForm(); err != nil {
fmt.Println(err)
return nil, err
}
req := &article_param.CreateReq{}
err := transport.ParseForm(r.Form, req)
if err != nil {
return nil, err
}
fmt.Printf("r.Form:%#v\n", r.Form)
fmt.Printf("req:%#v\n", req)
r.Body.Close()
return req, nil
}
// 2. encode response model.response -> http.response
func encodeCreateResponse(_ context.Context, w http.ResponseWriter, resp interface{}) error {
w.Header().Set("Content-Type", "application/json")
return json.NewEncoder(w).Encode(resp)
}
// make handler
func MakeCreateHandler(svc service.ArticleService) http.Handler {
handler := httptransport.NewServer(
endpoint.MakeCreateEndpoint(svc),
decodeCreateRequest,
encodeCreateResponse,
transport.ErrorServerOption(), // Custom error handling
)
return handler
}Router + Server
The router layer forwards to different transport layers according to the url
// router/httprouter/article.go
package httprouter
import (
svc "demo/service"
transport "demo/transport/http/article"
"net/http"
)
func RegisterRouter(mux *http.ServeMux) {
mux.Handle("/article/create", transport.MakeCreateHandler(svc.NewArticleService()))
mux.Handle("/article/detail", transport.MakeDetailHandler(svc.NewArticleService()))
}The server layer is used to start the http service and introduce router.
// server/http/server.go
package http
import (
"demo/router/httprouter"
"net"
"net/http"
)
var mux = http.NewServeMux()
var httpServer = http.Server{Handler: mux}
// http run
func Run(addr string, errc chan error) {
// Register routing
httprouter.RegisterRouter(mux)
lis, err := net.Listen("tcp", addr)
if err != nil {
errc <- err
return
}
errc <- httpServer.Serve(lis)
}Finally, call http. in a unified script. Run starts the HTTP service.
// cmd/server/sever.go
package main
import http "demo/server/http"
func main() {
errc := make(chan error)
go http.Run("0.0.0.0:8080", errc)
// After the grpc service is completed, start grpc here
log.WithField("error", <-errc).Info("Exit")
}Grpc + Protobuf development
Write protobuf
In grpc service, we use protobuf as the data format, so the first step is to write protobuf.
In protobuf, define service and message. Write protobuf mesage based on data model and protobuf service based on ArticleService interface of service layer.
// pb/article/article.proto
syntax = "proto3";
option go_package = ".;proto";
service ArticleService {
rpc Create(CreateReq) returns (CreateResp);
rpc Detail(DetailReq) returns (DetailResp);
}
message CreateReq {
string Title = 1;
string Content = 2;
int64 CateId = 3;
}
message CreateResp {
int64 Id = 1;
}
message DetailReq {
int64 Id = 1;
}
message DetailResp {
int64 Id = 1;
string Title = 2;
string Content = 3;
int64 CateId = 4;
int64 UserId = 5;
}proto files are similar to the data model, but don't confuse them. The model is "neutral".
What does the proto file mean?
The service keyword specifies a grpc service, which provides two methods: Create and Detail.
The message keyword specifies the message structure, which is composed of type variable name = sequence number. The final transmission protobuf is in binary format. It only encodes the value and does not encode the variable name. Therefore, the sequence number is required to parse the corresponding variable.
When the client calls the Create method of the ArticleService service, it needs to pass in the parameter set of the CreateReq structure, and the server will return the parameter set of the CreateResp structure.
Generate pb file and execute the following command in pb/article / Directory:
protoc --proto_path=./ --go_out=plugins=grpc:./ ./article.proto
The pb file of the generated Go version is as follows.
This is article pb. Go Server code for Server use.
// pb/article/article.pb.go
......
// ArticleServiceServer is the server API for ArticleService service.
type ArticleServiceServer interface {
Create(context.Context, *CreateReq) (*CreateResp, error)
Detail(context.Context, *DetailReq) (*DetailResp, error)
}
// UnimplementedArticleServiceServer can be embedded to have forward compatible implementations.
type UnimplementedArticleServiceServer struct {
}
func (*UnimplementedArticleServiceServer) Create(context.Context, *CreateReq) (*CreateResp, error) {
return nil, status.Errorf(codes.Unimplemented, "method Create not implemented")
}
func (*UnimplementedArticleServiceServer) Detail(context.Context, *DetailReq) (*DetailResp, error) {
return nil, status.Errorf(codes.Unimplemented, "method Detail not implemented")
}
func RegisterArticleServiceServer(s *grpc.Server, srv ArticleServiceServer) {
s.RegisterService(&_ArticleService_serviceDesc, srv)
}This is article pb. Go in the Client code, Pb file will be given to the Client, the Client code will be used by the Client.
// pb/article/article.pb.go
......
// ArticleServiceClient is the client API for ArticleService service.
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.
type ArticleServiceClient interface {
Create(ctx context.Context, in *CreateReq, opts ...grpc.CallOption) (*CreateResp, error)
Detail(ctx context.Context, in *DetailReq, opts ...grpc.CallOption) (*DetailResp, error)
}
type articleServiceClient struct {
cc grpc.ClientConnInterface
}
func NewArticleServiceClient(cc grpc.ClientConnInterface) ArticleServiceClient {
return &articleServiceClient{cc}
}
func (c *articleServiceClient) Create(ctx context.Context, in *CreateReq, opts ...grpc.CallOption) (*CreateResp, error) {
out := new(CreateResp)
err := c.cc.Invoke(ctx, "/ArticleService/Create", in, out, opts...)
if err != nil {
return nil, err
}
return out, nil
}
func (c *articleServiceClient) Detail(ctx context.Context, in *DetailReq, opts ...grpc.CallOption) (*DetailResp, error) {
out := new(DetailResp)
err := c.cc.Invoke(ctx, "/ArticleService/Detail", in, out, opts...)
if err != nil {
return nil, err
}
return out, nil
}Transport layer
Next, implement the transport layer of grpc service.
In the transport layer, decodeRequest, encodeResponse and grpc should be implemented Handler.
- decodeRequest parses the protobuf of grpc request and turns it into a "neutral" model structure;
- encodeResponse converts the "neutral" model structure to protobuf;
- The handler uses the integrated decodeRequest and encodeResponse of transport/grpc provided by git kit, and calls the endpoint layer.
// transport/grpc/article/create.go
package article
import (
"context"
"demo/params/article_param"
pb "demo/pb/article"
"fmt"
)
// 1. decode request pb -> model
func decodeCreateRequest(c context.Context, grpcReq interface{}) (interface{}, error) {
req, ok := grpcReq.(*pb.CreateReq)
if !ok {
fmt.Println("grpc server decode request Error!")
return nil, fmt.Errorf("grpc server decode request Error!")
}
// Filter data
request := &article_param.CreateReq{
Title: req.Title,
Content: req.Content,
CateId: req.CateId,
}
return request, nil
}
// 2. encode response model -> pb
func encodeCreateResponse(c context.Context, response interface{}) (interface{}, error) {
fmt.Printf("%#v\n", response)
resp, ok := response.(*article_param.CreateResp)
if !ok {
return nil, fmt.Errorf("grpc server encode response error (%T)", response)
}
r := &pb.CreateResp{
Id: resp.Id,
}
return r, nil
}The transport layer of grpc service is similar to that of http service. In addition, a little "glue" is needed to integrate grpc and go kit.
// transport/grpc/article/article.go
package article
import (
"context"
endpoint "demo/endpoint/article"
pb "demo/pb/article"
"demo/service"
grpctransport "github.com/go-kit/kit/transport/grpc"
)
// ArticleGrpcServer 1. Pb All methods of articleserviceserver implement "inheritance";
// 2. Two grpctransport. create and detail are defined Handler.
type ArticleGrpcServer struct {
createHandler grpctransport.Handler
detailHandler grpctransport.Handler
}
// When you call Create through grpc, Create only transfers data, and createHandler is called inside Create and handed over to go kit for processing
func (s *ArticleGrpcServer) Create (ctx context.Context, req *pb.CreateReq) (*pb.CreateResp, error) {
_, rsp, err := s.createHandler.ServeGRPC(ctx, req)
if err != nil {
return nil, err
}
return rsp.(*pb.CreateResp), err
}
func (s *ArticleGrpcServer) Detail (ctx context.Context, req *pb.DetailReq) (*pb.DetailResp, error) {
_, rsp, err := s.detailHandler.ServeGRPC(ctx, req)
if err != nil {
return nil, err
}
return rsp.(*pb.DetailResp), err
}
// NewArticleGrpcServer returns the article grpc server defined in proto
func NewArticleGrpcServer(svc service.ArticleService, opts ...grpctransport.ServerOption) pb.ArticleServiceServer {
createHandler := grpctransport.NewServer(
endpoint.MakeCreateEndpoint(svc),
decodeCreateRequest,
encodeCreateResponse,
opts...,
)
articleGrpServer := new(ArticleGrpcServer)
articleGrpServer.createHandler = createHandler
return articleGrpServer
}Function of ArticleGrpcServer
1. Pb All methods of the articleserviceserver interface implement "inheritance". It can also be said that the instance of the ArticleGrpcServer is a pb.ArticleServiceServer type.
2. Two grpctransport. create and detail are defined Handler. The purpose is to connect the go kit model.
When the Create method is called through grpc, create only transfers data, and createHandler is called inside create, so the request is handed over to go kit for processing.
The function of NewArticleGrpcServer is to return to the article grpc server defined in proto and expose the grpc service for the outer grpc router to call. It integrates multiple handler s.
The Handler of go kit calls endpoint, decodeRequest and encodeResponse.
createHandler := grpctransport.NewServer(
endpoint.MakeCreateEndpoint(svc),
encodeCreateResponse,
decodeCreateRequest,
opts...,
)Router + Server
Register the services exposed by the transport layer in the router layer.
// router/grpcrouter/article.go
package grpcrouter
import (
pb "demo/pb/article"
"demo/service"
transport "demo/transport/grpc/article"
"google.golang.org/grpc"
)
func RegisterRouter(grpcServer *grpc.Server) {
pb.RegisterArticleServiceServer(grpcServer, transport.NewArticleGrpcServer(service.NewArticleService()))
}The server layer introduces the router layer and starts the grpc service.
// server/grpc/server.go
package grpc
import (
"demo/router/grpcrouter"
"net"
grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
"google.golang.org/grpc"
)
var opts = []grpc.ServerOption{
grpc_middleware.WithUnaryServerChain(
RecoveryInterceptor,
),
}
var grpcServer = grpc.NewServer(opts...)
func Run(addr string, errc chan error) {
// Register grpcServer
grpcrouter.RegisterRouter(grpcServer)
lis, err := net.Listen("tcp", addr)
if err != nil {
errc <- err
return
}
errc <- grpcServer.Serve(lis)
}Finally, call grpc. in a unified script. Run starts the grpc service and the previously implemented http service.
// cmd/server/sever.go
package main
import http "demo/server/http"
import grpc "demo/server/grpc"
func main() {
errc := make(chan error)
go http.Run("0.0.0.0:8080", errc)
go grpc.Run("0.0.0.0:5000", errc)
log.WithField("error", <-errc).Info("Exit")
}Run script
go run cmd/server/sever.go
Done!
reference resources
github.com/junereycasuga/gokit-grp...
github.com/win5do/go-microservice-...
Personal blog synchronization articles Using git kit to realize microservices supporting http and grpc