gRPC: Quickly configure generic API to get process meta-information

introduce

This article describes how to quickly configure a common API through rk-boot and obtain process meta-information.

What is process meta-information?

The process meta-information contains the following:

Name

details

API List

Get API List

gRPC Error Information

gRPC corresponds to grpc-gateway error information

OS Information

Current OS Information

Process information

Display process meta information

Go Env Information

Go Environment Information

Config List

List Config file information

Log Information

Show Log Original Information

Metrics

Show Prometheus Metrics

Entry List

Entry list, see details

README

If README exists locally. MD file, will show

CA Certificate

List TLS/SSL certificate information

rely on

List go.mod information

LICENSE

If there are LICENSE files locally, they will be displayed

Git Information

If it's a Git project, it will show

We will use rk-boot To start the gRPC service.

Visit the following address for the complete tutorial:

install

go get github.com/rookie-ninja/rk-boot
go get github.com/rookie-ninja/rk-grpc

Quick Start

1. Create boot.yaml

boot.yaml tells rk-boot how to start the gRPC service. For Demo, in addition to opening the commonService, we have also started the following services.

You can also access it only through the Restful API without turning on the options below.

  • prom: Start the prometheus client in order to be able to validate/rk/v1/req requests
  • interceptors.metricsProm: Start the Prometheus middleware to validate/rk/v1/req requests
  • sw: Start the Swagger UI in order to be able to verify the commonService API through the Swagger UI
  • tv: Start RK TV to validate commonService data through the Web UI
---
grpc:
  - name: greeter                   # Required, Name of grpc entry
    port: 8080                      # Required, Port of grpc entry
    enabled: true                   # Required, Enable grpc entry
    commonService:
      enabled: true                 # Optional, Enable common service, default: false
    prom:
      enabled: true                 # Enable prometheus client
    sw:
      enabled: true                 # Optional, Enable swagger, default: false
    tv:
      enabled: true                 # Optional, Enable Rk TV, default: false
    interceptors:
      metricsProm:
        enabled: true               # Optional, Enable prometheus metrics

2. Create main.go

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	_ "github.com/rookie-ninja/rk-grpc/boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3. Folder structure

.
├── boot.yaml
├── go.mod
├── go.sum
└── main.go

4. Validation

$ go run main.go

4.1 API List

For a detailed explanation of each API, please refer to the API Details below.

gRPC Method

Restful API

introduce

rk.api.v1.RkCommonService.Healthy

GET /rk/v1/healthy

Return to process health

rk.api.v1.RkCommonService.Gc

GET /rk/v1/gc

Trigger Golang garbage collection and return memory usage before and after GC

rk.api.v1.RkCommonService.Info

GET /rk/v1/info

Return process meta information

rk.api.v1.RkCommonService.Configs

GET /rk/v1/configs

Return static Config Entry content information

rk.api.v1.RkCommonService.Apis

GET /rk/v1/apis

Returns a list of all API s registered with RPC

rk.api.v1.RkCommonService.Sys

GET /rk/v1/sys

Return OS information

rk.api.v1.RkCommonService.Entries

GET /rk/v1/entries

Returns all Entry information within a process

rk.api.v1.RkCommonService.Req

GET /rk/v1/req

If Prometheus middleware is turned on, the requested monitoring information is returned

rk.api.v1.RkCommonService.Logs

GET /rk/v1/logs

Returns process log meta information, including log path, etc.

rk.api.v1.RkCommonService.Certs

GET /rk/v1/certs

Return certificate information if TLS/SSL is used

rk.api.v1.RkCommonService.GwErrorMapping

GET /rk/v1/gwErrorMapping

Returns error code mapping from grpc-gateway to gRPC

rk.api.v1.RkCommonService.Deps

GET /rk/v1/deps

Return to go.mod information

rk.api.v1.RkCommonService.License

GET /rk/v1/license

Return License File Information

rk.api.v1.RkCommonService.Readme

GET /rk/v1/readme

Return README.md file information

rk.api.v1.RkCommonService.Git

GET /rk/v1/git

If compiled using the rk command line, git meta information for the current project is returned

4.2 Access through API

Direct access to boot.yaml-defined port to access. commonService uses/rk/v1 as the path prefix by default.

$ curl -X GET localhost:8080/rk/v1/healthy
{
    "healthy":true
}

4.3 Access via Swagger UI

Visit http://localhost:8080/sw

4.4 Access via RK TV

Visit http://localhost:8080/rk/v1/tv

5 API Details

5.1 /rk/v1/healthy

As long as the process is still running and the RPC service has no errors, true is always returned.

Note that in most cases, /rk/v1/health is eligible for the responsibility of serving the Health Check-up, except in some special cases.

For example, the process is still alive, but there is a problem within the service and it is in an "abnormal" state. In this case, the user is advised to implement an "health check" API as needed.

$ curl -X GET localhost:8080/rk/v1/healthy
{
    "healthy":true
}

5.2 /rk/v1/gc

Force the go process to do a garbage collection and return memory changes before and after GC.

Useful when dealing with DEBUG issues, online services are not recommended.

$ curl -X GET localhost:8080/rk/v1/gc 
{
    "memStatAfterGc":{
        "forceGcCount":1,
        "gcCountTotal":8,
        "lastGcTimestamp":"2021-12-16T00:35:02+08:00",
        "memAllocByte":6801880,
        "memUsedMb":6,
        "memUsedPercentage":0.09,
        "sysAllocByte":75645960
    },
    "memStatBeforeGc":{
        "forceGcCount":0,
        "gcCountTotal":6,
        "lastGcTimestamp":"2021-12-16T00:32:50+08:00",
        "memAllocByte":6873512,
        "memUsedMb":6,
        "memUsedPercentage":0.09,
        "sysAllocByte":75645960
    }
}

5.3 /rk/v1/info

Returns process meta information.

$ curl -X GET localhost:8080/rk/v1/info
{
    "appName":"rk",
    "az":"",
    "description":"Internal RK entry which describes application with fields of appName, version and etc.",
    "docsUrl":[],
    "domain":"",
    "gid":"20",
    "homeUrl":"",
    "iconUrl":"",
    "keywords":[],
    "maintainers":[],
    "realm":"",
    "region":"",
    "startTime":"2021-12-16T00:28:24+08:00",
    "uid":"501",
    "upTimeSec":566,
    "upTimeStr":"9 minutes",
    "username":"Dongxun Yin",
    "version":""
}

We can see that some of the Fields above are empty.

Field

How to configure

appName

Find from the current working directory. rk/rk.yaml file and read appName if rk. If yaml does not exist, the default value is returned: RK

version

Find from the current working directory. rk/rk.yaml file and read version if rk. If yaml does not exist, the default value is returned: ""

realm

Read Key as REALM value from environment variable

region

Read Key as REGION value from environment variable

az

Read Key AZ values from environment variables

DOMAIN

Read Key as DOMAIN value from environment variable

description

From boot. Read in yaml, here's an example

keywords

From boot. Read in yaml, here's an example

homeUrl

From boot. Read in yaml, here's an example

iconUrl

From boot. Read in yaml, here's an example

docsUrl

From boot. Read in yaml, here's an example

maintainers

From boot. Read in yaml, here's an example

It seems like a hassle, so let's take a complete example and let/rk/v1/info return the full information. Or use the above project, we make two changes.

  • At boot. Add app to yaml
  • Increase REALM, REGION, AZ, DOMAIN environment variables

Refer to the configuration of appName & version: demo

boot.yaml

---
# Add app related fields for demonstration!
app:
  description: "This is description for introducing common API."
  keywords: ["rk", "common API"]
  homeUrl: "https://rkdev.info"
  iconUrl: "https://rkdev.info"
  docsUrl: ["https://rkdev.info"]
  maintainers: ["Tony Stark", "Thor"]
grpc:
  - name: greeter                     # Required
    port: 8080                        # Required
    enabled: true                     # Required
    commonService:
      enabled: true                   # Optional, default: false
    tv:
      enabled: true                   # Optional, default: false
    sw:
      enabled: true                   # Optional, default: false
    prom:
      enabled: true                   # Optional, default: false
    interceptors:
      metricsProm:
        enabled: true                 # Optional, default: false

main.go

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	_ "github.com/rookie-ninja/rk-grpc/boot"
	"os"
)

// Application entrance.
func main() {
	os.Setenv("REALM", "rk-realm")
	os.Setenv("REGION", "rk-region")
	os.Setenv("AZ", "rk-az")
	os.Setenv("DOMAIN", "rk-domain")

	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

output

$ curl -X GET localhost:8080/rk/v1/info
{
    "appName":"rk",
    "az":"rk-az",
    "description":"This is description for introducing common API.",
    "docsUrl":[
        "https://rkdev.info"
    ],
    "domain":"rk-domain",
    "gid":"20",
    "homeUrl":"https://rkdev.info",
    "iconUrl":"https://rkdev.info",
    "keywords":[
        "rk",
        "common API"
    ],
    "maintainers":[
        "Tony Stark",
        "Thor"
    ],
    "realm":"rk-realm",
    "region":"rk-region",
    "startTime":"2021-12-16T01:38:31+08:00",
    "uid":"501",
    "upTimeSec":25,
    "upTimeStr":"25 seconds",
    "username":"Dongxun Yin",
    "version":""
}

5.4 /rk/v1/configs

Read Config Entry in the process. Must be at boot. Config is configured in yaml for display.

How to read Config in code, for reference gRPC: Distinguish profiles based on cloud native environments

Or use the above project, and we'll make a change.

  • Add config/config.yaml file
  • At boot. Add config entries to yaml

boot.yaml

---
# Add config related fields for demonstration!
config:
  - name: my-config                   # Required
    locale: "*::*::*::*"              # Required
    path: config/config.yaml         # Required
grpc:
  - name: greeter                     # Required
    port: 8080                        # Required
    enabled: true                     # Required
    commonService:
      enabled: true                   # Optional, default: false
    tv:
      enabled: true                   # Optional, default: false
    sw:
      enabled: true                   # Optional, default: false
    prom:
      enabled: true                   # Optional, default: false
    interceptors:
      metricsProm:
        enabled: true                 # Optional, default: false

main.go

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	_ "github.com/rookie-ninja/rk-grpc/boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

output

$ tree
.
├── boot.yaml
├── config
│   └── config.yaml
├── go.mod
├── go.sum
└── main.go
  
1 directory, 5 files

$ curl -X GET localhost:8080/rk/v1/configs
{
    "entries":[
        {
            "entryDescription":"Internal RK entry which read user config file into viper instance.",
            "entryMeta":{
                "key":"value"
            },
            "entryName":"my-config",
            "entryType":"ConfigEntry",
            "path":"/Users/dongxuny/workspace/rk/rk-demo/grpc/demo/config/config.yaml"
        }
    ]
}

5.5 /rk/v1/apis

Lists all API s registered with the RPC service.

$ curl -X GET localhost:8080/rk/v1/apis 
{
    "entries":[
        {
            "entryName":"greeter",
            "grpc":{
                "gw":{
                    "method":"GET",
                    "pattern":"/rk/v1/apis",
                    "port":8080,
                    "swUrl":"http://127.0.0.1:8080/sw/"
                },
                "method":"Apis",
                "port":8080,
                "service":"rk.api.v1.RkCommonService",
                "type":"Unary"
            },
            "rest":null
        },
        ...
    ]
}

5.6 /rk/v1/sys

$ curl -X GET localhost:8080/rk/v1/sys
{
    "cpuInfo":{
        "cacheSize":256,
        "cpuUsedPercentage":21.63,
        "logicalCoreCount":8,
        "mhz":2000,
        "modelName":"Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz",
        "physicalCoreCount":4,
        "vendorId":"GenuineIntel"
    },
    "goEnvInfo":{
        "goArch":"amd64",
        "goos":"darwin",
        "routinesCount":16,
        "startTime":"2021-12-16T01:48:50+08:00",
        "upTimeSec":222,
        "upTimeStr":"3 minutes",
        "version":"go1.16.3"
    },
    "memInfo":{
        "forceGcCount":0,
        "gcCountTotal":5,
        "lastGcTimestamp":"2021-12-16T01:51:11+08:00",
        "memAllocByte":7367384,
        "memUsedMb":7,
        "memUsedPercentage":0.1,
        "sysAllocByte":75645960
    },
    "netInfo":{
        "netInterface":[
            {
                "addrs":[
                    "127.0.0.1/8",
                    "::1/128",
                    "fe80::1/64"
                ],
                "flags":[
                    "up",
                    "loopback",
                    "multicast"
                ],
                "hardwareAddr":"",
                "mtu":16384,
                "multicastAddrs":[
                    "ff02::fb",
                    "224.0.0.251",
                    "ff02::2:ff33:9cc0",
                    "ff01::1",
                    "ff02::1",
                    "ff02::1:ff00:1",
                    "224.0.0.1"
                ],
                "name":"lo0"
            },
            ...
        ]
    },
    "osInfo":{
        "arch":"amd64",
        "hostname":"lark.local",
        "os":"darwin"
    }
}

5.7 /rk/v1/entries

List all Entries in the process.

$ curl -X GET localhost:8080/rk/v1/entries
{
    "entries":{
        "AppInfoEntry":[
            Object{...}
        ],
        "ConfigEntry":[
            Object{...}
        ],
        "EventLoggerEntry":[
            Object{...}
        ],
        "GrpcEntry":[
            Object{...}
        ],
        "ZapLoggerEntry":[
            Object{...}
        ]
    }
}

5.8 /rk/v1/req

Returns API Prometheus monitoring data. The Prometheus client & Prometheus middleware must be opened.

$ curl -X GET localhost:8080/rk/v1/req
{
    "metrics":[
        {
            "count":1,
            "elapsedNanoP50":724285,
            "elapsedNanoP90":724285,
            "elapsedNanoP99":724285,
            "elapsedNanoP999":724285,
            "grpcMethod":"Sys",
            "grpcService":"rk.api.v1.RkCommonService",
            "resCode":Array[1],
            "restMethod":"",
            "restPath":""
        },
        Object{...},
        ...
    ]
}

5.9 /rk/v1/logs

Returns Log Entry information if at boot. The log entry is configured in yaml and will be shown.

Please refer to gRPC: How do I manage log configuration properly? Add log configuration.

$ curl -X GET localhost:8080/rk/v1/logs
{
    "entries":{
        "EventLoggerEntry":[
            {
                "entryDescription":"Internal RK entry which is used to log event such as RPC request or periodic jobs.",
                "entryMeta":Object{...},
                "entryName":"eventLoggerDefault",
                "entryType":"EventLoggerEntry",
                "errorOutputPaths":[
                    "stderr"
                ],
                "outputPaths":[
                    "stdout"
                ]
            }
        ],
        "ZapLoggerEntry":Array[1]
    }
}

5.10 /rk/v1/certs

Returns the certificate information if TLS/SSL is used.

Please refer to gRPC: How do I turn on TLS/SSL?

5.11 /rk/v1/gwErrorMapping

Please refer to GRPC: How can gRPC provide Restful API services?

5.12 /rk/v1/deps, /rk/v1/license, /rk/v1/readme, /rk/v1/git

rk-boot reads from the current working directory. Rk/rk. The yaml file reads the specific file path.

This feature is in use rk The command line is compiled for packaging purposes.

Of course, users can configure it themselves. rk/rk.yaml files, but they can be cumbersome. Please refer to demo

$ rk build
[1/8] Clearing target folder
------------------------------------[OK]
[2/8] Execute user command before
- buf generate --path api/v1

------------------------------------[OK]
[3/8] Execute user script before
- No user scripts found!
------------------------------------[OK]
[4/8] Build go file
- go build -o target/bin/demo main.go
- cp -r boot.yaml target
- cp -r README.md target/.rk/README.md
- cp -r LICENSE target/.rk/LICENSE
- cp -r go.mod target/.rk/dep/go.mod
- cp -r cov.html target/.rk/ut/cov.html
cp: cov.html: No such file or directory
------------------------------------[OK]
[5/8] Copy to target folder
- cp -r api target/
------------------------------------[OK]
[6/8] Generate rk meta from on local
- Write files to target/.rk/rk.yaml
------------------------------------[OK]
[7/8] Execute user script after
- No user scripts found!
------------------------------------[OK]
[8/8] Execute user command after
- No user commands found!
------------------------------------[OK]

$ tree target/.rk 
target/.rk
├── LICENSE
├── README.md
├── dep
│   └── go.mod
├── rk.yaml
└── ut
  
2 directories, 4 files

$ cat target/.rk/rk.yaml 
git:
  branch: master
  commit:
    committer:
      email: dongxuny@gmail.com
      name: dongxuny
    date: Sun Dec 12 00:44:25 2021 +0800
    id: 04ae4f58a6f6c45614bfeae8c950becf247d06a2
    idAbbr: 04ae4f5
    sub: |
      Bump up rk-boot to v1.4.0
  tag: ""
  url: https://github.com/rookie-ninja/rk-demo
name: demo
version: master-04ae4f5

Keywords: Go api Microservices grpc

Added by ddevitt on Wed, 15 Dec 2021 23:37:29 +0200