Grpc: adjust data transfer size limit

Introduce

How does this article describe rk-boot adjustment gRPC Data transfer size limit .

grpc The size limit of exists in the receiver , In other words, there is no limit to how much data to send , The receiving default size is 4MB.

The example uses google.golang.org/grpc v1.38.0 edition .

What is? gRPC Data transfer size limit ？

gRPC The default maximum data transmission size of the server is 4MB, Sometimes , We need to transmit larger data , Like big pictures .

Please visit the following address for a complete tutorial ：

• https://rkdocs.netlify.app/cn

install

go get github.com/rookie-ninja/rk-boot
go get github.com/rookie-ninja/rk-grpc

Quick start

rk-boot Support through code & YAML How to adjust the size limit of files .

For a complete demonstration , We create a greeter API.

1. establish protobuf Related documents

We use buf Command line to compile protobuf, You need to create the following files .

file name	describe
api/v1/greeter.proto	protobuf file
buf.yaml	tell buf Where does the command line find protobuf file
buf.gen.yaml	tell buf How the command line compiles protobuf file

• api/v1/greeter.proto

syntax = "proto3";

package api.v1;

option go_package = "api/v1/greeter";

service Greeter {
  rpc Greeter (GreeterRequest) returns (GreeterResponse) {}
}

message GreeterRequest {
  bytes msg = 1;
}

message GreeterResponse {}

• buf.yaml

version: v1beta1
name: github.com/rk-dev/rk-demo
build:
  roots:
    - api

• buf.gen.yaml

version: v1beta1
plugins:
  # protoc-gen-go needs to be installed, generate go files based on proto files
  - name: go
    out: api/gen
    opt:
     - paths=source_relative
  # protoc-gen-go-grpc needs to be installed, generate grpc go files based on proto files
  - name: go-grpc
    out: api/gen
    opt:
      - paths=source_relative
      - require_unimplemented_servers=false

• compile protobuf file

$ buf generate

2. establish boot.yaml

We go through boot.yaml The way to 【 Cancel 】 Size limit , adopt boot.yaml We can lift the restrictions , But the limits cannot be adjusted .

Adjust the limit , It can be adjusted by code , We will also introduce it below .

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    noRecvMsgSizeLimit: true

3. establish server.go

We did greeter Interface .

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-demo/api/gen/v1"
	"github.com/rookie-ninja/rk-grpc/boot"
	"google.golang.org/grpc"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Get grpc entry with name
	grpcEntry := boot.GetEntry("greeter").(*rkgrpc.GrpcEntry)
	grpcEntry.AddRegFuncGrpc(registerGreeter)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

func registerGreeter(server *grpc.Server) {
	greeter.RegisterGreeterServer(server, &GreeterServer{})
}

type GreeterServer struct{}

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	return &greeter.GreeterResponse{}, nil
}

4. establish client.go

We try to transmit 10MB The data of .

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"github.com/rookie-ninja/rk-demo/api/gen/v1"
	"google.golang.org/grpc"
	"log"
)

func main() {
	opts := []grpc.DialOption{
		grpc.WithInsecure(),
		grpc.WithBlock(),
	}

	// 1: Create grpc client
	conn, client := createGreeterClient(opts...)
	defer conn.Close()

	kb := 1024
	mb := 1024*kb

	// 2: Call server with 10mb size of data
	if _, err := client.Greeter(context.Background(), &greeter.GreeterRequest{Msg: make([]byte, 10*mb, 10*mb)}); err != nil {
		panic(err)
	}
}

func createGreeterClient(opts ...grpc.DialOption) (*grpc.ClientConn, greeter.GreeterClient) {
	// 1: Set up a connection to the server.
	conn, err := grpc.DialContext(context.Background(), "localhost:8080", opts...)
	if err != nil {
		log.Fatalf("Failed to connect: %v", err)
	}

	// 2: Create grpc client
	client := greeter.NewGreeterClient(conn)

	return conn, client
}

5. Folder structure

.
├── api
│   ├── gen
│   │   └── v1
│   │       ├── greeter.pb.go
│   │       └── greeter_grpc.pb.go
│   └── v1
│       └── greeter.proto
├── boot.yaml
├── buf.gen.yaml
├── buf.yaml
├── client.go
├── go.mod
├── go.sum
└── server.go

6. verification

There won't be any mistakes .

$ go run server.go
$ go run client.go

Because the server is only allowed to receive by default 4MB Size data , If we were boot.yaml In the noRecvMsgSizeLimit Set to false, You will get the following mistakes .

rpc error: code = ResourceExhausted desc = grpc: received message larger than max (10485765 vs. 4194304)

adjustment 【 Server side 】 Transfer data size

In the last example , We use noRecvMsgSizeLimit Option cancelled gRPC The size limit of the server , This time, , We try to resize .

Or use the above protobuf file .

1. modify boot.yaml

This time we put noRecvMsgSizeLimit Set to false.

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    noRecvMsgSizeLimit: false

2. modify server.go

We go through AddServerOptions() Function to set the maximum value received by the server .

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-demo/api/gen/v1"
	"github.com/rookie-ninja/rk-grpc/boot"
	"google.golang.org/grpc"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Get grpc entry with name
	grpcEntry := boot.GetEntry("greeter").(*rkgrpc.GrpcEntry)
	grpcEntry.AddRegFuncGrpc(registerGreeter)
	
	// *************************************** //
	// *** Set server receive size to 20MB ***
	// *************************************** //
	kb := 1024
	mb := 1024*kb
	grpcEntry.AddServerOptions(grpc.MaxRecvMsgSize(20*mb))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

func registerGreeter(server *grpc.Server) {
	greeter.RegisterGreeterServer(server, &GreeterServer{})
}

type GreeterServer struct{}

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	return &greeter.GreeterResponse{}, nil
}

3. verification

There won't be any mistakes .

$ go run server.go
$ go run client.go

If we send more data than 20mb, The following error occurs .

rpc error: code = ResourceExhausted desc = grpc: received message larger than max (31457285 vs. 20971520)

adjustment 【 client 】 Transfer data size

If the data returned by the server is greater than 4MB, We need to resize on the client side .

kb := 1024
mb := 1024*kb

opts := []grpc.DialOption{
	grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(20*mb)),
	grpc.WithInsecure(),
	grpc.WithBlock(),
}