Skip to main content

Organize gRPC and protobuf code in Golang

At this point, it's a simple, effective and fast RPC framework with a lot of cross-language support. If you need something like it, there's not many other choices available with this big of an ecosystem. - Anonymous from HN about gRPC

In this article, I'll describe how to organize protobuf files messages and gRPC services in the Go sources. I'll briefly examine how to use protoc and plugins with the proper imports, and project structure.


A diagram describes how the repositories service-a, service-b requires echo-contracts:

how golang modules organized

Jumt to #

Project structure #

The repository echo-contracts describes our proto files. It is a standalone Go repository intended to import into other Go sources. Below is a project structure:

├── Makefile
├── docker
│   ├── Dockerfile
│   └── docker-compose.yaml
├── go.mod
├── go.sum
└── pb
    ├── message.proto
    └── service.proto

Makefile #

The Makefile contains a bunch of valuable targets.

.PHONY: help
help: ## Display available commands.
	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)

.PHONY: proto-clean
proto-clean: ## Clean generated proto.
	@rm -rf pb/message
	@rm -rf pb/service

.PHONY: proto-compile
proto-compile: ## Compile message protobuf and gRPC service files.
	PLATFORM=$(shell uname -m) PROTOC_VERSION=$(PROTOC_VERSION) docker-compose -f docker/docker-compose.yaml run --rm protogen

.PHONY: docker-config
docker-config: ## Dump docker-compose configuration.
	PLATFORM=$(shell uname -m) PROTOC_VERSION=$(PROTOC_VERSION) docker-compose -f docker/docker-compose.yaml config

Dockerfile #

In the Dockerfile, protoc with a particular version is downloaded. The version of protoc binary is declared as PROTOC_VERSION variable at the top of the Makefile and is passed to the docker container during the build stage.

protoc doesn't officially support Go as output, you need to install external plugins. gRPC is not the same as Protocol Buffers, gRPC uses Protocol Buffers, hence different plugins are needed to generate Go code messages and services.

Plugins installed:

FROM golang:1.21


RUN apt-get update && apt-get install -y unzip

# By default Intel chipset (x86_64) is assumed but if the host device is an Apple
# silicon (arm) chipset based then a relevant (aarch_64) release file is used.

RUN go install[email protected]
RUN go install[email protected]

RUN export ZIP=x86_64 && \
    if [ ${PLATFORM} = "arm64" ]; then export ZIP=aarch_64; fi && \
    wget --quiet${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-${ZIP}.zip && \
    unzip -o protoc-${PROTOC_VERSION}-linux-${ZIP}.zip -d /usr/local bin/protoc && \
    unzip -o protoc-${PROTOC_VERSION}-linux-${ZIP}.zip -d /usr/local 'include/*'

docker-compose #

I'm using docker-compose.yaml as an engine for generating proto.

      context: "."
    working_dir: "/source"
      - "../pb:/source"
    command: bash -c "
        protoc *.proto --proto_path=.

proto options #

Now we're instructing the protoc to generate code for us, we're planning to import echo-contracts as Go module and we need a proper package structure.

There is an options for --go_out and --go-grpc_opt:

For example, an input file pb/message.proto with a Go import path of and specified as the module prefix results in an output file at pb/message/message.pb.go.

module option

Here are our proto files:

syntax = "proto3";
package echo.service.v1;
option go_package = "";

message StringMessage {
    string value = 1;
syntax = "proto3";
package echo.service.v1;
option go_package = "";

import "message.proto";

service EchoService {
    rpc Echo(StringMessage) returns (StringMessage) {}

go 1.21.0

require ( v1.57.0 v1.31.0

require ( v1.5.3 // indirect v0.9.0 // indirect v0.7.0 // indirect v0.9.0 // indirect v0.0.0-20230525234030-28d5490b6b19 // indirect

generate #

At this step we can generate Go code from proto files. Use the command make proto-compile. Here is the directory structure after proto compilation:

├── Makefile
├── docker
│   ├── Dockerfile
│   └── docker-compose.yaml
├── go.mod
├── go.sum
└── pb
    ├── message
    │   └── message.pb.go
    ├── message.proto
    ├── service
    │   ├── service.pb.go
    │   └── service_grpc.pb.go
    └── service.proto

Release echo-contracts Using Go Modules, so it is published as Go module with the proper version.

usage #

To import our echo-contracts Go module, we need to declare it in the main.go file and go.mod as an external dependency:

package main

import (
	// ...
	message ""
	service ""
  // ...

// Some code here

And in go.mod:

module service-a

go 1.21.0

require (
  // Some imports here v0.0.1

Happy coding!