Go Engineering - Foundation - RD

功能需求

为 iamctl 新增 helloworld 命令,该命令向终端打印 hello world

开发阶段

代码开发

选择 Git Flow:适用于大型非开源项目

创建分支

基于 develop 分支,新建一个功能分支 feature/helloworld

1
$ git checkout -b feature/helloworld develop

branch 名要符合 Git Flow 的分支命名规范,会通过 pre-commit 的 githook 来确保分支名符合规范

1
2
3
$ md5 ./githooks/pre-commit ./.git/hooks/pre-commit
MD5 (./githooks/pre-commit) = 3324d20a738461f3b6347f9ce7dae6b6
MD5 (./.git/hooks/pre-commit) = 3324d20a738461f3b6347f9ce7dae6b6
./.git/hooks/pre-commit
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#!/usr/bin/env bash
LC_ALL=C

local_branch="$(git rev-parse --abbrev-ref HEAD)"

valid_branch_regex="^(master|develop)$|(feature|release|hotfix)\/[a-z0-9._-]+$|^HEAD$"

message="There is something wrong with your branch name. Branch names in this project must adhere to this contract: $valid_branch_regex.
Your commit will be rejected. You should rename your branch to a valid name and try again."

if [[ ! $local_branch =~ $valid_branch_regex ]]
then
echo "$message"
echo ${local_branch} 1111
exit 1
fi

exit 0

15bb43219269273baf70a27ea94e1279

git 默认不会提交 .git/hooks 下的 githooks 脚本,可以借助 Makefile 来同步 – 每次执行 make 命令时都会执行

scripts/make-rules/common.mk
1
2
# Copy githook scripts when execute makefile
COPY_GITHOOK:=$(shell cp -f githooks/* .git/hooks/)

生成模板

创建 helloworld 命令模板,包含 low code 思想 – 代码自动生成(提高开发效率 + 保证代码规范)

1
2
$ iamctl new helloworld -d internal/iamctl/cmd/helloworld
Command file generated: internal/iamctl/cmd/helloworld/helloworld.go

编辑 internal/iamctl/cmd/cmd.go

1
2
3
4
5
6
7
8
9
10
11
12
import (
...
"github.com/marmotedu/iam/internal/iamctl/cmd/helloworld"
)
...
{
Message: "Troubleshooting and Debugging Commands:",
Commands: []*cobra.Command{
validate.NewCmdValidate(f, ioStreams),
helloworld.NewCmdHelloworld(f, ioStreams), // 加载 helloworld 命令
},
},

生成代码

通过 make gen 生成的存量代码要具有幂等性

1
2
3
4
5
6
7
8
9
10
11
12
$ make gen
===========> Installing codegen
===========> Generating iam error code go source files
===========> Generating error code markdown documentation
===========> Generating missing doc.go for go packages
pkg/rollinglog/doc.go
pkg/rollinglog/distribution/doc.go
pkg/rollinglog/example/doc.go
pkg/rollinglog/klog/doc.go
pkg/rollinglog/logrus/doc.go
pkg/rollinglog/rolling/doc.go
pkg/validator/example/doc.go
Makefile
1
2
3
4
## gen: Generate all necessary files, such as error code files.
.PHONY: gen
gen:
@$(MAKE) gen.run
scripts/make-rules/gen.mk
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
.PHONY: gen.run
gen.run: gen.clean gen.errcode gen.docgo.doc

.PHONY: gen.clean
gen.clean:
@rm -rf ./api/client/{clientset,informers,listers}
@$(FIND) -type f -name '*_generated.go' -delete

.PHONY: gen.errcode
gen.errcode: gen.errcode.code gen.errcode.doc

.PHONY: gen.errcode.code
gen.errcode.code: tools.verify.codegen
@echo "===========> Generating iam error code go source files"
@codegen -type=int ${ROOT_DIR}/internal/pkg/code

.PHONY: gen.errcode.doc
gen.errcode.doc: tools.verify.codegen
@echo "===========> Generating error code markdown documentation"
@codegen -type=int -doc \
-output ${ROOT_DIR}/docs/guide/zh-CN/api/error_code_generated.md ${ROOT_DIR}/internal/pkg/code

.PHONY: gen.docgo.doc
gen.docgo.doc:
@echo "===========> Generating missing doc.go for go packages"
@${ROOT_DIR}/scripts/gendoc.sh
scripts/make-rules/tools.mk
1
2
3
.PHONY: tools.verify.%
tools.verify.%:
@if ! which $* &>/dev/null; then $(MAKE) tools.install.$*; fi

版权检查(开源软件)

如果有新文件添加,需要检查新文件有没有添加版权头信息

1
2
3
$ make verify-copyright
===========> Verifying the boilerplate headers for all files
...
Makefile
1
2
3
4
## verify-copyright: Verify the boilerplate headers for all files.
.PHONY: verify-copyright
verify-copyright:
@$(MAKE) copyright.verify
scripts/make-rules/copyright.mk
1
2
3
4
5
6
7
8
.PHONY: copyright.verify
copyright.verify: tools.verify.addlicense
@echo "===========> Verifying the boilerplate headers for all files"
@addlicense --check -f $(ROOT_DIR)/scripts/boilerplate.txt $(ROOT_DIR) --skip-dirs=third_party,vendor,_output

.PHONY: copyright.add
copyright.add: tools.verify.addlicense
@addlicense -v -f $(ROOT_DIR)/scripts/boilerplate.txt $(ROOT_DIR) --skip-dirs=third_party,vendor,_output
scripts/make-rules/tools.mk
1
2
3
.PHONY: tools.verify.%
tools.verify.%:
@if ! which $* &>/dev/null; then $(MAKE) tools.install.$*; fi

自动添加版权头

1
$ make add-copyright

代码格式化

1
2
3
4
5
$ make format          
===========> Installing golines
go: downloading github.com/segmentio/golines v0.9.0
...
===========> Formating codes
Makefile
1
2
3
4
5
6
7
8
## format: Gofmt (reformat) package sources (exclude vendor dir if existed).
.PHONY: format
format: tools.verify.golines tools.verify.goimports
@echo "===========> Formating codes"
@$(FIND) -type f -name '*.go' | $(XARGS) gofmt -s -w
@$(FIND) -type f -name '*.go' | $(XARGS) goimports -w -local $(ROOT_PACKAGE)
@$(FIND) -type f -name '*.go' | $(XARGS) golines -w --max-len=120 --reformat-tags --shorten-comments --ignore-generated .
@$(GO) mod edit -fmt
Package Desc
gofmt 格式化 go 代码
goimports 自动增删依赖包,并将依赖包按照字母序排序并分类
golines 把超过 120 行的代码按照 golines 规则,格式化成 < 120 行的代码
go mod edit -fmt 格式化 go.mod

静态代码检查

1
2
3
$ make lint
===========> Run golangci to lint source codes
...

单元测试

1
2
3
$ make test
===========> Run unit test
...
Makefile
1
2
3
4
## test: Run unit test.
.PHONY: test
test:
@$(MAKE) go.test

并非所有包都需要单元测试;mock_.*.go 文件中的函数是不需要单元测试的

scripts/make-rules/golang.mk
1
2
3
4
5
6
7
8
9
.PHONY: go.test
go.test: tools.verify.go-junit-report
@echo "===========> Run unit test"
@set -o pipefail;$(GO) test -race -cover -coverprofile=$(OUTPUT_DIR)/coverage.out \
-timeout=10m -shuffle=on -short -v `go list ./...|\
egrep -v $(subst $(SPACE),'|',$(sort $(EXCLUDE_TESTS)))` 2>&1 | \
tee >(go-junit-report --set-exit-code >$(OUTPUT_DIR)/report.xml)
@sed -i '/mock_.*.go/d' $(OUTPUT_DIR)/coverage.out # remove mock_.*.go files from test coverage
@$(GO) tool cover -html=$(OUTPUT_DIR)/coverage.out -o $(OUTPUT_DIR)/coverage.html

检查单元测试覆盖率,如果单元测试覆盖率不达标,禁止合并到 develop 和 master 分支 – CI Pipeline

1
2
3
$ make cover # 默认测试覆盖率至少为60%

$ make cover COVERAGE=90
Makefile
1
2
3
4
## cover: Run unit test and get test coverage.
.PHONY: cover
cover:
@$(MAKE) go.test.cover
scripts/make-rules/golang.mk
1
2
3
4
.PHONY: go.test.cover
go.test.cover: go.test
@$(GO) tool cover -func=$(OUTPUT_DIR)/coverage.out | \
awk -v target=$(COVERAGE) -f $(ROOT_DIR)/scripts/coverage.awk

构建

1
2
3
$ make build

$ make build BINS="iam-apiserver iamctl"

默认操作

1
$ make
Makefile
1
2
3
4
5
# Build all by default, even if it's not first
.DEFAULT_GOAL := all

.PHONY: all
all: tidy gen add-copyright format lint cover build

代码提交

commit + push

只添加与 feature/helloworld 相关的改动,而非 git add .

1
$ git add internal/iamctl/cmd/helloworld internal/iamctl/cmd/cmd.go

.git/hooks/commit-msg 会检查 commit message 是否符合 Angular Commit Message 规范

1
2
$ git commit -m "feat: add new iamctl command 'helloworld'"
$ git push origin feature/helloworld
.git/hooks/commit-msg
1
2
3
#!/usr/bin/env bash

go-gitlint --msg-file="$1
.gitlint
1
2
3
--subject-regex=^((Merge branch.*of.*)|((revert: )?(feat|fix|perf|style|refactor|test|ci|docs|chore)(\(.+\))?: [^A-Z].*[^.]$))
--subject-maxlen=72
--body-regex=^([^\r\n]{0,72}(\r?\n|$))*$

Github Actions

配置 Github Actions,当有代码被 Push 后,会触发 CI Pipeline,Pipeline 会执行 make all

6819f96bda8dcb214c3b7eeba2f37022

线上 CI 流程与本地 CI 流程要完全保持一致

.github/workflows/iamci.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
name: IamCI

on:
push:
branchs:
- '*'
pull_request:
types: [ opened, reopened ]

jobs:

iamci:
name: Test with go ${{ matrix.go_version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
environment:
name: iamci

strategy:
matrix:
go_version: [ 1.16 ]
os: [ ubuntu-latest ]

steps:

- name: Set up Go ${{ matrix.go_version }}
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go_version }}
id: go

- name: Check out code into the Go module directory
uses: actions/checkout@v2

- name: Run go modules tidy
run: |
make tidy

- name: Generate all necessary files, such as error code files
run: |
make gen

- name: Check syntax and styling of go sources
run: |
make lint

- name: Run unit test and get test coverage
run: |
make cover

- name: Build source code for host platform
run: |
make build

- name: Collect Test Coverage File
uses: actions/upload-artifact@v1.0.0
with:
name: main-output
path: _output/coverage.out

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1

- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Build docker images for host arch and push images to registry
run: |
make push

提交 PR

新 PR 被创建后,会触发 CI Pipeline

53f4103f5c8cabb76ef2fddaec3a54ab

Code Review

如果 Review 不通过,开发者可以直接在 feature/helloworld 上修正代码,并再次 push(会触发 CI Pipeline)

39d992c7bdb35848706bce792877e8ce

接受 PR

使用 Create a merge commit,底层操作为 git merge --no-ff方便回溯

30de6bb6c8ff431ec56debbc0f5b667d

关闭 PR

合并到 develop 后,会触发 CI Pipeline,关闭 PR

444b0701f8866b50a49bd0138488c873

测试阶段

创建分支

基于 develop 分支,创建 release 分支

1
2
$ git checkout -b release/1.0.0 develop
$ make

Bug Fix

直接在 release/1.0.0 分支上修改代码,本地构建并 push 代码,线上 CI 成功后,则将代码提交给测试同学

1
2
3
4
$ make
$ git add internal/iamctl/cmd/helloworld/
$ git commit -m "fix: fix helloworld print bug"
$ git push origin release/1.0.0

代码合并

测试通过后,将 feature 分支合并到 master 分支和 develop 分支 – 测试阶段的产物

1
2
3
4
5
6
$ git checkout develop
$ git merge --no-ff release/1.0.0

$ git checkout master
$ git merge --no-ff release/1.0.0
$ git tag -a v1.0.0 -m "add print hello world"

删除分支

删除 feature 分支,选择性地删除 release 分支

1
$ git branch -d feature/helloworld

Makefile 技巧

help 自动解析

通过 sed 命令,自动解析 Makefile 中以 ## 开头的注释行,自动生成 make help 输出

Makefile
1
2
3
4
5
6
## help: Show this help info.
.PHONY: help
help: Makefile
@echo -e "\nUsage: make <TARGETS> <OPTIONS> ...\n\nTargets:"
@sed -n 's/^##//p' $< | column -t -s ':' | sed -e 's/^/ /'
@echo "$$USAGE_OPTIONS"

Options

scripts/make-rules/common.mk
1
2
3
4
# Minimum test coverage
ifeq ($(origin COVERAGE),undefined)
COVERAGE := 60
endif
1
2
$ make
$ make COVERAGE=90

CHANGELOG

CHANGELOG 用来展示每个版本之间的变更内容,作为 Release Note 的一部分,借助 git-chglog 自动生成

1
2
3
4
$ tree .chglog
.chglog
├── CHANGELOG.tpl.md
└── config.yml

语义化版本号

借助 gsemve 工具,gsemver 会根据 Commit Message 自动生成版本号

scripts/ensure_tag.sh
1
2
3
4
5
6
#!/usr/bin/env bash

version=v`gsemver bump`
if [ -z "`git tag -l $version`" ];then
git tag -a -m "release version $version" $version
fi

后续的 Makefile 和 Shell 都会用到 scripts/make-rules/common.mk 中的 VERSION 变量

scripts/make-rules/common.mk
1
2
3
ifeq ($(origin VERSION), undefined)
VERSION := $(shell git describe --tags --always --match='v*')
endif

如果符合条件的 tag 指向最新提交,则只显示 tag 名字,否则会显示:该 tag 后有多少次提交 + 最新的 commit id

1
2
$ git describe --tags --always --match='v*'
v1.0.0-3-g1909e47
参数 描述
--tags 不仅仅使用带有注释的 tag,而是使用 refs/tags 名称空间下的任何 tag
--always 显示唯一缩写的提交对象作为后备
--match 只考虑与给定模式相匹配的 tag