参考博文
https://ewanvalentine.io/microservices-in-golang-part-1/
这个博文是作者微服务系统的第一篇,本学习笔记基于目前的5篇而成
part 1 利用gRPC,protobuf定义服务
本人在学习过程中没有严格按照博文中放在github目录,而是在主目录中创建一个wdyshippy的目录,目录中文件结构如下
.
├── consignment-cli
│ ├── cli.go
│ ├── consignment-cli
│ ├── consignment.json
│ ├── Dockerfile
│ └── Makefile
└── consignment-service
├── consignment-service
├── Dockerfile
├── main.go
├── Makefile
└── proto
└── consignment
├── consignment.pb.go
└── consignment.proto
4 directories,11 files
执行的protoc命令
protoc -I. --go_out=plugins=grpc:/home/wdy/wdyshippy/consignment-service proto/consignment/consignment.proto
如果make build时报错
makefile:2: *** 遗漏分隔符
原因是在编写makefile文件时,命令前面应该是tab写为了空格
在这个阶段中,consignment-service的Makefile文件内容如下
build:
protoc -I. --go_out=plugins=grpc:$(GOPATH)/src/github.com/ewanvalentine/shipper/consignment-service \
proto/consignment/consignment.proto
part 2 - Docker and go-micro
创建Dockefiel
$ touch consignment-service/Dockerfile
写入如下内容
FROM alpine:latest
RUN mkdir /app
WORKDIR /app
ADD consignment-service /app/consignment-service
CMD ["./consignment-service"]
原作者提示如果是在linux机子上编译测试时,将alpine改为debian
FROM debian:latest
RUN mkdir /app
WORKDIR /app
ADD consignment-service /app/consignment-service
CMD ["./consignment-service"]
WORKDIR 表示 /app目录作为上下文目录用来装载我们的程序consignment-service
接下来为Makefile文件增加内容用来生成docker image
build:
...
GOOS=linux GOARCH=amd64 go build
docker build -t consignment-service .
除此之外再添加
run:
docker run -p 50051:50051 consignment-service
用来执行consignment-service
docker run -p 后面的参数,第一个端口号是对外的端口,第二个是内部的端口。
@H_403_169@Go-micro使用Go-micro来加入service discovery功能
首先安装
go get -u github.com/micro/protobuf/{proto,protoc-gen-go}
修改makefile文件中protoc的参数,将参数plugins=后面的grpc更改为micro
build:
protoc -I. --go_out=plugins=micro:/home/wdy/wdyshippy/consignment-service proto/consignment/consignment.proto
...
...
import中 要引入 micro “github.com/micro/go-micro”
// consignment-service/main.go
package main
import (
// Import the generated protobuf code
"fmt"
pb "github.com/EwanValentine/shippy/consignment-service/proto/consignment"
micro "github.com/micro/go-micro"
"golang.org/x/net/context"
)
server接口实现时,response的位置改变为输入参数,输出参数只有error
func (s *service) CreateConsignment(ctx context.Context,req *pb.Consignment,res *pb.Response) error {
...
func (s *service) GetConsignments(ctx context.Context,req *pb.GetRequest,res *pb.Response) error {
...
还有就是server的初始化
func main() {
repo := &Repository{}
// Create a new service. Optionally include some options here.
srv := micro.NewService(
// This name must match the package name given in your protobuf definition
micro.Name("go.micro.srv.consignment"),micro.Version("latest"),)
// Init will parse the command line flags.
srv.Init()
// Register handler
pb.RegisterShippingServiceHandler(srv.Server(),&service{repo})
// Run the server
if err := srv.Run(); err != nil {
fmt.Println(err)
}
}
最后就是不需要在代码中硬编码端口号,Go-micro通过环境变量或命令行参数来传递。
run:
docker run -p 50051:50051 \
-e MICRO_SERVER_ADDRESS=:50051 \
-e MICRO_REGISTRY=mdns consignment-service
如上,通过 -e MICRO_SERVER_ADDRESS=:50051 指定服务地址,通过-e MICRO_REGISTRY=mdns指定service discovery功能使用mdns。在实际项目中mdns很少使用,大部分使用consul。
import (
...
"github.com/micro/go-micro/cmd"
microclient "github.com/micro/go-micro/client"
)
func main() {
cmd.Init()
// Create new greeter client
client := pb.NewShippingServiceClient("go.micro.srv.consignment",microclient.DefaultClient)
...
}
通过docker启动consignment-service后,执行客户端程序会连接失败,原因在于server允许在docker中使用的是docker中的dmns,和客户端使用的不是同一个,所以发现不了server,解决方法就是把客户端程序也放入到docker 中,使用同一个dmns。
创建consignment-cli/Makefile文件
build:
GOOS=linux GOARCH=amd64 go build
docker build -t consignment-cli .
run:
docker run -e MICRO_REGISTRY=mdns consignment-cli
Dockerfile文件
FROM alpine:latest
RUN mkdir -p /app
WORKDIR /app
ADD consignment.json /app/consignment.json
ADD consignment-cli /app/consignment-cli
CMD ["./consignment-cli"]
作者最后又提供了一个更加标准的Dockerfile文件,这个文件中包含了consignment-service的开发环境和生存环境
在开发环境中通过引用github.com/golang/dep/cmd/dep 来自动更新包依赖
RUN go get -u github.com/golang/dep/cmd/dep
# Create a dep project,and run `ensure`,which will pull in all
# of the dependencies within this directory.
RUN dep init && dep ensure
完整文件如下
# consignment-service/Dockerfile
# We use the official golang image,which contains all the
# correct build tools and libraries. Notice `as builder`,
# this gives this container a name that we can reference later on.
FROM golang:1.9.0 as builder
# Set our workdir to our current service in the gopath
WORKDIR /go/src/github.com/EwanValentine/shippy/consignment-service
# Copy the current code into our workdir
COPY . .
# Here we're pulling in godep,which is a dependency manager tool,
# we're going to use dep instead of go get,to get around a few
# quirks in how go get works with sub-packages.
RUN go get -u github.com/golang/dep/cmd/dep
# Create a dep project,and run `ensure`,which will pull in all
# of the dependencies within this directory.
RUN dep init && dep ensure
# Build the binary,with a few flags which will allow
# us to run this binary in Alpine.
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo .
# Here we're using a second FROM statement,which is strange,
# but this tells Docker to start a new build process with this
# image.
FROM alpine:latest
# Security related package,good to have.
RUN apk --no-cache add ca-certificates
# Same as before,create a directory for our app.
RUN mkdir /app
WORKDIR /app
# Here,instead of copying the binary from our host machine,
# we pull the binary from the container named `builder`,within
# this build context. This reaches into our prevIoUs image,finds
# the binary we built,and pulls it into this container. Amazing!
COPY --from=builder /go/src/github.com/EwanValentine/shippy/consignment-service/consignment-service .
# Run the binary as per usual! This time with a binary build in a
# separate container,with all of the correct dependencies and
# run time libraries.
CMD ["./consignment-service"]
part 3 - docker-compose and datastores
介绍了docker-compose的安装 Install docker-compose: https://docs.docker.com/compose/install/
还有 docker-compose的使用
docker-compose.yml的内容如下:
version: '3.1'
services: consignment-cli: build: ./consignment-cli environment: MICRO_REGISTRY: "mdns" consignment-service: build: ./consignment-service ports: - 50051:50051 environment: MICRO_ADDRESS: ":50051" MICRO_REGISTRY: "mdns" DB_HOST: "datastore:27017" vessel-service: build: ./vessel-service ports: - 50052:50051 environment: MICRO_ADDRESS: ":50051" MICRO_REGISTRY: "mdns"
还介绍了数据库包
最后介绍了go-micro客户端另外一种编写方式
package main
import (
"log"
"os"
pb "github.com/EwanValentine/shippy/user-service/proto/user"
microclient "github.com/micro/go-micro/client"
"github.com/micro/go-micro/cmd"
"golang.org/x/net/context"
"github.com/micro/cli"
"github.com/micro/go-micro"
)
func main() {
cmd.Init()
// Create new greeter client
client := pb.NewUserServiceClient("go.micro.srv.user",microclient.DefaultClient)
// Define our flags
service := micro.NewService(
micro.Flags(
cli.StringFlag{
Name: "name",Usage: "You full name",},cli.StringFlag{
Name: "email",Usage: "Your email",cli.StringFlag{
Name: "password",Usage: "Your password",cli.StringFlag{
Name: "company",Usage: "Your company",),)
// Start as service
service.Init(
micro.Action(func(c *cli.Context) {
name := c.String("name")
email := c.String("email")
password := c.String("password")
company := c.String("company")
// Call our user service
r,err := client.Create(context.TODO(),&pb.User{
Name: name,Email: email,Password: password,Company: company,})
if err != nil {
log.Fatalf("Could not create: %v",err)
}
log.Printf("Created: %s",r.User.Id)
getAll,err := client.GetAll(context.Background(),&pb.Request{})
if err != nil {
log.Fatalf("Could not list users: %v",err)
}
for _,v := range getAll.Users {
log.Println(v)
}
os.Exit(0)
}),)
// Run the server
if err := service.Run(); err != nil {
log.Println(err)
}
}
传递参数运行如下
$ docker-compose run user-cli command \
--name="Ewan Valentine" \ --email="ewan.valentine89@gmail.com" \ --password="Testing123" \ --company="BBC"
Part 4 - Authentication with JWT
运行两个数据库
$ docker run -d -p 5432:5432 postgres
$ docker run -d -p 27017:27017 mongo
JWT
http://cryto.net/~joepie91/blog/2016/06/13/stop-using-jwt-for-sessions/
Cheat_Sheet_for_Java”>https://www.owasp.org/index.php/JSON_Web_Token(JWT)_Cheat_Sheet_for_Java
https://auth0.com/blog/json-web-token-signing-algorithms-overview/
https://tools.ietf.org/html/rfc7518#section-3
Go library for this: github.com/dgrijalva/jwt-go
User-service
负责用户信息的认证和token的发放校验
https://github.com/EwanValentine/shippy/blob/master/user-service/token_service.go
Token_service.go中代码负责jwt的编码(Encode)和解码(Decode),用于server的调用
Encode
// Encode a claim into a JWT
func (srv *TokenService) Encode(user *pb.User) (string,error) {
// Create the Claims
claims := CustomClaims{
user,jwt.StandardClaims{
ExpiresAt: 15000,Issuer: "go.micro.srv.user",}
// Create token
token := jwt.NewWithClaims(jwt.SigningMethodHS256,claims)
// Sign token and return
return token.SignedString(key)
}
上面代码有误,设置ExpiresAt 为15000,运行程序会报token过期,正确代码为
// Encode a claim into a JWT
func (srv *TokenService) Encode(user *User) (string,error) {
expireToken := time.Now().Add(time.Hour * 72).Unix()
// Create the Claims
claims := CustomClaims{
user,jwt.StandardClaims{
ExpiresAt: expireToken,claims)
// Sign token and return
return token.SignedString(key)
}
Decode
func (srv *TokenService) Decode(token string) (*CustomClaims,error) {
// Parse the token
tokenType,err := jwt.ParseWithClaims(string(key),&CustomClaims{},func(token *jwt.Token) (interface{},error) {
return key,nil
})
// Validate the token and return the custom claims
if claims,ok := tokenType.Claims.(*CustomClaims); ok && tokenType.Valid {
return claims,nil
} else {
return nil,err
}
}
上面代码运行会panic
2018/01/17 01:39:31 panic recovered: runtime error: invalid memory address or nil pointer dereference
2018/01/17 01:39:31 goroutine 35 [running]:
runtime/debug.Stack(0xc420121920,0x2,0x2)
/home/wdy/go/src/runtime/debug/stack.go:24 +0x79
github.com/micro/go-micro/server.(*rpcServer).accept.func1(0xb8dfe0,0xc4200e4f80)
/home/wdy/svn/cloud_trunk/factory/branches/tob_material/main/tobmaterialsys/src/github.com/micro/go-micro/server/rpc_server.go:60 +0x124
panic(0x8d1720,0xbbe270)
/home/wdy/go/src/runtime/panic.go:489 +0x2cf
main.(*TokenService).Decode(0xc420011560,0xc4202cf080,0x15b,0xc420266af0,0xc42011e780,0xc4200326b8)
/home/wdy/wdyshippy/user-service/token_service.go:45 +0x10c
main.(*service).ValidateToken(0xc4200b1740,0x7f90d890c050,0xc420271140,0xc420271170,0xc420032738,0x41168c)
/home/wdy/wdyshippy/user-service/handler.go:76 +0x11c
_/home/wdy/wdyshippy/user-service/proto/user.(*UserService).ValidateToken(0xc420011af0,0x0,0x0)
/home/wdy/wdyshippy/user-service/proto/user/user.pb.go:310 +0x5b
reflect.Value.call(0xc42005ea00,0xc42000e1d0,0x13,0x95de61,0x4,0xc420032bb0,0x901c80,0x913a40,...)
/home/wdy/go/src/reflect/value.go:434 +0x91f
reflect.Value.Call(0xc42005ea00,0x40,0x38)
/home/wdy/go/src/reflect/value.go:302 +0xa4
github.com/micro/go-micro/server.(*service).call.func1(0x7f90d890c050,0xb8fdc0,0xc420132f00,0x0)
原因在于Decode中jwt.ParseWithClaims的第一个参数应该是token,正确代码为:
// Decode a token string into a token object
func (srv *TokenService) Decode(tokenString string) (*CustomClaims,error) {
// Parse the token
token,err := jwt.ParseWithClaims(tokenString,func(token *jwt.Token) (interface{},error) {
return key,nil
})
// Validate the token and return the custom claims
if claims,ok := token.Claims.(*CustomClaims); ok && token.Valid {
return claims,nil
} else {
return nil,err
}
}
https://github.com/EwanValentine/shippy/blob/master/user-service/handler.go
负责用户的创建,密码校验和token发放和校验
golang.org/x/crypto/bcrypt
用户创建
func (srv *service) Create(ctx context.Context,req *pb.User,res *pb.Response) error {
// Generates a hashed version of our password
hashedPass,err := bcrypt.GenerateFromPassword([]byte(req.Password),bcrypt.DefaultCost)
if err != nil {
return err
}
req.Password = string(hashedPass)
if err := srv.repo.Create(req); err != nil {
return err
}
res.User = req
return nil
}
用户密码校验,校验成功则发放token
func (srv *service) Auth(ctx context.Context,req *pb.User,res *pb.Token) error {
log.Println("Logging in with:",req.Email,req.Password)
user,err := srv.repo.GetByEmail(req.Email)
log.Println(user)
if err != nil {
return err
}
// Compares our given password against the hashed password
// stored in the database
if err := bcrypt.CompareHashAndPassword([]byte(user.Password),[]byte(req.Password)); err != nil {
return err
}
token,err := srv.tokenService.Encode(user)
if err != nil {
return err
}
res.Token = token
return nil
}
token校验
func (srv *service) ValidateToken(ctx context.Context,req *pb.Token,res *pb.Token) error {
// Decode token
claims,err := srv.tokenService.Decode(req.Token)
if err != nil {
return err
}
log.Println(claims)
if claims.User.Id == "" {
return errors.New("invalid user")
}
res.Valid = true
return nil
}
consignment-cli
客户端请求的时候加上token
github.com/micro/go-micro/Metadata
...
ctx := Metadata.NewContext(context.Background(),map[string]string{
"token": token,})
r,err := client.CreateConsignment(ctx,consignment)
if err != nil {
log.Fatalf("Could not create: %v",err)
}
log.Printf("Created: %t",r.Created)
getAll,err := client.GetConsignments(ctx,&pb.GetRequest{})
if err != nil {
log.Fatalf("Could not list consignments: %v",err)
}
for _,v := range getAll.Consignments {
log.Println(v)
}
...
consignment-server
https://github.com/EwanValentine/shippy/tree/master/consignment-service
// shippy-consignment-service/main.go
func main() {
...
// Create a new service. Optionally include some options here.
srv := micro.NewService(
// This name must match the package name given in your protobuf definition
micro.Name("go.micro.srv.consignment"),micro.Version("latest"),// Our auth middleware
micro.WrapHandler(AuthWrapper),)
...
}
...
// AuthWrapper is a high-order function which takes a HandlerFunc
// and returns a function,which takes a context,request and response interface.
// The token is extracted from the context set in our consignment-cli,that
// token is then sent over to the user service to be validated.
// If valid,the call is passed along to the handler. If not,
// an error is returned.
func AuthWrapper(fn server.HandlerFunc) server.HandlerFunc {
return func(ctx context.Context,req server.Request,resp interface{}) error {
Meta,ok := Metadata.FromContext(ctx)
if !ok {
return errors.New("no auth Meta-data found in request")
}
// Note this is now uppercase (not entirely sure why this is...)
token := Meta["Token"]
log.Println("Authenticating with token: ",token)
// Auth here
authClient := userService.NewUserServiceClient("go.micro.srv.user",client.DefaultClient)
_,err := authClient.ValidateToken(context.Background(),&userService.Token{
Token: token,})
if err != nil {
return err
}
err = fn(ctx,req,resp)
return err
}
}
AuthWarpper是一个中间件,接受一个HandlerFunc,进行某种处理后 返回HandlerFunc。
该中间件从context提取token,然后发送到user-service中校验token是否有效,如果有效再继续执行真正的操作fn。
Part 5 - Event brokering with Go Micro
修改 user-service/main.go
func main() {
...
// Init will parse the command line flags.
srv.Init()
// Get instance of the broker using our defaults
pubsub := srv.Server().Options().Broker
// Register handler
pb.RegisterUserServiceHandler(srv.Server(),&service{repo,tokenService,pubsub})
...
}
修改user-service/handler.go
const topic = "user.created"
type service struct {
repo Repository
tokenService Authable
PubSub broker.Broker
}
func (srv *service) Create(ctx context.Context,bcrypt.DefaultCost)
if err != nil {
return err
}
req.Password = string(hashedPass)
if err := srv.repo.Create(req); err != nil {
return err
}
res.User = req
if err := srv.publishEvent(req); err != nil {
return err
}
return nil
}
func (srv *service) publishEvent(user *pb.User) error {
// Marshal to JSON string
body,err := json.Marshal(user)
if err != nil {
return err
}
// Create a broker message
msg := &broker.Message{
Header: map[string]string{
"id": user.Id,Body: body,}
// Publish message to broker
if err := srv.PubSub.Publish(topic,msg); err != nil {
log.Printf("[pub] Failed: %v",err)
}
return nil
}
除此之外 user-service handler.go import添加
```go "github.com/micro/go-micro/broker" _ "github.com/micro/go-plugins/broker/nats" ```
user-service/Makefile中添加
-e MICRO_BROKER=nats \ -e MICRO_BROKER_ADDRESS=0.0.0.0:4222 \
email-service参见原作者的repo https://github.com/EwanValentine/shippy-email-service
srv.Init()
pubsub := srv.Server().Options().Broker
在go-micro中, srv.Init()
会搜索所有的配置,包括plugin,环境变量以及命令行参数,然后把这些配置初始化为service的组成部分。为了使用这些配置实例,需要通过 srv.Server().Options()
,在本项目例子中,通过指定
-e MICRO_BROKER=nats \ -e MICRO_BROKER_ADDRESS=0.0.0.0:4222 \
go-micro会找到NATS的broker plugin,创建对应的实例,用于之后的连接和使用。
在email-server中,用来订阅event
if err := pubsub.Connect(); err != nil {
log.Fatal(err)
}
// Subscribe to messages on the broker
_,err := pubsub.Subscribe(topic,func(p broker.Publication) error {
var user *pb.User
if err := json.Unmarshal(p.Message().Body,&user); err != nil {
return err
}
log.Println(user)
go sendEmail(user)
return nil
})
在user-service中,用来发布event
func (srv *service) publishEvent(user *pb.User) error {
// Marshal to JSON string
body,err)
}
return nil
}
遇到的问题以及解决方法
NATS配置
本人成功的方法如下
wdy@wdy:~$ docker pull nats
Using default tag: latest
latest: Pulling from library/nats
f169c9506d74: Pull complete
bb9eff5cafb0: Pull complete
Digest: sha256:61fcb1f40da2111434fc910b0865c54155cd6e5f7c42e56e031c3f35a9998075
Status: Downloaded newer image for nats:latest
wdy@wdy:~$ docker run -p 4222:4222 -p 8222:8222 -p 6222:6222 --name gnatsd -ti nats:latest
[1] 2018/01/17 05:45:07.167855 [INF] Starting nats-server version 1.0.4
[1] 2018/01/17 05:45:07.167935 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2018/01/17 05:45:07.167961 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2018/01/17 05:45:07.167964 [INF] Server is ready
[1] 2018/01/17 05:45:07.168111 [INF] Listening for route connections on 0.0.0.0:6222
user-service nats连接失败
User-service 执行make run
之后报如下错误
nats: no servers available for connection
解决办法,将user_serice中的makefile的
run:
docker run -p 50053:50051 \
-e MICRO_SERVER_ADDRESS=:50051 \
-e MICRO_REGISTRY=mdns \
user-service
改为
run:
docker run --net="host" \
-p 50055 \
-e MICRO_SERVER_ADDRESS=:50055 \
-e MICRO_REGISTRY=mdns \
-e MICRO_BROKER=nats \
-e MICRO_BROKER_ADDRESS=0.0.0.0:4222 \
//TODO 等作者6-11的更新后,本文再同步更新