GRPC Health Checks in Kubernetes

The gRPC framework is becoming the de facto standard for backend services. Services no longer need to implement custom REST support given the transcoding provided by cloud platforms. These transcoding appliances expose REST endpoints that forward requests to the deployed gRPC service. If there is no business requirement for implementing custom REST endpoints, then why would we do so just to support liveness health checks? 

Support in Kubernetes

Support for using gRPC for health checks has lagged the frameworks adoption. This is to be expected. The old way (before K8s 1.23) was running an independent health probe. Before the release of K8s 1.23 an independent health probe was required to query the health of gRPC services. The gRPC liveness probe was then introduced with the release of K8s 1.24. This version also enables the required GRPCContainerProbe Feature Gate by default. These are quick advances in gRPC support. What does it really take to deploy a service that uses gRPC health checks? What are the minimum requirements?

Example service

The simplest K8s YAML configuration to demonstrate liveness health checks might be less than the following. It is important to consider what we are health checking. The following example configures a health check upon a service which is overlayed upon a deployment with one container.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: example-deploy
 namespace: default
spec:
 replicas: 1
 selector:
   matchLabels:
     app: example
 template:
   metadata:
     labels:
       app: example
   spec:
     containers:
     - name: example
       image: example:v0
       imagePullPolicy: Never
       ports:
       - containerPort: 8000
       livenessProbe:
         grpc:
           port: 8000
         initialDelaySeconds: 5
         periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
 name: example-service
spec:
 # type: NodePort
 selector:
   app: example
 ports:
   - protocol: TCP
     port: 8000

Let's take a closer look at what this basic example creates in the K8s cluster.  The most important tidbit is the liveness probe on the container:

livenessProbe:
  grpc:
    port: 8000

The full YAML configuration will create a deployment with one container that has a service on port 8000. The service is implemented in a Docker image tagged example:v0. That Docker image will be deployed as a single pod. In order to meet the requirements for the gRPC livenessProbe, the service implemented and deployed in the Docker image must implement that health check. 

Implementing the health service

The health check contract that K8s expects is the GRPC Health Checking Protocol. There are implementations available for most languages. This main.go shows the full registration of the gRPC health service that always returns success.

package main
import (
 "flag"
 "fmt"
 "log"
 "net"

 "google.golang.org/grpc"
 "google.golang.org/grpc/health"
 "google.golang.org/grpc/health/grpc_health_v1"
 "google.golang.org/grpc/reflection"
)

func main() {
 port := flag.Int("port", 8000, "The server port")
 host := flag.String("host", "", "The server host, perhaps 'localhost'")
 flag.Parse()

 s := grpc.NewServer()
 defer s.Stop()
 hs := health.NewServer() // will default to respond with SERVING
 grpc_health_v1.RegisterHealthServer(s, hs) // registration
 // register your own services
 reflection.Register(s)
 address := fmt.Sprintf("%s:%d", *host, *port)
 log.Default().Printf("starting service on  %s", address)
 listener, err := net.Listen("tcp", address)
 if err != nil {
   panic(err)
 }
 err = s.Serve(listener)
 if err != nil {
   panic(err)
 }
}

The key piece here is that the default implementation of the health service is registered.

 hs := health.NewServer() // will default to respond with SERVING
 grpc_health_v1.RegisterHealthServer(s, hs) // registration

All that's left is to build a Docker image (docker build -t example:v0 .) and then deploy it to K8s with the above YAML file: kubectl apply -f service.yaml. 

Did it work? Is the health endpoint being called? Simply confirming that the pod has not been restarted (kubectl get pods) doesn't really tell us the health check is passing. Maybe the health check isn't even being called. Some adjustments could be made to the provided health server, however, implementing the service is simple. Here is a minimal example implementation.

grpc_health_v1.RegisterHealthServer(s, &Health{}) // replaces default registration
...
type Health struct {
 grpc_health_v1.UnimplementedHealthServer
}
func (h Health) Check(context.Context, *grpc_health_v1.HealthCheckRequest) (*grpc_health_v1.HealthCheckResponse, error) {
 log.Default().Println("serving health")
 return &grpc_health_v1.HealthCheckResponse{Status: grpc_health_v1.HealthCheckResponse_SERVING}, nil
}

This example changes the grpc_health_v1 registration to a custom health server that will log and return SERVING. The steps below can be used to deploy the updated service so that it logs (kubectl logs deploy/example-deploy) a new line every 10 seconds. That is the period configured in the YAML above. 

1. Build a new Docker image: docker build -t example:v1 .
2. Update the service.yaml to reference the new version of the Docker image: (image: example:v1).
3. Deploy to K8s: (kubectl apply -f service.yaml).

This confirms the health check is being called. Should we have a little fun with it?  It's simple to put the deployed service pod into crash loop backoff. Just change the returned health status to grpc_health_v1.HealthCheckResponse_NOT_SERVING.

Conclusion

Over a short period of time K8s support for gRPC health checks has moved from full custom solutions to native support. Kubernetes 1.25 (and probably above) requires no extra components to health check your service through the gRPC framework.  Be sure to locate information that is specific to the Kubernetes version you are using when reading documentation and articles. Build from the basic example with verification at each step to review your progress. The subsequent steps might be deprecated even if not documented as such.