Let’s Encrypt Wildcard Cert Auto-renewal on GKE

Let’s Encrypt Wildcard Cert & GCP Cert-Manager

This article demonstrates how to create a wildcard certificate using Let’s encrypt and configure auto-renewal of a wildcard certificate using cert-manager on GKE ingress. The implementation of workload identity in GKE is also demonstrated.

Introduction

In today’s modern era, securing websites/applications plays a very important role in protecting an organization’s data from being stolen or hacked. Since these applications are accessed by end users for different purposes, they expect the confidential information shared to be protected. Therefore, it is important for modern applications to have an SSL certificate to be trusted. Generating and renewing an SSL certificate can be a daunting task, as it involves many further steps and can even lead to manual error.

Let’s Encrypt is a certificate authority that offers free TLS/SSL certificates that can be used to secure your website with HTTPS.

cert-manager is a third-party Kubernetes controller that automates the process of obtaining and renewing TLS/SSL certificates from Let’s Encrypt. The cert-manager uses the HTTP01 or DNS01 solver to verify domain ownership before generating the SSL certificate.

Note: As of today, Wildcard certificate is only supported by DNS01 solver.

Wildcard certificates are very convenient for any organization that wants to have multiple subdomains for their applications.

Prerequisites

1. GKE private cluster with workload identity enabled.
2. Helm installed.
3. Kubectl installed.
4. Reserve a global Static IP and create a DNS A record.

Note: If cluster was created without workload identity enabled, enable workload identity at the cluster level and enable the GKE Metadata Server at node pool level.

Implementation Steps

First, the nginx application is deployed and exposed using the GKE ingress controller.

- Nginx Deployment Object

This is our sample application which is used for demonstration.

Create a deployment manifest file and apply the nginx deployment using

• kubectl apply -f deployment.yaml -n cert-manager

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

- Nginx Service Object

We will expose our application deployed in the above step using the following service.

Create a Service manifest file as a Nodeport and apply the nginx service using

• kubectl apply -f service.yaml -n cert-manager

- Install Cert Manager using Helm

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yamlkubectl get all -n cert-manager

By applying the above command it will create the following deployments.

- Service Account for DNS01 solver

A service account is required to complete the ACME challenge. It proves that we are the rightful owner of the DNS zone.

Create a service account with roles Workload Identity and DNS Admin.

gcloud auth loginexport PROJECT_ID=<project_id># service account
gcloud iam service-accounts create wildcard-dns01-solver --display-name "wildcard-dns01-solver"# iam role DNS admin
gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member serviceAccount:wildcard-dns01-solver@$PROJECT_ID.iam.gserviceaccount.com \
   --role roles/dns.admin# iam role Workload Identity
gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member serviceAccount:wildcard-dns01-solver@$PROJECT_ID.iam.gserviceaccount.com \
   --role roles/iam.workloadIdentityUser

There are 2 ways to use the service account created in the above step.

1. Downloading the SA keys.
2. Using workload Identity. (Recommended & More secure)

- GKE Workload Identity

In this article, we will follow GCP best practices by using the workload identity method.

GKE workload Identity is a secure way to impersonate IAM service accounts to access any Google cloud resources.

A k8s service account with the name cert-manager is the created through helm installation.

Annotating k8s service account — In this step, we are linking k8s service account to the Google service account.

# iam binding
gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:$PROJECT_ID.svc.id.goog[cert-manager/cert-manager]" \
  wildcard-dns01-solver@$PROJECT_ID.iam.gserviceaccount.com# Annotating K8S service account
kubectl annotate serviceaccount --namespace=cert-manager cert-manager \
"iam.gke.io/gcp-service-account=wildcard-dns01-solver@$PROJECT_ID.iam.gserviceaccount.com"

- Cert Manager — Create Clusterissuer Object

A native k8s certificate management controller that can handle multiple certificate issuer sources, such as Let’s Encrypt, using the Automated Certificate Management Environment (ACME) protocol.

Create Clusterissuer manifest file. Apply the cluster issuer using

• kubectl apply -f clusterissuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace : cert-manager
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: '<email-id>'
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - dns01:
          cloudDNS:
            project: my-project
        selector:
          dnsZones:
          - '<domain-name>'
          - '*.<domain-name>'

Cert Manager — Create Certificate object

The certificate created will be stored in k8s secret. By default, rotation policy: Never is set.

With rotation policy: Always certificates will be automatically refreshed in stored secret. Cert-manager regenerates a new private key on each certificate renewal and stores it in the same secret.

Create the wildcard Certificate manifest file.

Apply the certificate using

• kubectl apply -f certificate.yaml -n cert-manager

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-prod
  namespace : cert-manager
spec:
  secretName: wildcard-prod
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: '*.<domain-name>'
  dnsNames:
  - "*.<domain-name>"

Check the certificate status post deployment of Certificate Object. Wildcard certificates will be stored in secret.

kubectl get certificate -n cert-manager

Note: A wildcard certificate status of false means that the certificate was not successfully issued. To troubleshoot the exact errors, check the logs of CertificateRequest, ACME Orders & Challenges objects.

Deploying GKE Ingress with Wildcard Certificate

Create an Ingress service object to expose the application to the Internet.

Create Ingress manifest files with tls and host fields to attach SSL certificates using the wildcard certificate stored in secrets.

Apply the ingress service using

• kubectl apply -f ingress.yaml -n cert-manager

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sampleapp-2-ingress
  namespace: cert-manager
  annotations:
    kubernetes.io/ingress.global-static-ip-name: sampleapp-ip-2
    cert-manager.io/cluster-issuer: letsencrypt-prod
    acme.cert-manager.io/dns01-edit-in-place: "true"
  labels:
    app: nginx
spec:
  tls:
  - secretName: wildcard-prod
  rules:
  - host: tls1.<domain-name>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

Test the connectivity of the application and also validate the ssl certificate attached to the domain.

Finally, the issued certificate is successfully attached to the application.

References

• https://cert-manager.io/docs/configuration/acme/
• https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity

Thank you for reading :)
Questions?
If you have any questions, I’ll be happy to read them in the comments.