Identity-Aware Proxy (IAP) is a Google Cloud service that allows guarding access to Virtual Machines (VMs) and applications using users’ identity and context. It is often used to allow connections from computers to VMs without actually requiring exposing these VMs on the public Internet.
However, it is also useful to guard access to applications by specifying who can access them directly in GCP IAM.
IAP can be plugged into a load balancer targeting services in a cluster to secure access to specific components running in Kubernetes.
This part describes how to set up a GKE cluster with an Ingress controller plugged behind an HTTP(S) (layer 7) load balancer with IAP, used to secure access to tooling components. The following examples are given using Ingress-NGINX Controller but another Ingress controller can be used.
IAP on a load balancer requires an IAP OAuth client and credentials for this client. Everything can be created as code using Terraform:
resource "google_iap_brand" "this" {
support_email = "<support email>"
application_title = "Cloud IAP protected app"
project = var.project_id
}
resource "google_iap_client" "iap" {
display_name = "<IAP client name>"
brand = google_iap_brand.this.name
}
resource "google_secret_manager_secret" "iap_id" {
secret_id = "iap-id"
project = var.project_id
replication {
automatic = true
}
}
resource "google_secret_manager_secret_version" "iap_id" {
secret = google_secret_manager_secret.iap_id.id
secret_data = google_iap_client.iap.client_id
}
resource "google_secret_manager_secret" "iap_secret" {
secret_id = "iap-secret"
project = var.project_id
replication {
automatic = true
}
}
resource "google_secret_manager_secret_version" "iap_secret" {
secret = google_secret_manager_secret.iap_secret.id
secret_data = google_iap_client.iap.secret
}The IAP credentials are stored securely in the Secret Manager where they can be retrieved from the cluster with External Secrets for instance.
The Ingress controller and the load balancer with IAP can directly be created using Kubernetes resources.
The first thing is to retrieve the IAP OAuth credentials from the previous step so that they can be used by the load balancer. This can be done with External Secrets:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: gcp-external-iap-secret
spec:
secretStoreRef:
kind: ClusterSecretStore
# This supposes that a ClusterSecretStore "secret-store" is already in the cluster and plugged with GCP Secret Manager
# See https://external-secrets.io/v0.8.3/provider/google-secrets-manager/ to learn how to configure it
name: secret-store
target:
# Name of the Kubernetes secret ref
name: iap-secret
creationPolicy: Owner
data:
- secretKey: client_secret
remoteRef:
key: iap-secret
- secretKey: client_id
remoteRef:
key: iap-id
Then to configure the load balancer, three GKE-specific resources must be added: a BackendConfig, a FrontendConfig, and a ManagedCertificate. These resources tell GCP how to configure the front end of the layer 7 load balancer and its backend.
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: ingress-nginx-iap
spec:
# Automatically redirect HTTP to HTTPS
redirectToHttps:
enabled: true
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
name: managed-cert
spec:
# Domain names of resources that will be accessed with IAP
domains:
- "<FQDN_1>"
- "<FQDN_>"
---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: ingress-nginx-iap
spec:
iap:
# Enable IAP
enabled: true
# Kubernetes secret ref
oauthclientCredentials:
secretName: iap-secret
healthCheck:
checkIntervalSec: 30
timeoutSec: 5
healthyThreshold: 1
unhealthyThreshold: 2
type: HTTPS
requestPath: /healthz
To create the HTTP(S) load balancer, an ingress with the ingress class GCE must be added to the cluster. GKE clusters have a built-in GCE Ingress controller that creates layer 7 load balancers reading ingress configurations.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: iap-ingress
annotations:
# Redriect HTTP to HTTPS
ingress.kubernetes.io/ssl-redirect: "true"
# GCE Ingress class
kubernetes.io/ingress.class: gce
# Static IP to use for the load balancer
kubernetes.io/ingress.global-static-ip-name: "<ip resource name>"
# Name of the FrontendConfig to use
networking.gke.io/v1beta1.FrontendConfig: ingress-nginx-iap
# Name of the managed certificate to use
networking.gke.io/managed-certificates: managed-cert
spec:
defaultBackend:
service:
# Name of the ClusterIP service the load balancer targets
name: ingress-nginx-iap-controller
port:
name: https
# Secret with the certificate to use for the load balancer (if generated by cert-manager)
# See https://cert-manager.io/docs/concepts/certificate/
# tls:
# - secretName: <certificate>
Finally, the Ingress-NGINX controller must be configured with additional information for the load balancer to use IAP. The values of the Helm chart can be updated for that:
...
controller:
...
service:
type: ClusterIP
annotations:
# Protocol for communications between the LB and the Ingress Controller
cloud.google.com/app-protocols: '{"https":"HTTPS"}'
# Name of the BackendConfig (⚠️ Important for IAP to work!)
cloud.google.com/backend-config: '{"default": "iap-ingress"}'
# To use container native load balancer
# See https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing
cloud.google.com/neg: '{"ingress": true}'
# Election ID used to elect leader instance (if multiple instances of the same Ingress Controller)
# 🚧 This value has to differ between two Ingress Controllers
electionID: iap
# This section refers to the creation of the IngressClass resource
# IngressClass resources are supported since k8s >= 1.18 and required since k8s >= 1.19
ingressClassResource:
# Name of the ingressClass
# 🚧 This value has to differ between two Ingress Controllers
name: iap
# Controller-value of the controller that is processing this ingressClass
# 🚧 This value has to differ between two Ingress Controllers
controllerValue: "k8s.io/ingress-nginx-iap"
# For backwards compatibility with ingress.class annotation, use ingressClass.
# Algorithm is as follows, first ingressClassName is considered, if not present, controller looks for ingress.class annotation
# 🚧 This value has to differ between two Ingress Controllers
ingressClass: iap
Users still need to be authorized to access the Ingress controller (and tooling components). They need to be given the role roles/iap.httpsResourceAccessor either in the scope of the project or the scope of the load balancer’s IAP application resource.
The final step is to create the right Ingress resources for the tooling components to be reachable from the Ingress controller configured with IAP. It is exactly the same thing as configuring any ingress, except the IngressClass iap must be specified on the Ingress resource.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: iap-protected-ingress
spec:
# Ingress class of the Ingress controller with IAP
ingressClassName: iap
rules:
- host: "<host>"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: "<service to target>"
port:
name: "<service port to target>"
The setup that is shown in the previous part has several benefits and drawbacks.
X-Goog-Authenticated-User-Email forwards the user's email address and X-Goog-Iap-Jwt-Assertion forwards information on the authenticated user through a signed JWT. Unfortunately, some components (like ArgoCD) are not able to decode JWT headers to access user information. Relying on the X-Goog-Authenticated-User-Email header is way less secure as it does not provide proof of authenticity like the JWT signature that can be verified. This means the right Kubernetes RBAC and Network Policies must be set in the cluster to prevent users to craft their own headers when accessing components.groups information is not forwarded by IAP. Components that provide Role-Based Access Control (RBAC) need to be configured on a per-user basis, or a dedicated component to retrieve groups must be installed in the cluster (like Dex for instance).Using IAP to protect access to the web resources of GKE tooling components is interesting to easily secure and protect them while following Zero Trust principles. However, this method has drawbacks that cannot be ignored.
Especially, it does not forward groups information that allows for efficient management of user rights. It is not too much of an issue when only a few people manage the infrastructure, but it is not manageable at scale.
On large infrastructure, combining IAP with Dex can be a good alternative: IAP grants access to components from the Internet while Dex authenticates users and retrieves the necessary information (e.g. groups) for authorization (performed by components).
Of course, IAP alone is not sufficient to fully protect clusters and components. It comes as an additional layer to other mechanisms such as Role-Based Access Control (RBAC), Network Policies, or workload security hardening.