How to encrypt data at rest with a Customer Managed Key with Terraform

Why encrypt data at rest with a CMEK

Default encryption

As detailed in Google Cloud documentation :

“All data stored within Google Cloud is encrypted at rest”

This default behavior for which no setup or configuration is needed manages the encryption of data at rest using AES-256 encryption standards. Therefore, why should I use another encryption mechanism?

You may encounter industry requirements such as compliance (GDPR, Bank regulations…) or locality of cryptographical material (ownership of the key against Cloud Act) that cannot be met with the Google default’s encryption mechanism.

Customer-Managed Encryption Key

Customer-managed encryption keys (CMEK) are encryption keys managed by the customer with Cloud KMS. This GCP feature will enable you to own, manage and rotate the encryption keys and therefore have more control over the keys used to encrypt data at rest.

When used, the CMEK is within your control, and data protected with a CMEK key cannot be decrypted without access to that key.

Using CMEK gives you control over more aspects of the lifecycle and management of your keys, such as (but not limited to) the following abilities:

• You can control Google's ability to decrypt data at rest by disabling the keys used to protect that data.
• You can protect your data using a key that meets a specific locality or residency requirement.
• You can automatically or manually rotate the keys used to protect your data at rest.
• You can protect your data using a Cloud HSM key, a Cloud External Key Manager key, or an existing key that you import into Cloud KMS.

ℹ️ Using CMEK incurs additional costs related to Cloud KMS and additional maintenance. Therefore, if you do not have any specific requirements regarding your keys to encrypt data at rest, google cloud’s default encryption at rest is your best choice; keep it simple!

Customer-Supplied Encryption Key

In the following part, you will learn how to create and use a CMEK key.

Instead of creating a CMEK, you can rather use your own key to avoid storing it inside GCP Cloud KMS. It is called Customer-Supplied Encryption Key and enables you to encrypt data at rest by using keys from your industry’s HSM, for example. As shown in the below picture from Google Cloud Blog, customer-managed encryption is the middle ground between the more automated process (default encryption) and the more controlled process (customer-supplied encryption).

Create a Customer Managed Key with Terraform

In this part, you will learn how to create a CMEK with Terraform, which will then be stored inside Cloud KMS.

User’s IAM permissions

The user who creates the keyring and crypto key needs the following IAM permissions:

• cloudkms.keyRings.getIamPolicy
• cloudkms.keyRings.setIamPolicy

These permissions are granted to the predefined roles/cloudkms.admin role.

You can learn more about granting permissions to manage keys in the Cloud KMS documentation.

Key

When creating a key, you need to add it to a keyring. In our example, we are creating a new keyring (with the google_kms_key_ring resource) alongside the key (with the google_kms_crypto_key resource).

Specific parameters you need to set are the following:

• location of your keyring and key

⚠️ Be sure to choose the same location as the location of the resource you want to encrypt with. You will not be able to encrypt data at rest stored in a different location than your key.

• rotation_period of your key

This is the period after which a new version of the key is automatically created.

• destroy_scheduled_duration of your key

This is the duration during which keys will stay in the soft-deleted state before being definitely deleted. The default duration is set to 24 hours.

⚠️ Be aware that this parameter can only be set during creation and cannot be changed afterward.

• algorithm

This defines the algorithm that will be used to encrypt your key. Please refer to GCP documentation to find different algorithms enabled for encryption.

⚠️ GKE VM Disks can be encrypted only by a key that uses symmetric encryption.

• protection_level of your key

Defines the protection type for the key between SOFTWARE protection and HSM (Hardware Security Module) protection.

Google documentation describes that “Cloud HSM is a cloud-hosted Hardware Security Module (HSM) service that allows you to host encryption keys and perform cryptographic operations in a cluster of FIPS 140-2 Level 3 certified HSMs.”

Therefore, HSM keys enable an additional hardware layer of security but come with a price 10 times higher than Software keys.

resource "google_kms_key_ring" "this" {
  name     = "my_keyring"
	# Must be the same location as the location of data at rest 
	# you will encrypt with the CMEK
  location = "europe-west1"  
}

resource "google_kms_crypto_key" "this" {
  name            = "my_key"
  key_ring        = google_kms_key_ring.this.id
	# Must be in seconds - 365 days in this example
  rotation_period = "31540000s"
	# Must be in seconds. Must be between 24 hours and 120 days - 7 days in this example
  destroy_scheduled_duration = "604800s"

  version_template {
    algorithm = "GOOGLE_SYMMETRIC_ENCRYPTION"
		# Must be SOFTWARE or HSM
    protection_level = "SOFTWARE"
  }

  lifecycle {
    prevent_destroy = true 
  }
}

⚠️ CMEK keys are sensible resources in Terraform because data at rest protected with a CMEK key cannot be decrypted without access to that key. Therefore, you can add in your module a lifecycle hook to prevent your key from being destroyed through Terraform. The lifecycle hook needs to be deleted for this resource to be deleted.

GKE CMEK integration with Terraform

A service that supports CMEK is said to have a CMEK integration. GKE has multiple CMEK integrations for protecting different types of data related to the service. Data at rest on node boot disks and application-layer secrets can be protected with CMEK.

Encrypt data at rest on node boot disks

You can protect data at rest on node boot disks, which are part of your cluster’s node pools in your GKE cluster, with the CMEK key.

⚠️ You cannot change the boot disk type, so you cannot enable customer-managed encryption for node boot disks on an existing cluster or node pool. However, you can create a new node pool for your cluster with customer-managed encryption enabled. Migrate your workflow by manually draining nodes from the old node pool. Then delete the previous node pool.

CMEK requirements

ℹ️ Your key ring and key must have the following requirements to be used to protect your node boot disk:

• The key must use symmetric encryption
• The keyring must have the same location as the GKE cluster
  
  

Compute Engine service account permission

You must grant the compute engine service account permissions to use the key. This is required for your node boot disks to access and use the CMEK. Nodes in your cluster use the compute engine service account, not your GKE service account.

Use the terraform google_project_iam_member resource to add the Cloud KMS CryptoKey Encrypter/Decrypter role to the compute engine service account.

# Datasource to get google project id
data "google_project" "project" {}

resource "google_project_iam_member" "compute-system" {
  role     = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
  member   = "serviceAccount:service-${data.google_project.project.number}@compute-system.iam.gserviceaccount.com"
}

Terraform block

To encrypt the data at rest of your node boot disks with Terraform, use boot_disk_kms_key parameter in your google_container_node_pool resource. You need to set this parameter to the Customer Managed Encryption Key you want to use to encrypt the boot disk attached to each node in the node pool.

In the following example, the KMS key previously created is used.

⚠️ To use this parameter for google_container_node_pool resource, you need to use the google-beta provider. To do so, you must add a provider block.

resource "google_container_node_pool" "this" {
  provider  =  google-beta

[...]

  node_config {
		boot_disk_kms_key = google_kms_crypto_key.this.id
[...]
  }
}

Encrypt Application-layer secrets

You can protect Kubernetes secrets with the CMEK key. Secrets are a sensitive type of data at rest stored inside your cluster that must be protected.

ℹ️ You can enable application-layer secret encryption on new or existing GKE clusters.

CMEK requirements

ℹ️ Your key ring and key must have the following requirements to be used to protect your GKE secrets:

• The keyring must have the same location as the GKE cluster
  
  

GKE service account permission

You must grant the GKE service account permissions to use the key. This is required for your cluster to access and use the CMEK.

⚠️ GKE cluster is not using the same service account as the nodes in your cluster. If you already have encrypted node boot disks, your current permissions are insufficient, and you need to add the following permission to encrypt secrets.

Use the terraform google_project_iam_member resource to add the Cloud KMS CryptoKey Encrypter/Decrypter role to the GKE service account.

# Data to get google project id
data "google_project" "project" {}

resource "google_project_iam_member" "container-engine" {
  role     = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
  member   = "serviceAccount:service-${data.google_project.project.number}@container-engine-robot.iam.gserviceaccount.com"
}

Terraform block

To encrypt data at rest stored as secrets with Terraform, use database_encryption block in your google_container_cluster resource. You need to set key_name parameter of the block to the Customer Managed Encryption Key you want to use to encrypt secrets.

In the following example, the KMS key previously created is used.

⚠️ GKE only supports keys from Cloud KMS. You cannot use another Kubernetes KMS provider or another encryption provider.

resource "google_container_cluster" "this" {

[...]
database_encryption {
    state = "ENCRYPTED"
    key_name = google_kms_crypto_key.this.id
  }
}

Other resources

You might want to protect other services in your infrastructure, such as GCP secrets, with the CMEK key.

⚠️ As for previous resources, if you temporarily disable or permanently destroy the CMEK key, the data at rest encrypted with that key cannot be decrypted and therefore will be inaccessible.

CMEK requirements

ℹ️ Your key ring and key must have the following requirements to be used to protect your GCP secrets:

• The key must use symmetric encryption
• The keyring must have the same location as the GKE cluster
  
  

Service agent identity

You must create a service agent identity and grant this service identity access to the CMEK Cloud KMS key created to encrypt and decrypt your secrets.

Use the terraform google_project_service_identity resource to create the service agent identity.

⚠️ Once again, you will need the google-beta provider. To do so, you must add a provider block with your resource.

Then use the google_kms_crypto_key_iam_binding resource to add the Cloud KMS CryptoKey Encrypter/Decrypter role for a specific key to the identity.

⚠️ The identity binding gives encrypt/decrypt permission on a single specific key.

resource "google_project_service_identity" "secretmanager" {
  provider = google-beta
  service  = "secretmanager.googleapis.com"
}

resource "google_kms_crypto_key_iam_binding" "this" {
  crypto_key_id = google_kms_crypto_key.this.id
  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"

  members = [
    "serviceAccount:${google_project_service_identity.secretmanager.email}"
  ]
}

Terraform block

To encrypt secrets with Terraform, use customer_managed_encryption block in your google_secret_manager_secret resource. You need to set kms_key_name parameter inside the block to the Customer Managed Encryption Key you want to use to encrypt secrets.

In the following example, the KMS key previously created is used.

⚠️ KMS key used needs to be the same one as the one the service agent identity has permission on. Otherwise, GCP secret manager will not be able to encrypt/decrypt your data at rest.

resource "google_secret_manager_secret" "this" {
	[...]
  replication {
    user_managed {
      replicas {
        customer_managed_encryption {
            kms_key_name = google_kms_crypto_key.this.id
       }
      }
    }
  }
}

Conclusion

Customer-managed keys are a good way to gain back ownership of your keys and are relatively easy to manage. Nevertheless, keep in mind that they will increase your costs and add management needs, a bit overkill if you do not have security or compliance needs.

Data at rest stored in your GKE and secrets are not the only services you can encrypt with a customer-managed key. Lots of other services in GCP offer this possibility. Do not hesitate to deep-dive search into GCP documentation to find ways to meet your requirements.

What do you think you’ll be using CMEK for? Does this service fits your business requirements, or will you stick with default GCP at rest encryption?