Enable encryption in transit

YugabyteDB Anywhere allows you to protect data in transit by using the following:

Server-to-server encryption for intra-node communication between YB-Master and YB-TServer nodes.
Client-to-server encryption for communication between clients and nodes when using CLIs, tools, and APIs for YSQL and YCQL.

Note

Client-to-server encryption in transit is not supported for YEDIS. Before you can enable client-to-server encryption, you first must enable server-to-server encryption.

YugabyteDB Anywhere lets you create a new self-signed certificate, use an existing self-signed certificate, or upload a third-party certificate from external providers, such as Venafi or DigiCert (which is only available for an on-premises cloud provider).

You can enable encryption in transit (TLS) during universe creation and change these settings for an existing universe.

Self-signed certificates generated by YugabyteDB Anywhere

YugabyteDB Anywhere can create self-signed certificates for each universe. These certificates may be shared between universes within a single instance of YugabyteDB Anywhere. The certificate name has the following format:

yb-environment-universe_name, where environment is the environment type (either dev, stg, demo, or prod) that was used during the tenant registration (admin user creation), and universe-name is the provided universe name. YugabyteDB Anywhere generates the root certificate, root private key, and node-level certificates (assuming node-to-node encryption is enabled), and then provisions those artifacts to the database nodes any time nodes are created or added to the cluster. The following three files are copied to each node:

The root certificate (ca.cert).
The node certificate (node.ip_address.crt).
The node private key (node.ip_address.key).

YugabyteDB Anywhere retains the root certificate and the root private key for all interactions with the cluster.

Use YugabyteDB Anywhere-generated certificates to enable TLS

When you create a universe, you can enable TLS using certificates generated by YugabyteDB Anywhere, as follows:

Create a new universe via Universes > Create Universe and then configure it.
Based on your requirements, select Enable Node-to-Node TLS or Enable Client-to-Node TLS or both.
Choose an existing certificate from the Root Certificate list or create a new certificate by accepting the default option Create new certificate.

To view the certificate, navigate to Configs > Security > Encryption in Transit > Self Signed.

You can also modify TLS settings for an existing universe, as follows:

Navigate to either Dashboard or Universes and open a specific universe.
Click Actions > Edit Security > Encryption in-Transit to open the TLS Configuration dialog and then proceed as follows:
- If encryption in transit is currently disabled for the universe, enable it via the Encryption in Transit for this Universe field, as per the following illustration:
  
  Use the expanded TLS Configuration dialog shown in the following illustration to change the settings to meet your requirements:
- If encryption in transit is currently enabled for the universe, you can either disable or modify it, as follows:
  - To disable encryption in transit, disable the Encryption in Transit for this Universe field and then click OK.
  - To modify encryption in-transit settings, leave the Encryption in Transit for this Universe field enabled and make the necessary changes to other fields.
    
    If you are changing certificates, you need to be aware that this requires restart of the YB-Master and YB-TServer processes and can result in downtime. To avoid downtime, you should accept the default value (enabled) for the Rolling Upgrade field to trigger a sequential node-by-node change with a specific delay between node upgrades (as opposed to a simultaneous change of certificates in every node which occurs when the Rolling Upgrade field is disabled). If you select the Create new certificate option when changing certificates, the corresponding certificates will be rotated, that is, replaced with new certificates.

Self-signed self-provided certificates

Instead of using YugabyteDB Anywhere-provided certificates, you can use your own self-signed certificates that you upload to YugabyteDB Anywhere by following the procedure described in Use self-signed self-provided certificates to enable TLS.

The certificates must meet the following criteria:

Be in the .crt format and the private key must be in the .pem format, with both of these artifacts available for upload.
Contain IP addresses of the target database nodes or DNS names as the Subject Alternative Names (wildcards are acceptable).

YugabyteDB Anywhere produces the node (leaf) certificates from the uploaded certificates and copies the certificate chain, leaf certificate, and private key to the nodes in the cluster.

Use self-signed self-provided certificates to enable TLS

When you create a universe, you can enable TLS using your own certificates, as follows:

Navigate to Configs > Security > Encryption in Transit.
Click Add Certificate to open the Add Certificate dialog.
Select Self Signed.
Click Upload Root Certificate, then browse to the root certificate file (<file-name>.crt) and upload it.
Click Upload Key, then browse to the root certificate file (<file-name>.key) and upload it.
In the Certificate Name field, enter a meaningful name for your certificate.
In the Expiration Date field, specify the expiration date of the root certificate. To find this information, execute the openssl x509 -in <root-crt-file-path> -text -noout command and note the Validity Not After date.
Click Add to make the certificate available.
Go to Universes > Create Universe to open the Create Universe dialog.
Configure the universe.
Based on your requirements, select Enable Node-to-Node TLS and Enable Client-to-Node TLS.
Select an existing certificate from the Root Certificate list and then select the certificate that you have uploaded.
Create the universe.

You can also modify TLS settings for an existing universe by navigating to Universes, opening a specific universe, clicking Actions > Edit Security > Encryption in-Transit to open the TLS Configuration dialog, and then following the procedure described in Use YugabyteDB Anywhere-generated certificates to enable TLS for an existing universe.

Custom CA-signed self-provided certificates

For universes created with an on-premise cloud provider, instead of using self-signed certificates, you can use third-party certificates from external CAs. The third-party CA root certificate must be configured in YugabyteDB Anywhere. You have to copy the custom CA root certificate, node certificate, and node key to the appropriate database nodes using the procedure described in Use custom CA-signed certificates to enable TLS.

The certificates must adhere to the following criteria:

Be stored in a .crt file, with both the certificate and the private key being in the PEM format.

If your certificates and keys are stored in the PKCS12 format, you can convert them to the PEM format.
Contain IP addresses of the database nodes or DNS names as the Subject Alternative Names (wildcards are acceptable).

Use custom CA-signed certificates to enable TLS

The following procedure describes how to install certificates on the database nodes. You have to repeat these steps for every database node that is to be used in the creation of a universe.

Step 1: Obtain the keys and the custom CA-signed certificates for each of the on-premise nodes for which you are configuring node-to-node TLS. In addition, obtain the keys and the custom signed certificates for client access for configuring client-to-node TLS.

Step 2: For each on-premise node, copy the custom CA root certificate, node certificate, and node key to that node's file system.

If you are enabling client-to-node TLS, make sure to copy the client certificate and client key to each of the nodes.

In addition, ensure the following:

The file names and file paths of different certificates and keys are identical across all the database nodes. For example, if you name your CA root certificate as ca.crt on one node, then you must name it ca.crt on all the nodes. Similarly, if you copy ca.crt to /opt/yugabyte/keys on one node, then you must copy ca.crt to the same path on other nodes.
The yugabyte system user has read permissions to all the certificates and keys.

Step 3: Create a CA-signed certificate in YugabyteDB Anywhere, as follows:

Navigate to Configs > Security > Encryption in Transit.
Click Add Certificate to open the Add Certificate dialog.
Select CA Signed, as per the following illustration:
Upload the custom CA root certificate as the root certificate.

If you use an intermediate CA/issuer, but do not have the complete chain of certificates, then you need to create a bundle by executing the cat intermediate-ca.crt root-ca.crt > bundle.crt command, and then use this bundle as the root certificate. You might also want to verify the certificate chain.
Enter the file paths for each of the certificates on the nodes. These are the paths from the previous step.
In the Certificate Name field, enter a meaningful name for your certificate.
Use the Expiration Date field to specify the expiration date of the certificate. To find this information, execute the openssl x509 -in <root_crt_file_path> -text -noout command and note the Validity Not After date.
Click Add to make the certificate available.
Go to Universes > Create Universe to open the Create Universe dialog.
Configure the universe.
Based on your requirements, select Enable Node-to-Node TLS and Enable Client-to-Node TLS.
Select an existing certificate from the Root Certificate list and then select the certificate that you have uploaded.
Create the universe.

You can rotate certificates for universes configured with the same type of certificates. This involves replacing existing certificates with new database node certificates.

Convert certificates and keys from PKCS12 to PEM format

If your certificates and keys are stored in the PKCS12 format, you can convert them to the PEM format using OpenSSL.

You start by extracting the certificate via the following command:

openssl pkcs12 -in cert-archive.pfx -out cert.pem -clcerts -nokeys

To extract the key and write it to the PEM file unencrypted, execute the following command:

openssl pkcs12 -in cert-archive.pfx -out key.pem -nocerts -nodes

If the key is protected by a passphrase in the PKCS12 archive, you are prompted for the passphrase.

Verify certificate chain

To verify the certificate chain, execute the following command:

openssl verify -CAfile bundle.pem cert.pem

The bundle.pem file is a certificate bundle containing the root certificate and any intermediate certificates in the PEM format.

Rotate custom CA-signed certificates

You can rotate certificates for universes configured with the same type of certificates. This involves replacing existing certificates with new database node certificates.

You rotate the existing custom certificates and replace them with new database node certificates issued by the same custom CA that issued the original certificates as follows:

Step 1: Follow Step 1 of Use custom CA-signed certificates to enable TLS to obtain a new set of certificates for each of the nodes.

Step 2: Follow Step 2 of Use custom CA-signed certificates to enable TLS to copy the certificates to the respective nodes.

Step 3: Follow Step 3 of Use custom CA-signed certificates to enable TLS to create a new CA-signed certificate in YugabyteDB Anywhere.

Step 4: Edit the universe to use the new certificates, as follows:

Navigate to the universe for which you are rotating the keys.
Select Actions > Edit Security, as shown in the following illustration:
Select Encryption in-Transit to open the TLS Configuration dialog.
Complete the TLS Configuration dialog shown in the following illustration:
- Select the new certificate which you created in Step 3.
- Modifying certificates requires restart of YB-Master and YB-TServer processes, which can result in downtime. To avoid downtime, you should accept the default value (enabled) for the Rolling Upgrade field to trigger a sequential node-by-node change with a specific delay between node upgrades (as opposed to a simultaneous change of certificates in every node which occurs when the Rolling Upgrade field is disabled).
- Click OK.
Typically, this process takes time, as it needs to wait for the specified delay interval after each node is upgraded.

Expand the universe

You can expand universes configured with custom CA-signed certificates.

Before adding new nodes to expand an existing universe, you need to prepare those nodes by repeating Step 2 of Use custom CA-signed certificates to enable TLS for each of the new nodes you plan to add to the universe. You need to ensure that the certificates are signed by the same external CA and have the same root certificate. In addition, ensure that you copy the certificates to the same locations that you originally used when creating the universe.

When the universe is ready for expansion, complete the Edit Universe dialog to add new nodes.

Custom HashiCorp Vault-provided certificates

YugabyteDB Anywhere allows you to add an encryption in transit configuration using HashiCorp Vault with a public key infrastructure (PKI) secret engine. This configuration can be used to enable TLS for different clusters and YugabyteDB instances. You can apply this configuration to node-to-node encryption, client-to-node encryption, or both.

For the correct configuration, the following criteria must be met:

HashiCorp Vault is unsealed.
HashiCorp Vault with the PKI secret engine is configured and enabled.
HashiCorp Vault URL is accessible by YugabyteDB Anywhere.
Because HashiCorp Vault is accessed via an authentication token mechanism, a token must be created beforehand while creating a key provider with appropriate permissions.
HashiCorp Vault needs to be running and always accessible to YugabyteDB Anywhere.
HashiCorp PKI certificate revocation list (CRL) or CA URLs must be accessible from each node server.
Appropriate certificates and roles have been created for YugabyteDB Anywhere usage.
Node servers are able to validate certificates.
Required permissions have been provided to perform various key management operations.

For details, see Configure HashiCorp Vault.

Configure HashiCorp Vault

Before you can start configuring HashiCorp Vault, install it on a virtual machine, as per instructions provided in Install Vault. The vault can be set up as a multi-node cluster. Ensure that your vault installation meets the following requirements:

Has transit secret engine enabled.
Its seal and unseal mechanism is secure and repeatable.
Its token creation mechanism is repeatable.

You need to configure HashiCorp Vault in order to use it with YugabyteDB Anywhere, as follows:

Create a vault configuration file that references your nodes and specifies the address, as follows:

storage "raft" {
 path  = "./vault/data/"
 node_id = "node1"
}

listener "tcp" {
 address   = "127.0.0.1:8200"
 tls_disable = "true"
}

api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
ui = true
disable_mlock = true
default_lease_ttl = "768h"
max_lease_ttl = "8760h"

Replace 127.0.0.1 with the vault web address.

For additional configuration options, see Parameters.

Initialize the vault server by following instructions provided in Operator init.
Allow access to the vault by following instructions provided in Unsealing.
Enable the secret engine by executing the following command:
```
vault secrets enable pki
```

Configure the secret engine, as follows:

Create a root CA or configure the top-level CA.
Optionally, create an intermediate CA chain and sign them.

Create an intermediate CA for YugabyteDB, as per the following example:

export pki=pki
export pki_int="pki_int"
export role_i=RoleName
export ip="s.test.com"

vault secrets enable -path=$pki_int pki
vault secrets tune -max-lease-ttl=43800h $pki_int
vault write $pki_int/intermediate/generate/internal common_name="test.com Intermediate Authority" ttl=43800h -format=json | jq -r '.data.csr' > pki_int.csr

\# *** dump the output of the preceding command in pki_int.csr

vault write $pki/root/sign-intermediate csr=@pki_int.csr format=pem_bundle ttl=43800h -format=json | jq -r .data.certificate > i_signed.pem

\# *** dump the output in i_signed.pem

vault write $pki_int/intermediate/set-signed certificate=@i_signed.pem
vault write $pki_int/config/urls issuing_certificates="http://127.0.0.1:8200/v1/pki_int/ca" crl_distribution_points="http://127.0.0.1:8200/v1/pki_int/crl"

Create the vault policy, as per the following example:

# Enable secrets engine
path "sys/mounts/*" {
 capabilities = ["create", "read", "update", "delete", "list"]
}

# List enabled secrets engine
path "sys/mounts" {
 capabilities = ["read", "list"]
}

# Work with pki secrets engine
path "pki*" {
 capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

Generate a token with appropriate permissions (as per the referenced policy) by executing the following command:
```
vault token create -no-default-policy -policy=pki_policy
```
You may also specify the following for your token:
- ttl — Time to live (TTL). If not specified, the default TTL of 32 days is used, which means that the generated token will expire after 32 days.
- period — If specified, the token can be infinitely renewed.
Create a role that maps a name in the vault to a procedure for generating a certificate, as follows:
```
vault write <PKI_MOUNT_PATH>/roles/<ROLE_NAME> allow_any_name=true allow_subdomains=true max_ttl="8640h"
```
Credentials are generated against this role.

Issue certificates for nodes or a YugabyteDB client:

For a node, execute the following:

vault write <PKI_MOUNT_PATH>/issue/<ROLE_NAME> common_name="<NODE_IP_ADDR>" ip_sans="<NODE_IP_ADDR>" ttl="860h"

For YugabyteDB client, execute the following:

vault write <PKI_MOUNT_PATH>/issue/<ROLE_NAME> common_name="<CLIENT_DB_USER>"

Use HashiCorp Vault-provided certificates to enable TLS

When you create a universe, you can enable TLS using certificates provided by HashiCorp Vault, as follows:

Navigate to Configs > Security > Encryption in Transit.
Click Add Certificate to open the Add Certificate dialog.
Select Hashicorp.
In the Config Name field, enter a meaningful name for your configuration.
In the Vault Address field, specify a valid URL that includes the port number. The format is http://0.0.0.0:0000, which corresponds to VAULT_HOSTNAME:0000
In the Secret Token field, specify the secret token for the vault.
In the Role field, specify the role used for creating certificates.
Optionally, provide the secret engine path on which the PKI is mounted. If you do not supply this information, pki/ will be used.
Click Add to make the certificate available.
Go to Universes > Create Universe to open the Create Universe dialog.
Configure the universe.
Based on your requirements, select Enable Node-to-Node TLS and Enable Client-to-Node TLS.
Select an existing certificate from the Root Certificate list and then select the certificate that you have uploaded.
Create the universe.

You can also edit TLS settings for an existing universe by navigating to Universes, opening a specific universe, clicking Actions > Edit Security > Encryption in-Transit to open the TLS Configuration dialog, and then modifying the required settings.

Kubernetes cert-manager

For a universe created on Kubernetes, YugabyteDB Anywhere allows you to configure an existing running instance of the cert-manager as a TLS certificate provider for a cluster, assuming that the following criteria are met:

The cert-manager is running in the Kubernetes cluster.
A root or intermediate CA (either self-signed or external) is already configured on the cert-manager. The same root certificate file must be prepared for upload to YugabyteDB Anywhere.
An Issuer or ClusterIssuer Kind is configured on the cert-manager and is ready to issue certificates using the previously-mentioned root or intermediate certificate.

During the universe creation, you can enable TLS certificates issued by the cert-manager, as follows:

Upload the root certificate to YugabyteDB Anywhere:
- Prepare the root certificate in a file (for example, root.crt).
- Navigate to Configs > Security > Encryption in Transit and click Add Certificate.
- On the Add Certificate dialog shown in the following illustration, select K8S cert-manager:
- In the Certificate Name field, enter a meaningful name for your certificate configuration.
- Click Upload Root Certificate and select the root certificate file that you prepared.
- Click Add to make the certificate available.
Configure the Kubernetes-based cloud provider by following instructions provided in Configure region and zones. In the Add new region dialog shown in the following illustration, you would be able to specify the Issuer name or the ClusterIssuer name for each zone. Because an Issuer Kind is a Kubernetes namespace-scoped resource, the zone definition should also set the Namespace field value if an Issuer Kind is selected:
Create the universe:
- Navigate to Universes and click Create Universe.
- In the Provider field, select the cloud provided that you have configured in step 2.
- Complete the fields based on your requirements, and select Enable Node-to-Node TLS or Enable Client-to-Node TLS.
- Select the root certificate that you have uploaded in step 1.
- Click Create.

Troubleshoot

If you encounter problems, you should verify the name of Issuer or ClusterIssuer in the Kubernetes cluster, as well as ensure that the Kubernetes cluster is in Ready state. You can use the following commands:

kubectl get ClusterIssuer

kubectl -n <namespace> Issuer

Connecting to clusters

Using TLS, you can connect to the YSQL and YCQL endpoints.

Connect to a YSQL endpoint with TLS

If you created your universe with the Client-to-Node TLS option enabled, then you must download client certificates to your client computer to establish connection to your database, as follows:

Navigate to the Certificates page and then to your universe's certificate.
Click Actions and select Download YSQL Cert, as shown in the following illustration. This triggers the download of the yugabytedb.crt and yugabytedb.key files.
Optionally, when connecting to universes that are configured with custom CA-signed certificates, obtain the root CA and client YSQL certificate from your administrator. These certificates are not available on YugabyteDB Anywhere for downloading.
For testing with a ysqlsh client, paste the yugabytedb.crt and yugabytedb.key files into the <home-dir>/.yugabytedb directory and change the permissions to 0600, as follows:
```
mkdir ~/.yugabytedb; cd ~/.yugabytedb
cp <DownloadDir>/yugabytedb.crt .
cp <DownloadDir>/yugabytedb.key .
chmod 600 yugabytedb.*
```

Run ysqlsh using the sslmode=require option, as follows:

cd <yugabyte software install directory>
bin/ysqlsh -h 172.152.43.78 -p 5433 sslmode=require

ysqlsh (11.2-YB-2.3.3.0-b0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

yugabyte=#

To use TLS from a different client, consult the client-specific documentation. For example, if you are using a PostgreSQL JDBC driver to connect to YugabyteDB, see Configuring the client for more details.

If you are using PostgreSQL/YugabyteDB JDBC driver with SSL, you need to convert the certificates to DER format. To do this, you need to perform only steps 6 and 7 from Set up SSL certificates for Java applications section after downloading the certificates.

Connect to a YCQL endpoint with TLS

If you created your universe with the Client-to-Node TLS option enabled, then you must download client certificates to your client computer to establish connection to your database, as follows:

Navigate to the Certificates page and then to your universe's certificate.
Click Actions and select Download Root Cert, as shown in the following illustration. This triggers the download of the root.crt file.
Optionally, when connecting to universes that are configured with custom CA-signed certificates, obtain the root CA and client YSQL certificate from your administrator. These certificates are not available on YugabyteDB Anywhere for downloading.
Set SSL_CERTFILE environment variable to point to the location of the downloaded root certificate.

Run ycqlsh using the -ssl option, as follows:

cp <DownloadDir>/root.crt ~/.yugabytedb/root.crt
export SSL_CERTFILE=~/.yugabytedb/root.crt
bin/ycqlsh 172.152.43.78 --ssl

Connected to local cluster at 172.152.43.78:9042.
[ycqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
ycqlsh>

To use TLS from a different client, consult the client-specific documentation. For example, if you are using a Cassandra driver to connect to YugabyteDB, see SSL .

Validating certificates

When configuring and using certificates, SSL issues may occasionally arise. You can validate your certificates and keys as follows:

Verify that the CA CRT and CA private key match by executing the following commands:

openssl rsa -noout -modulus -in ca.key | openssl md5
openssl x509 -noout -modulus -in ca.crt | openssl md5

\# outputs should match

Verify that the CA CRT is actually a certificate authority by executing the following command:

openssl x509 -text -noout -in ca.crt

\# Look for fields

X509v3 Basic Constraints:

  CA:TRUE

Verify that certificates and keys are in PEM format (as opposed to the DER or other format). If these artifacts are not in the PEM format and you require assistance with converting them or identifying the format, consult Converting certificates.
Ensure that the private key does not have a passphrase associated with it. For information on how to identify this condition, see Decrypt an encrypted SSL RSA private key.

Enforcing TLS versions

As TLS 1.0 and 1.1 are no longer accepted by PCI compliance, and considering significant vulnerabilities around these versions of the protocol, it is recommended that you migrate to TLS 1.2 or later versions.

You can set the TLS version for node-to-node and client-node communication. To enforce TLS 1.2, add the following flag for YB-TServer:

ssl_protocols = tls12

To enforce the minimum TLS version of 1.2, you need to specify all available subsequent versions for YB-TServer, as follows:

ssl_protocols = tls12,tls13

In addition, as the ssl_protocols setting does not propagate to PostgreSQL, it is recommended that you specify the minimum TLS version (ssl_min_protocol_version) for PostgreSQL by setting the following YB-TServer gflag:

--ysql_pg_conf_csv="ssl_min_protocol_version=TLSv1.2"