TLS, like its predecessor SSL, uses certificates to validate the authenticity of the communication. You can either use certificates created by a commercial certificate provider (such as Digitcert, GeoTrust, or Symantec) or you can create your own certificates. If you use a commercial provider, that provider also handles the authentication of the certificate. If you create a local or self-signed certificate, you need to provide the certificate and authentication to the server and clients yourself.

If you choose to use a locally created certificate, you must first generate the certificate key store and trust store. For traditional TLS validation of the data repository, you will need a certificate for the cluster servers. (All nodes in the cluster use the same certificate.) For mutual TLS, or mTLS, you will need separate certificates for the server and the clients.

You can create local certificates in PEM format using the freely available OpenSSL command line utility. (See the OpenSSL web site, https://openssl.org, for more information and download instructions.) For encryption with authentication of the server, you need to create a key store and a trust store for the server. For example:

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 \
  -noenc -subj "/CN=mydb.myorg.org" \
  -keyout keystore.pem -out truststore.pem

If you wish to use mutual TLS (mTLS), you need key and trust stores for both the server and the clients. The easiest way to do this is to create a common certificate to act as the root authority for both, then create key and trust stores based on the root authority. You can also append the trust store to the key store so they can easily be shared. By doing this, you can use the root authority trust store for both clients and servers. For example:

# define the root authority
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 \
  -noenc -subj "/CN=mydb.myorg.org" \
  -keyout root.keystore.pem -out root.truststore.pem
cat root.truststore.pem >> root.keystore.pem


# Derive server key from root authority
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 \
  -noenc -subj "/CN=mydb.myorg.org" \
  -CA root.keystore.pem \
  -keyout server.keystore.pem -out truststore.tmp
cat truststore.tmp >> server.keystore.pem

# Derive client key from root authority
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 \
  -noenc -subj "/CN=mydb.myorg.org" \
  -CA root.keystore.pem \
  -keyout client.keystore.pem -out truststore.tmp
cat truststore.tmp >> client.keystore.pem

Certificates specify the authenticity of the server and the truststore allows the server to authenticate the certificate provided by the client applications in mutual TLS (mTLS). But there may be times when you want to explicitly exclude certain client certificates. This is done using a certificate revocation list (CRL). You can add one or more CRLs to the Volt server by creating a directory where PEM-formatted CRLs are stored, which is then referenced in the cluster configuration.

You can create your own CRL. Or, if you purchase certificates from a commercial authority, you may receive CRLs from the vendor. CRLs received from commercial authorities are often sent in a binary format such as DER or PKCS. In this case, you need to convert the CRL to PEM format using an openssl command such as the following:

$ openssl x509 -inform der -in new-crl.der -out /etc/ssl/local/crls/new-crl.pem

Once you create the key store and the trust store, you can reference them in the database configuration file to enable TLS when initializing the database root directory. For example, using the key store and trust store created in Example 12.1, “Creating a TLS Certificate for the Database Servers”, the configuration might look like this:

deployment:
  ssl:
    enabled: true
    keystore:
      path: /etc/ssl/local/keystore.pem
    truststore:
      path: /etc/ssl/local/truststore.pem

If you are purchasing a commercial certificate, rather than creating a self-signed certificate, the certificate authority will provide you with a signed certificate, which you can use as the root authority when creating your trust and key store. If you are using certificate revocation lists (CRLs), you must also specify the CRL directory. For example:

deployment:
  ssl:
    keystore:
      path: /etc/ssl/local/keystore.pem 
    truststore:
      path: /etc/ssl/local/truststore.pem
    crl: 
      path: /etc/ssl/local/crls

Once you identify the certificates to use, you need to enable TLS for the server using the deployment.ssl.enabled property. You can also specify which ports to encrypt using subproperties of deployment.ssl:

For each type of port, you specify that the ports are either enabled ("true") or disabled ("false"). The default is false. For example, the following configuration enables TLS encryption on the external, internal, and XCDR ports:

deployment:
  ssl:
    enabled: true
    external: true
    internal: true
    dr: true
    keystore:
      path: /etc/ssl/local/keystore.pem
    truststore:
      path: /etc/ssl/local/truststore.pem

Note that if you enable TLS encryption for the XDCR port, other clusters replicating from this cluster must include the appropriate client configuration when they enable XDCR. See Section 12.7.7, “Configuring Database Replication (XDCR) With TLS/SSL” for information on setting up TLS when configuring XDCR.

Also, enabling TLS encryption on the internal port means that all intra-cluster communication must be encrypted and decrypted as it passes between nodes. Consequently, any operations that require interactions between cluster nodes (such as K-safety or multi-partition transactions) may take longer and therefore impact overall latency. Be sure to benchmark your application with and without TLS encryption before enabling internal port encryption on production systems.

Finally, it is important to note that all ports where TLS is enabled and all the servers within a single cluster use the same certificate.

If you choose to use mutual TLS, or mTLS, you need to configure both the servers and clients with the appropriate key stores and trust stores, as well as enabling mTLS on the server. To do this on the server, you must:

For example, using the key stores for client and server and the shared trust store created in Example 12.2, “Creating Mutual TLS Certificates for Both Servers and Clients” and enabling mTLS, the configuration might look like the following. Note that there are separate properties — deployment.ssl.drclientauthrequired and deployment.ssl.clientauthrequired respectively — to enable mTLS for XDCR and for the external ports, so you can choose which ports require mTLS authentication. (Mutual TLS is not available for the internal ports.) In the following example, both XDCR and external ports have mTLS enabled :

deployment:
  ssl:
    enabled: true
    external: true
    dr: true
    clientauthrequired: true
    drclientauthrequired: true
    keystore:
      path: /etc/ssl/local/server.keystore.pem
    truststore:
      path: /etc/ssl/local/root.truststore.pem

Once you enable TLS for the external interfaces on your database servers, you must also enable TLS on the command line utilities so they use the appropriate protocols to connect to the servers. (The voltdb utility is the one exception. Since it only operates on the local server it does not require a network connection.)

When invoking the command line utilities, such as voltadmin and sqlcmd, you use the --ssl option to activate encryption with TLS-enabled VoltDB servers. If the servers are using a commercially-provided certificate, you can specify sqlcmd with the --ssl option without an argument, or in the case of the other commands, with an empty string as an argument. For example:

$ sqlcmd --ssl
$ voltadmin --ssl=""

If the servers are using a local or self-signed certificate you must also specify a Java properties file as an argument to the --ssl option. For example:

$ sqlcmd --ssl=localcert.properties

The properties file must identify the filename and location of the trust store. So, using the trust store generated by the example in Section 12.7.1, “Creating the TLS/SSL Certificates”, the localcert.properties file might look like the following:

trustStore=/etc/ssl/local/truststore.pem

If you configure the servers for mTLS on the external ports and create a root certificate authority for the client and server keystores, the command line utilities can use the root trust store to validate the server, but must also include the key store for the client certificate. You do this by adding a line identifying the client key store in the properties file. The following example shows the use of a properties file, mtlscert.properties, for accessing a database configured for mutual TLS using the key and trust store files created in Example 12.2, “Creating Mutual TLS Certificates for Both Servers and Clients”:

$ cat mtlscert.properties
trustStore=/etc/ssl/local/root.truststore.pem
keyStore=/etc/ssl/local/client.keystore.pem
$ sqlcmd --ssl=mtlscert.properties

Just as the command line tools must specify how to connect to an TLS-enabled server, client applications must also establish an appropriate connection. Using the VoltDB Java API, you can enable TLS by setting the appropriate attributes of the client configuration. Specifically, if you are using a self-signed certificate, you must provide the path to the trust store. You can do this using either the .trustStore() or .trustStoreFromPropertyFile() method. For example, the following two commands are equivalent, assuming the localcert.properties file matches the properties file described in Section 12.7.5, “Using the VoltDB Command Line Utilities with TLS/SSL”:

config.trustStore("/etc/ssl/local/truststore.pem", "");
config.trustStoreFromPropertyFile("localcert.properties");

After setting the trust store properties you can enable TLS communication using the .enableSSL() method and create the client connection. For example:

ClientConfig2 config = new ClientConfig2()
             .username("JDoe")
             .password("JDsPasswd")
             .trustStoreFromPropertyFile("localcert.properties")
             .enableSSL();
client = ClientFactory.createClient(config);

If the server enables mTLS for the external ports, the client applications must provide both a trust store and a key store, in the same way the VoltDB command line utilities do. In fact, using the .trustStoreFromPropertyFile() method, you can specify the same property file specifying both the root trust store and client key store to provide mTLS authentication with the servers. Or you can explicitly identify the trust and key store using the .trustStoreWithMutualAuth() method. So, using the trust and key stores from the preceding examples, the following two statements are equivalent:

config.trustStoreWithMutualAuth(
                "/etc/ssl/local/root.truststore.pem", "",
                "/etc/ssl/local/client.keystore.pem", "");
config.trustStoreFromPropertyFile("mtlscert.properties");

When using a commercially generated certificate, you do not need to specify the trust store and can use just the .enableSSL() method.

When using TLS encryption on the DR port, the DR snapshots and binary logs are encrypted as they pass from the producer cluster to the consumer cluster. Since in XDCR the clusters act as both producer and consumer, this means that the clusters must not only have TLS enabled for the DR port, but must configure TLS for their connections to the other clusters.

Section 12.7.3, “Configuring TLS/SSL on the VoltDB Server” describes how to enable TLS encryption for the DR port, which must be done before the cluster starts. To configure TLS connectivity for communication with the other clusters, you add the ssl property to the connection list entry within the DR configuration. The value of the ssl property is either blank — for commercial certificates — or the path to a Java properties file specifying the trust store for the remote cluster(s) when using a locally-generated certificate. These property values are the same as the --ssl argument you use when running the command line utilities described in Section 12.7.5, “Using the VoltDB Command Line Utilities with TLS/SSL”.

The configuration might look like this:

XDCR Cluster

deployment:
  ssl:
    enabled: true
    dr: true
    keystore:
      path: /etc/ssl/local/keystore.pem
    truststore:
      path: /etc/ssl/local/truststore.pem
  dr:
    id: 1
    role: xdcr
    connection:
      source: NYCSvrA,NYCSvrB
      ssl: /etc/ssl/local/nyccert.properties

When using mutual TLS, the configuration and TLS property file should include the root trust store and server key store, since the server acts as both a server and a client in the XDCR relationship. For example:

XDCR Cluster With Mutual TLS

deployment:
  ssl:
    enabled: true
    dr: true
    drclientauthrequired: true
    keystore:
      path: /etc/ssl/local/server.keystore.pem
    truststore:
      path: /etc/ssl/local/root.truststore.pem
  dr:
    id: 1
    role: xdcr
    connection:
      source: NYCSvrA,NYCSvrB
      ssl: /etc/ssl/local/nycmtlscert.properties

When using commercially purchased certificates, the ssl subproperties are left blank; so each cluster can, if you choose, use a separate certificate. However, when using locally-generated certificates, there is only one properties file specified in the ssl property. So to use separate certificates for each cluster in the XDCR relationship, you should start by creating a root authority certificate as you would for mutual TLS as described in Example 12.2, “Creating Mutual TLS Certificates for Both Servers and Clients”, then create the certificates for each cluster off the root certificate. This way you can use the root trust store to authenticate all of the clusters.

TLS certificates have an expiration date. So there will come a time when you want to update or replace the existing certificate. The following instructions explain how to update the TLS certificates and CRLs on a running Volt cluster.

To update the existing TLS certificate on a running system, you simply replace the current keystore and truststore PEM files — using the exact same file names and locations as specified in the deployment.ssl.keystore.path and deployment.ssl.truststore.path properties — then issue the voltadmin ssl reload command. If the configuration also specifies a CRL directory with the deployment.ssl.crl.path property, any CRLs in that directory will be reloaded as well.

When issuing the voltadmin ssl reload command, you must use a properties file pointing to the original truststore when invoking voltadmin, since the original certificate is still in effect until the command is completed:

$ voltadmin --ssl=oldcert.properties ssl reload

After issuing the command, you will need to use a properties file pointing to the new truststore when invoking Volt command line utilities, as described in Section 12.7.5, “Using the VoltDB Command Line Utilities with TLS/SSL”.