Stream store with TLS

Testing pravega-tls mode:

Pravega is a stream store that comprises of a control plane server and a data plane server. The former is called controller and the latter is called segmentstore. Both are available to communicate with a client but only the controller provisions the REST api.

Both the controller and the segmentstore can enable TLS. The pravega source includes default certificates, keys, keystore and truststore in its config folder. 

TLS is enabled by including the following options in the config folder:

  controller.security.tls.enable: "true"

  controller.security.tls.trustStore.location: /opt/pravega/conf/client.truststore.jks

  controller.security.tls.server.certificate.location: /opt/pravega/conf/server-cert.crt

  controller.security.tls.server.privateKey.location: /opt/pravega/conf/server-key.key

  controller.security.tls.server.keyStore.location: /opt/pravega/conf/server.keystore.jks

  controller.security.tls.server.keyStore.pwd.location: /opt/pravega/conf/server.keystore.jks.passwd

  controller.segmentstore.connect.channel.tls: "true"

where the corresponding files are available in the pravega/pravega:latest image that is loaded on a host container.

This configuration did not work because it could not be found by the controller server as shown from the logs included here:

2020-08-03 00:04:17,403 1512 [ControllerServiceStarter STARTING] INFO  i.p.c.s.ControllerServiceStarter - Controller serviceConfig = ControllerServiceConfigImpl(threadPoolSize=80, storeClientConfig=StoreClientConfigImpl(storeType=PravegaTable, zkClientConfig=Optional[ZKClientConfigImpl(connectionString: zookeeper-client:2181, namespace: pravega/pravega, initialSleepInterval: 5000, maxRetries: 5, sessionTimeoutMs: 10000, secureConnectionToZooKeeper: false, trustStorePath is unspecified, trustStorePasswordPath is unspecified)]), hostMonitorConfig=io.pravega.controller.store.host.impl.HostMonitorConfigImpl@3b434741, controllerClusterListenerEnabled=true, timeoutServiceConfig=TimeoutServiceConfig(maxLeaseValue=120000), tlsEnabledForSegmentStore=, eventProcessorConfig=Optional[ControllerEventProcessorConfigImpl(scopeName=_system, commitStreamName=_commitStream, commitStreamScalingPolicy=ScalingPolicy(scaleType=FIXED_NUM_SEGMENTS, targetRate=0, scaleFactor=0, minNumSegments=2), abortStreamName=_abortStream, abortStreamScalingPolicy=ScalingPolicy(scaleType=FIXED_NUM_SEGMENTS, targetRate=0, scaleFactor=0, minNumSegments=2), scaleStreamName=_requeststream, scaleStreamScalingPolicy=ScalingPolicy(scaleType=FIXED_NUM_SEGMENTS, targetRate=0, scaleFactor=0, minNumSegments=2), commitReaderGroupName=commitStreamReaders, commitReaderGroupSize=1, abortReaderGroupName=abortStreamReaders, abortReaderGroupSize=1, scaleReaderGroupName=scaleGroup, scaleReaderGroupSize=1, commitCheckpointConfig=CheckpointConfig(type=Periodic, checkpointPeriod=CheckpointConfig.CheckpointPeriod(numEvents=10, numSeconds=10)), abortCheckpointConfig=CheckpointConfig(type=Periodic, checkpointPeriod=CheckpointConfig.CheckpointPeriod(numEvents=10, numSeconds=10)), scaleCheckpointConfig=CheckpointConfig(type=None, checkpointPeriod=null), rebalanceIntervalMillis=120000)], gRPCServerConfig=Optional[GRPCServerConfigImpl(port: 9090, publishedRPCHost: null, publishedRPCPort: 9090, authorizationEnabled: false, userPasswordFile is unspecified, tokenSigningKey is specified, accessTokenTTLInSeconds: 600, tlsEnabled: false, tlsCertFile is unspecified, tlsKeyFile is unspecified, tlsTrustStore is unspecified, replyWithStackTraceOnError: false, requestTracingEnabled: true)], restServerConfig=Optional[RESTServerConfigImpl(host: 0.0.0.0, port: 10080, tlsEnabled: false, keyFilePath is unspecified, keyFilePasswordPath is unspecified)])

When the values were overridden, the following error was encountered:

java.lang.IllegalArgumentException: File does not contain valid certificates: /opt/pravega/conf/client.truststore.jks

        at io.netty.handler.ssl.SslContextBuilder.trustManager(SslContextBuilder.java:182)

        at io.pravega.client.netty.impl.ConnectionPoolImpl.getSslContext(ConnectionPoolImpl.java:280)

        at io.pravega.client.netty.impl.ConnectionPoolImpl.getChannelInitializer(ConnectionPoolImpl.java:237)

        at io.pravega.client.netty.impl.ConnectionPoolImpl.establishConnection(ConnectionPoolImpl.java:194)

        at io.pravega.client.netty.impl.ConnectionPoolImpl.getClientConnection(ConnectionPoolImpl.java:128)

        at io.pravega.client.netty.impl.ConnectionFactoryImpl.establishConnection(ConnectionFactoryImpl.java:62)

        at io.pravega.client.netty.impl.RawClient.<init>(RawClient.java:87)

        at io.pravega.controller.server.SegmentHelper.updateTableEntries(SegmentHelper.java:403)

        at io.pravega.controller.store.stream.PravegaTablesStoreHelper.lambda$addNewEntry$10(PravegaTablesStoreHelper.java:178)

        at io.pravega.controller.store.stream.PravegaTablesStoreHelper.lambda$null$54(PravegaTablesStoreHelper.java:534)

        at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)

        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)

        at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.security.cert.CertificateException: found no certificates in input stream

        at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:98)

        at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:64)

        at io.netty.handler.ssl.SslContext.toX509Certificates(SslContext.java:1071)

        at io.netty.handler.ssl.SslContextBuilder.trustManager(SslContextBuilder.java:180)

        ... 19 common frames omitted

This required passing custom certificate, key, truststore and keystore to the controller.

In the form of:

        TLS_ENABLED = "true"

        TLS_KEY_FILE = "/etc/secret-volume/controller01.key.pem"

        TLS_CERT_FILE = "/etc/secret-volume/controller01.pem"

        TLS_TRUST_STORE = "/etc/secret-volume/client.truststore.jks"

        TLS_ENABLED_FOR_SEGMENT_STORE = "true"

        REST_KEYSTORE_FILE_PATH = "/etc/secret-volume/server01.keystore.jks"

        REST_KEYSTORE_PASSWORD_FILE_PATH = "/etc/secret-volume/pass-secret-tls"

These options are also hard to pass without the use of environment variables especially when used in conjunction with other software that installs and configures the pravega container image. One of the ways to overcome this is to edit the configmap with the overrides above and recreate the pods.