https://pinot.apache.org/ logo
Join Slack
Powered by
Hello... i have a question related to Kafka and SS...
# general
s
Hello... i have a question related to Kafka and SSL specifically. I submitted schema and REALTIME table specs but i can see that my SSL configuration is not correct. I would like to understand for a standard deployment of Pinot using Helm on K8s where i would expect the SSL cert location to be to i can configure SLL correctly for my table - adding a segment of my spec to the thread
Copy code
"tableIndexConfig": {
        "loadMode": "MMAP",
        "streamConfigs": {
            "streamType": "kafka",
            "security.protocol": "SSL",
            "ssl.truststore.location": "/opt/pinot/kafka.client.truststore.jks",
            "stream.kafka.topic.name": "MY-TOPIC",
            "stream.kafka.consumer.type": "lowlevel",
            "stream.kafka.consumer.prop.auto.offset.reset": "largest",
            "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
            "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
            "realtime.segment.flush.threshold.rows": "0",
            "realtime.segment.flush.threshold.time": "24h",
            "realtime.segment.flush.segment.size": "100M",
            "stream.kafka.zk.broker.url": "<http://z-1.res-primary-dev.0hn5z0.c13.kafka.us-east-1.amazonaws.com:2181|z-1.res-primary-dev.0hn5z0.c13.kafka.us-east-1.amazonaws.com:2181>",
            "stream.kafka.broker.list": "<http://b-3.res-primary-dev.0hn5z0.c13.kafka.us-east-1.amazonaws.com:9094,b-2.res-primary-dev.0hn5z0.c13.kafka.us-east-1.amazonaws.com:9094,b-1.res-primary-dev.0hn5z0.c13.kafka.us-east-1.amazonaws.com:9094|b-3.res-primary-dev.0hn5z0.c13.kafka.us-east-1.amazonaws.com:9094,b-2.res-primary-dev.0hn5z0.c13.kafka.us-east-1.amazonaws.com:9094,b-1.res-primary-dev.0hn5z0.c13.kafka.us-east-1.amazonaws.com:9094>",
            "schema.registry.url": "kafka-schema-registry-cp-schema-registry.kafka-schema-registry.svc.cluster.local:8081",
            "stream.kafka.decoder.prop.schema.registry.rest.url":  "kafka-schema-registry-cp-schema-registry.kafka-schema-registry.svc.cluster.local:8081"
        }
    }
@User
m
s
Thanks @User - i actually read this one but was unsure of some actual values to use. I can keep experimenting. For example what should the 
ssl.truststore.location
be and do i need to do any setup
m
@User ^^
a
Hi @User - afaik the Helm template wasn’t updated yet to handle the injection of SSL certs and keystores. The fastest way to get you to a working setup would be to create a configmap with prep’d keystore/truststore and then hack the deployment spec to include them as local volumes on the path you set up above. If you build out a more generic solution, we’d be very glad to include them in the chart as a contribution. There’s just so much to do around Pinot right now.
s
thats very useful to know - makes sense, thanks both!
@User @User - finally finding some Pinot time this morning... the SSL solution is straightforward it turns out; • I logged into the Pinot K8s Pods to confirm that the openjdk-11 is used; as shown in example settings below, the standard truststore and password are known and can be added in the spec for creating the REALTIME table. `Example`: for a K8s deployment consuming from AWS MSK with SSL enabled and using Kafka confluent schema registry for AVRO messages;
Copy code
"security.protocol": "SSL",
"ssl.keystore.location": "/usr/local/openjdk-11/lib/security/cacerts",
"ssl.keystore.password": "changeit",
"stream.kafka.zk.broker.url": "z-1.SERVER:2181",
"stream.kafka.broker.list": "b-3.SERVER:9094,b-2.SERVER:9094,b-1.SERVER:9094",
"schema.registry.url": "<http://kafka-schema-registry-cp-schema-registry.kafka-schema-registry.svc.cluster.local:8081>",
"stream.kafka.decoder.prop.schema.registry.rest.url":  "<http://kafka-schema-registry-cp-schema-registry.kafka-schema-registry.svc.cluster.local:8081>"
Im delighed to see messages coming through in Pinot Server logs!
🙌 1
a
👍
m
Awesome, thanks for sharing @User
@User can we create an issue for help template update for SSL certs and keystore?
a
err … I don’t parse. Just create a GH issue with the details?
s
but just to be clear; i dont think it an issue; maybe a note in the docs with a recommended default setting for truststore . i did not need to change my K8s setup but simply point to the correct cert location in the spec. If you know you know but given i spend most of my time these days in a python world, i had to dig around to know where that cert lives - and for any Pinot docker version, it will always live in the same place
Open in Slack
PreviousNext
Powered by