Hello everyone, can somebody help me about KafkaSi...
# troubleshooting
t
Hello everyone, can somebody help me about KafkaSink, i have problem with this. I create three KafkaSink(each produce to a topic, all topic have same settings), one DataStream sink to it and two sidoutput of this datastream sink to others. However, when i access to Kafka UI, i found that message produce by kafkaSink only exist about 1 minute, and then auto disappeared.
p
The issue lives in your topic configuration. Kafka has the following configs
log.retention.ms
and
log.retention.bytes
t
but i use go sarama produce message to same conf topic, it still durable. And i only create two small message for dev purpose.
p
Are you using exactly once producer? If so, are you sure the transactions are being commited?
t
I create three kafkaSinks and each KafkaSink produces to a separate topics. I think each KafkaSink create a producer, i use AT_LEAST_ONE mode. I try EXACTLY mode but it error and i can't find the error message, just only trace.
Copy code
KafkaSink<String> prometheusTextSink = KafkaSink.<String>builder()
        .setBootstrapServers(KAFKA_BOOTSTRAP_SERVERS)
        .setRecordSerializer(KafkaRecordSerializationSchema.builder()
            .setTopic(KAFKA_PROMETHEUS_TEXT_SINK_TOPIC)
            .setValueSerializationSchema(new SimpleStringSchema())
            .build())
        .setKafkaProducerConfig(kakfaProps)
        .setDeliveryGuarantee(KAFK_DELIVERY_GUARANTEE_SINK)
        .setTransactionalIdPrefix("streaming-metrics")
        .setProperty("transaction.timeout.ms", "60000")
        .build();

    KafkaSink<String> errorPrometheusTextSink = KafkaSink.<String>builder()
        .setBootstrapServers(KAFKA_BOOTSTRAP_SERVERS)
        .setRecordSerializer(KafkaRecordSerializationSchema.builder()
            .setTopic(KAFKA_ERROR_PROMETHEUS_TEXT_SINK_TOPIC)
            .setValueSerializationSchema(new SimpleStringSchema())
            .build())
        .setKafkaProducerConfig(kakfaProps)
        .setDeliveryGuarantee(KAFK_DELIVERY_GUARANTEE_SINK)
        .setTransactionalIdPrefix("streaming-metrics")
        .setProperty("transaction.timeout.ms", "60000")
        .build();

    KafkaSink<byte[]> prometheusRemoteKafkaSink = KafkaSink.<byte[]>builder()
        .setBootstrapServers(KAFKA_BOOTSTRAP_SERVERS)
        .setRecordSerializer(KafkaRecordSerializationSchema.builder()
            .setTopic(KAFKA_PROMETHEUS_REMOTE_WRITE_SINK_TOPIC)
            .setValueSerializationSchema(new SerializationSchema<byte[]>() {
              public byte[] serialize(byte[] element) {
                return element;
              };
            })
            .build())
        .setKafkaProducerConfig(kakfaProps)
        .setDeliveryGuarantee(KAFK_DELIVERY_GUARANTEE_SINK)
        .setTransactionalIdPrefix("streaming-metrics")
        .setProperty("transaction.timeout.ms", "60000")
        .build();
my code here, kakfaProps is just authentication info
p
I'd remove the transaction configs from your sink if you're using
AT_LEAST_ONCE
config. I think it's likely to be something related to a transaction. Even more after seeing your
transaction.timeout.ms
is 60s.
t
this additional prop i just add more when i try EXACT mode, before i add it and use AT_LEAST_ONCE, message still not durable
hmm i just try another way and now messages a durable
i move the code configure env to beginning of my code
and delete old message and create new at my source kafka topic
may be kafka have some settings about message timestamp can be add to a topic
Do you guess any reson for this problem, pls help me if you know
p
The problems from where I can see so far could be related to the things I've pointed. I'm not sure how the moving changed the outcome.