UPD: this is not an iossue with kinesis sink, its ...
# troubleshooting
m
UPD: this is not an iossue with kinesis sink, its temporal join question - and actually exactly this problem https://stackoverflow.com/questions/70688670/flink-temporal-join-not-showing-data Hi! If anyone has an experience with aws kinesis sink cinnector - i will be really happy to hear your thoughts! Question is the following: i cant understand how bundling of outcoming records works. I found to settings which affects this: • ‘sink.batch.max-size’ seems to define max number of outcoming records, and • ‘sink.requests.max-buffered’ seems to define maximum difference between already sent and what is now in buffer. As soon as new records come and increase this difference more than this setting, records are sent to AWS. Question: if there are no new records in kinesis sink, this leftover seems to stuck there forever, waiting until new records will come. I see no timeouts which affects this behaviour. Basically i need “enrichment” pattern - when some timeout comes, send record even if it’s 1 record. How to achieve this?
j
I believe there is a
maxTimeInBufferMs
setting to handle this case.
maxTimeInBufferMS the maximum amount of time an element may remain in the buffer. In most cases elements are flushed as a result of the batch size (in bytes or number) being reached or during a snapshot. However, there are scenarios where an element may remain in the buffer forever or a long period of time. To mitigate this, a timer is constantly active in the buffer such that: while the buffer is not empty, it will flush every maxTimeInBufferMS milliseconds.
The above snippet is from the scala/java aws sdk
KinesisStreamSink
m
Thanks! I see that there is no such setting in https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kinesis/, so my guess it shoud be set under sink.producer.maxTimeInBufferMs?
a
Hi @Mikhail Spirin are you refering to Table API connector or Datastream one?
m
Im using table API connector
Anyway i see strong consistency between Table and Dataastream connectors settings, so probably the same:
Copy code
sink.http-client.read-timeout
sink.http-client.protocol.version
sink.batch.max-size
sink.requests.max-inflight
sink.requests.max-buffered
sink.flush-buffer.size
sink.flush-buffer.timeout
sink.fail-on-error
and
Copy code
kds_sink = KinesisStreamsSink.builder() \
    .set_kinesis_client_properties(sink_properties) \                      # Required
    .set_serialization_schema(SimpleStringSchema()) \                      # Required
    .set_partition_key_generator(PartitionKeyGenerator.fixed()) \          # Required
    .set_stream_name("your-stream-name") \                                 # Required
    .set_fail_on_error(False) \                                            # Optional
    .set_max_batch_size(500) \                                             # Optional
    .set_max_in_flight_requests(50) \                                      # Optional
    .set_max_buffered_requests(10000) \                                    # Optional
    .set_max_batch_size_in_bytes(5 * 1024 * 1024) \                        # Optional
    .set_max_time_in_buffer_ms(5000) \                                     # Optional
    .set_max_record_size_in_bytes(1 * 1024 * 1024) \                       # Optional
    .build()
but unfortunately doesn’t work in my case….
a
yeah, this should be the case, what do you mean it doesn't work?
m
Yes and thanks, but unfortunately doesnt work for me… 10 records are incoming, 4 + 1 is 2 kinesis sink inserts. ‘sink.flush-buffer.timeout’ = ‘5000’. 10 minutes have passed already…
If i push new incoming records, even 1 - previous buffer of 5 records will be pushed to kinesis, and again there will be a gap of this 1 record
a
Which version of the flink/the connector are you using?
m
flink 1.16, connector is flink-connector-kinesis-4.1.0-1.16.jar
a
Hmmmm, this should not be the case, did you try setting the batch size to 1 to verify? The only relevant issue I can think of is that appears after restoring from state FLINK-29827. this is solved in 1.16.1. But that only surfaces after restoring from state.
m
Its already set to 1. Like this:
Copy code
WITH (
    'connector' = 'kinesis',
    'aws.region' = 'us-west-2',
    'stream' = 'flink-eventstreamer-poc-prod',
    'format' = 'json',
    'sink.batch.max-size' = '1',
    'sink.requests.max-buffered' = '2',
    'sink.flush-buffer.timeout' = '5000'
);
I can spend some time and recreate issue without all my complications and joins, to verify that it’s not that
a
yeah, I would do that as well. If you could verify that the records are idle in the sink operator itself please raise a Jira for it, I would be happy to take a look.
m
So i didn’t manage to reproduce it with simplest scenario like here, but i will try to get what makes it stuck exactly
Copy code
CREATE TEMPORARY TABLE kafka_source (
            ...
        ) WITH (
            'connector' = 'kafka',
            'topic' = 'flink-eventstreamer-poc-dev',
            'scan.startup.mode' = 'latest-offset',
            'format' = 'json',
            'json.fail-on-missing-field' = 'false',
            'json.ignore-parse-errors' = 'true'
        );


CREATE TABLE kinesis_sink (
            ...
        ) PARTITIONED BY ( ... )
        WITH (
            'connector' = 'kinesis',
            'aws.region' = 'us-west-2',
            'stream' = 'flink-eventstreamer-poc-prod',
            'format' = 'json',
            'sink.batch.max-size' = '5',
            'sink.requests.max-buffered' = '10',
            'sink.flush-buffer.timeout' = '10000'
        );


INSERT INTO kinesis_sink
        SELECT
             ...
        FROM kafka_source;
👍 1
Ok, its not an issue wirth kinesis sink. I actually do temporal join, and this is exactly what causes the problem. That is why setting which was mentioned above, didn’t work… I see that exact behavior with SQL client. This is my code (ts is now() when i send test records)
Copy code
CREATE TEMPORARY TABLE kafka_source (
  a STRING, 
  ts TIMESTAMP(3), 
  WATERMARK FOR ts AS ts - INTERVAL '15' SECOND
) WITH (
  'connector' = 'kafka',
   ...
  'scan.startup.mode' = 'latest-offset', 
  'format' = 'json', 
  'json.fail-on-missing-field' = 'false', 
  'json.ignore-parse-errors' = 'true'
);

CREATE TEMPORARY TABLE enrichment_kafka_source (
   b INT, 
   a STRING PRIMARY KEY
) WITH (
  'connector' = 'upsert-kafka',
  ...
   'key.format' = 'raw', 
  'value.format' = 'json', 
  'properties.auto.offset.reset' = 'earliest'
);

SELECT 
  e.b AS b, 
  s.a AS a 
FROM 
  kafka_source AS s 
  LEFT JOIN enrichment_kafka_source FOR SYSTEM_TIME AS OF f.ts AS e 
ON s.a = e.a;
a
Aha, I guess you if you do a window join this should work
m
You mean something form here right? https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/queries/window-join/ I thought temporal join and this one is not connected… Cant be used at the same time
a
Aha, I think this depends on your use case sorry 😄 I meant replacing the temporal with window join.
m
Found an issue. It’s about this - https://stackoverflow.com/questions/70688670/flink-temporal-join-not-showing-data Setting “table.exec.source.idle-timeout” for table nvironment helped
👍 2