Hi all, any idea about this error: Table sink '*...
# troubleshooting
i
Hi all, any idea about this error: Table sink '*anonymous_datastream_sink$1*' doesn't support consuming update and delete changes which is produced by node ChangelogNormalize(key=[userid])
d
Can you clarify exactly which sync you are using?
Basically we should first check if that sync or another suitable one exists to support updates/deletes. If not then we are probably looking at a two-step process
i
• Store the data using paimon with the following options:
Copy code
" 'write-mode' = 'upsert'," +
	" 'sequence.field' = 'date', " +
Copy code
tableEnv.getConfig().getConfiguration().setString("table.exec.sink.upsert-materialize", "none");
	tableEnv.getConfig().getConfiguration().setString("changelog-producer", "input");
        tableEnv.getConfig().getConfiguration().setString("changelog.time-retained", "1 day");
and after I'm trying to show this information with
tableEnv.toDataStream(source).print();
the table is defined a dynamic bucket and primary key without partition
@D. Draco O'Brien, I think that the question would be If I'm creating a table only with PK, Should I create a snapshot to get results in a query? , or I have to process the query as bacth instead of streaming ?
only I'm trying to do :
tableEnv.executeSql("SELECT * FROM table LIMIT 5").print();
d
I dont think batch is optimal here because of the nature of CDC and it might miss some important data.
• Just to be clear you are using https://nightlies.apache.org/flink/flink-cdc-docs-master/docs/connectors/pipeline-connectors/paimon/ correct? If this is correct than it states “Only support Paimon primary key table, so the source table must have primary keys.”
As the Sync? And Debezium CDC as the source?
i
I'm reading from CDC but with kafka, the case is that kafka topic is populated from Yugabyte debezium, so I'm using as source :
and after I move the data to apachae paimon (s3)
so :
Screenshot 2024-07-11 at 15.51.22.png
and after i create paimon table.
and I insert the topic_products in paimon table
d
How are you creating the Paimon table and inserting into it?
It sounds like you are not using the Paimon Sync
i
more or less:
Copy code
"CREATE CATALOG paimon WITH ('type' = 'paimon', 'warehouse' = '%s', 's3.access-key' = '%s', 's3.secret-key' = '%s')
		
		tableEnv.executeSql("USE CATALOG paimon");

				"CREATE TABLE IF NOT EXISTS  name  (" +
						"  X1 STRING, " +
						"  X2 STRING, " +
						"  X3 INTEGER, " +
						"   PRIMARY KEY (X1) NOT ENFORCED)" +
						" WITH (" +
						" 'write-mode' = 'upsert'," +
						"'changelog-producer' = 'input') ");
Copy code
tableEnv.executeSql(
       "INSERT INTO name " +
             "SELECT * " +
             "FROM tmp_debezium"
);
I'm using a
" 'sequence.field' = 'date' "
and
tableEnv.getConfig().getConfiguration().setString("table.exec.sink.upsert-materialize", "none");
too
In s3 bucket I see the files:
the table with two folders (bucket-0) and schema
in the bucket-0 I have de orc files
but I can not achieve to query this data
It's true that in the job that is writting to s3 / paimon, I see an error in a checkpoint, and I don't know if this can be related with the case that I can not query the data
d
Whats kind of error are seeing in the checkpoint?
i
Error while processing AcknowledgeCheckpoint message
org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize the pending checkpoint 52. Failure reason: Failure to finalize checkpoint.
Caused by: com.facebook.presto.hive.s3.PrestoS3FileSystem$UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: YWNHC3RYT25S6TGM; S3 Extended Request ID: 15aN8R64TiVuNPZ50Cxhwkb5DtnYNnr6rIExWu1Cu96HXHj/d4IyljGuiktBy/ldVIdZIR30cOaIlmJ8fTkwYQWkUG2kIjsO93t45Cwpj0M=; Proxy: null), S3 Extended Request ID: 15aN8R64TiVuNPZ50Cxhwkb5DtnYNnr6rIExWu1Cu96HXHj/d4IyljGuiktBy/ldVIdZIR30cOaIlmJ8fTkwYQWkUG2kIjsO93t45Cwpj0M= (Path:.... )
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: xxx; S3 Extended Request ID: xxxx; Proxy: null)
Could not properly discard completed checkpoint 52.
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: sss; S3 Extended Request ID: lnCAF5V4JNWoi7jaY8DIIZLGFXxRS/0c6/i3I6XGxuy07yU2+L8IlljOFAF/6BLgdETg4lgmJPoNbRfYgzCVsL5e5A8hcHD9RpTO+0wDTMg=; Proxy: null)
d
Looks like issue …
i
all these errors... I know a lot of errors.. I don't know what is saying access denied, because my s3 is writting in the buckets correctlys
d
Do you have minio setup locally?
i
minio?, I'm running in a Flink locally, yes
d
Yes, you can try it using Minio first I think. There is also LocalStack
i
sorry for my ignorance, but what is Minio?
d
I think its a like local equivalent of S3 file system
i
ok, I see
do you think that is a problem t execute locally ?
d
It should run fine locally
i
ok ok
d
“S3 Complaint Object Stores # The S3 Filesystem also support using S3 compliant object stores such as MinIO, Tencent’s COS and IBM’s Cloud Object Storage. Just configure your endpoint to the provider of the object store service.”
So yes it supports MinIO
i
ok, let's try with MinIO
d
Yep you might also like: https://www.localstack.cloud/ later
i
because yo think that the problem is reelated with the checkpoint correct?, Could you confirm that I don't need read as bacth from the apache paimon with CDC upsert, correct?
d
Apache Paimon is kind of a new system so I am not certain. But I was looking at the Sync for it and yes it seems to accept upserts and that’s basically similar to what you are doing.
I think its probably very flexible actually given the goal of being a lakehouse and all that.
i
ok, thanks so much, I'm going to try to fix the problem with the checkpoint
d
so there is way to do it. Its actually streaming lakehouse so yes its for streaming
i
great , thanks Draco.
d
sure, I really like to learn more about lakehouses.
gratitude thank you 1
i
because sorry again, because I have a doubt, is mandatory to enable the checkpoint in Flink?
d
Well for a production system I think you would need checkpointing. I am not sure if however you are just processing but I think since you need to be aware or CDC needs to be aware of Kafka offsets I would say yes.
Have you tried also the Paimon Sync? It might have more info on this question.
i
ok, thanks so much Draco.