Hi all, I have came across a problem with a Postgr...
# troubleshooting
t
Hi all, I have came across a problem with a Postgres JDBC connector. I am not sure, but I don’t think this is expected behavior. I have the following context: • Pipeline source Postgres CDC • Pipeline sink Postgres JDBC If I make a change in the source table, and print the changelog stream:
Copy code
-D[1,83819211,545900928]
+I[-83819211,83819211,545900928]
This is expected, it should remove the record having primary key 1, and insert the record having primary key -83819211. If I check the database after this change, I see the following:
Copy code
-----------+----------------+--------------------
 -83819211 |       83819211 |          545900928
         1 |       83819211 |          545900928
It suggests that the delete statement has not been executed. Anyone that can help?
I can confirm that no DELETE statement is executed by Flink, I have examined the Postgres logs.
m
Are you using DataStream or Table API?
t
Table API
m
Have you configured the primary key in the DDL?
t
Yes I have, this is what I get when printing the result from the schema builder:
Copy code
(
  `family_id` INT NOT NULL,
  `representative` INT NOT NULL,
  `invention_title_id` BIGINT,
  CONSTRAINT `PK_family_id` PRIMARY KEY (`family_id`) NOT ENFORCED
)
COMMENT ''
WITH (
  'password' = 'root',
  'connector' = 'jdbc',
  'driver' = 'org.postgresql.Driver',
  'table-name' = 'family',
  'url' = 'jdbc:<postgresql://localhost:5432/database>',
  'username' = 'root'
)
I do get these INFO messages:
Copy code
org.apache.flink.table.runtime.operators.sink.SinkUpsertMaterializer [] - The state is cleared because of state ttl. This will result in incorrect result. You can increase the state ttl to avoid this.
Not sure if it may have anything to do with it?
I did some more research, and found: • By converting the table to a changelog datastream which prints to stdout, it actually prints the I/D messages as described before, while • Inserting the table into a print sink, only shows INSERT messages.
Alright, some more observations: • In the example (left out for simplicity), I actually use a column
last_modified
, • When inserting I use
current_timestamp
function to insert timestamps upon insert. • It turns out that adding this functionality causes:
org.apache.flink.table.runtime.operators.sink.SinkUpsertMaterializer [] - The state is cleared because of state ttl. This will result in incorrect result. You can increase the state ttl to avoid this.
• After removing this column, no messages like the above are shown anymore. Even better, the print sink table starts showing I/D messages again. Now I will test again with the Postgres JDBC sink to see if that also works.
I have tested it with the Postgres JDBC sink. It does work. 🎉 Interestingly
current_timestamp
caused this bug. Is there any explanation for that?