hi all... trying to write from Apache Flink 1.18.1 to Paimon on Hadoop HDFS (3.3.5). the "reference/metadata objects/schema's created" and bucket0's created. but never any data... and if I look at docker compose logs, i see the following

flink-taskmanager-2      | 2024-08-05 18:50:07,618 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Source: t_f_avro_salescompleted_x[16] (1/1)#251 (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) switched from CREATED to DEPLOYING.
flink-taskmanager-2      | 2024-08-05 18:50:07,618 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Loading JAR files for task Source: t_f_avro_salescompleted_x[16] (1/1)#251 (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) [DEPLOYING].
flink-taskmanager-2      | 2024-08-05 18:50:07,620 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Source: t_f_avro_salescompleted_x[16] (1/1)#251 (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) switched from DEPLOYING to INITIALIZING.
devlab-flink-jobmanager  | 2024-08-05 18:50:07,621 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: t_f_avro_salescompleted_x[16] (1/1) (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) switched from DEPLOYING to INITIALIZING.
devlab-flink-jobmanager  | 2024-08-05 18:50:07,633 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: t_f_avro_salescompleted_x[16] registering reader for parallel task 0 (#251) @ 172.19.0.12
flink-taskmanager-2      | 2024-08-05 18:50:07,648 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Source: t_f_avro_salescompleted_x[16] (1/1)#251 (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) switched from INITIALIZING to RUNNING.
devlab-flink-jobmanager  | 2024-08-05 18:50:07,648 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: t_f_avro_salescompleted_x[16] (1/1) (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) switched from INITIALIZING to RUNNING.
devlab-flink-jobmanager  | 2024-08-05 18:50:07,997 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: t_f_avro_salescompleted_x[16] (1/1) (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) switched from RUNNING to CANCELING.
devlab-flink-jobmanager  | 2024-08-05 18:50:07,997 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 (#251) of source Source: t_f_avro_salescompleted_x[16].
flink-taskmanager-2      | 2024-08-05 18:50:07,997 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Attempting to cancel task Source: t_f_avro_salescompleted_x[16] (1/1)#251 (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251).
flink-taskmanager-2      | 2024-08-05 18:50:07,997 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Source: t_f_avro_salescompleted_x[16] (1/1)#251 (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) switched from RUNNING to CANCELING.
flink-taskmanager-2      | 2024-08-05 18:50:07,997 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Triggering cancellation of task code Source: t_f_avro_salescompleted_x[16] (1/1)#251 (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251).
flink-taskmanager-2      | 2024-08-05 18:50:07,998 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Source: t_f_avro_salescompleted_x[16] (1/1)#251 (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) switched from CANCELING to CANCELED.
flink-taskmanager-2      | 2024-08-05 18:50:07,998 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Freeing task resources for Source: t_f_avro_salescompleted_x[16] (1/1)#251 (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251).
flink-taskmanager-2      | 2024-08-05 18:50:08,001 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Un-registering task and sending final execution state CANCELED to JobManager for task Source: t_f_avro_salescompleted_x[16] (1/1)#251 e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251.
devlab-flink-jobmanager  | 2024-08-05 18:50:08,002 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: t_f_avro_salescompleted_x[16] (1/1) (e2c7bb3077c1d5c22e0bccba65bfc32b_bc764cd8ddf7a0cff126f51c16239658_0_251) switched from CANCELING to CANCELED.

Ok, so logs indicate that the task responsible for reading from the source t_f_avro_salescompleted_x and presumably writing to Paimon is being cancelled right away after it starts running.

ok, will google. these are the same Flink job and task manager that i was using to write to iceberg on S3/MinIO. I can't imagine it being permission as the various table definition objects are created..

How about availability of t_f_avro_salescompleted_x - are you sure data can be read ok from it?

will go look... the difference between this one and the previous is the output stream/leg, the input, aka topics and then flink virtual tables are all the same.

I will take a quick look at Flink 1.18.1 to see if there are any known issues with HDFS 3.3.5 but I think that’s supposed to be compatible.

that seems to be the sweet combination/versions.

Probably worth a quick check if there are any known issues. Resource constraints are a bit unlikely given its more or less the same setup as before

the dif of this build is pushing out to a hadoop dfs cluster, so thats extra resources, previous version was to a single node/container MinIO S3 service, but can't see it being resources, this MBP is not exactly slow.

source is Kafka right?

yes, Kafka topic => Flink virtual table => paimon on hdfs

So your quite sure that the sales_completed_source is available and producing data right?

interesting, originally i thought somethine wrong with CTAS, so tried the CTAT where 1=2, followed by the insert into <>

Btw, do you have Paimon logging setup as well?

paimon is just a open table format, stored on hdfs, not much to enable.

well … are you using Paimon connector?

no... using jar file in flink lib, and then the catalog definition as per above sql file and then the table creates as per above.

I think you might need the Flink connector for Hadoop ..

<dependencies>
    <dependency>
        <groupId>org.apache.paimon</groupId>
        <artifactId>paimon-flink</artifactId>
        <version>${paimon.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-hadoop-fs</artifactId>
        <version>${flink.version}</version>
    </dependency>
</dependencies>

will need to dig, from the paimon quick start for flink it did not imply anything like that.

ok, are you using Paimon also as the Catalog store for Flink?

yes..

I see. According to https://www.alibabacloud.com/help/en/flink/developer-reference/apache-paimon-connector it would appear that you only need the connector in the case where you are using another catalog.

this could be the cause, but it's a far fetch one.

Yeah well I breaking it up a bit into separate jobs will show what’s happening

figured, get it first to work like this before i try and move catalog into hms.

Yep I am curious to find out how well it supports catalogs.

i got a 90% work done to create a 2 node hive environment, a hive server and a meta server backed by postgres. once i got the simple hdfs cat working i will try the stand alone hms, and if that works then try my hive envionment.

Looking forward to seeing the results on that.

heheheheh, 😉 my small little blog has turned into a multipart monster,