Hi all I m trying to make a simple proof of concept ahead of Apache Flink #troubleshooting

Hi all, I'm trying to make a simple proof of conce...

Arran

01/27/2023, 3:53 PM

Hi all, I'm trying to make a simple proof of concept ahead of a larger migration of some existing basic data streaming jobs into Flink from another framework we're moving away from. The PoC is a streaming env with two KafkaSources, that sink to a JDBC connector to a PGSQL database, using Flink 1.16. I'm able to run the job without issue and continuously within my IDE (IntelliJ on windows), however when submitting it to a standalone flink deployment (running on a Linux EC2), every 29m50-something seconds it cancels and restarts, like clockwork. The regularity suggests a misconfiguration, but I've been through everything I can think of and am now a little lost about where to look. Any thoughts?

sap1ens

01/27/2023, 5:10 PM

Have you looked at the checkpoints? Do they look healthy, etc.?

Arran

01/30/2023, 8:19 AM

It looked to be checkpointing just fine, but I'm new to Flink so may have missed something. How would you define a healthy vs unhealthy checkpoint? I should have included that there's no exceptions that I can find in the logs.

Arran

01/30/2023, 8:22 AM

Copy code

2023-01-30 03:56:59,711 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 30 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1675051019710 for job e7a1b5ea667817b13388efa7b3095834.
2023-01-30 03:56:59,721 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 30 for job e7a1b5ea667817b13388efa7b3095834 (670 bytes, checkpointDuration=11 ms, finalizationTime=0 ms).
2023-01-30 03:56:59,722 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 30 as completed for source Source: humanTaskCompletion:Flink_Source.
2023-01-30 03:56:59,722 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 30 as completed for source Source: humanTaskCreation:Flink_Source.
2023-01-30 03:57:38,075 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job human-tasks-job (e7a1b5ea667817b13388efa7b3095834) switched from state RUNNING to CANCELLING.
2023-01-30 03:57:38,075 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: humanTaskCreation:Flink_Source -> Flat Map -> Sink: JDBC Sink: flink_humantasks (1/1) (10b538f960324bae13d3e0709536dfcd_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to CANCELING.
2023-01-30 03:57:38,078 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: humanTaskCompletion:Flink_Source -> Flat Map -> Sink: JDBC Sink: flink_humantasks (1/1) (10b538f960324bae13d3e0709536dfcd_6cdc5bb954874d922eaee11a8e7b5dd5_0_0) switched from RUNNING to CANCELING.
2023-01-30 03:57:38,128 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: humanTaskCompletion:Flink_Source -> Flat Map -> Sink: JDBC Sink: flink_humantasks (1/1) (10b538f960324bae13d3e0709536dfcd_6cdc5bb954874d922eaee11a8e7b5dd5_0_0) switched from CANCELING to CANCELED.
2023-01-30 03:57:38,136 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: humanTaskCreation:Flink_Source -> Flat Map -> Sink: JDBC Sink: flink_humantasks (1/1) (10b538f960324bae13d3e0709536dfcd_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from CANCELING to CANCELED.
2023-01-30 03:57:38,143 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job human-tasks-job (e7a1b5ea667817b13388efa7b3095834) switched from state CANCELLING to CANCELED.
2023-01-30 03:57:38,143 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Stopping checkpoint coordinator for job e7a1b5ea667817b13388efa7b3095834.
2023-01-30 03:57:38,147 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Clearing resource requirements of job e7a1b5ea667817b13388efa7b3095834
2023-01-30 03:57:38,165 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Job e7a1b5ea667817b13388efa7b3095834 reached terminal state CANCELED.
2023-01-30 03:57:38,242 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Job e7a1b5ea667817b13388efa7b3095834 has been registered for cleanup in the JobResultStore after reaching a terminal state.
2023-01-30 03:57:38,247 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Stopping the JobMaster for job 'human-tasks-job' (e7a1b5ea667817b13388efa7b3095834).
2023-01-30 03:57:38,258 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Closing SourceCoordinator for source Source: humanTaskCreation:Flink_Source.
2023-01-30 03:57:38,262 INFO  org.apache.kafka.common.utils.AppInfoParser                  [] - App info kafka.admin.client for humanTaskCreation:Flink_Source:712ddf02-d413-4df3-9f79-d0131ea55171-enumerator-admin-client unregistered
2023-01-30 03:57:38,274 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Closing SourceCoordinator for source Source: humanTaskCompletion:Flink_Source.
2023-01-30 03:57:38,275 INFO  org.apache.kafka.common.metrics.Metrics                      [] - Metrics scheduler closed
2023-01-30 03:57:38,275 INFO  org.apache.kafka.common.metrics.Metrics                      [] - Closing reporter org.apache.kafka.common.metrics.JmxReporter
2023-01-30 03:57:38,275 INFO  org.apache.kafka.common.metrics.Metrics                      [] - Metrics reporters closed
2023-01-30 03:57:38,275 INFO  org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Shutting down
2023-01-30 03:57:38,282 INFO  org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Checkpoint with ID 30 at 'file:/tmp/flink/checkpoints/e7a1b5ea667817b13388efa7b3095834/chk-30' not discarded.
2023-01-30 03:57:38,283 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Disconnect TaskExecutor localhost:40614-0385cc because: Stopping JobMaster for job 'human-tasks-job' (e7a1b5ea667817b13388efa7b3095834).
2023-01-30 03:57:38,284 INFO  org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] - Releasing slot [4700bc4f7b340f4cf825d0ae7c9d6031].
2023-01-30 03:57:38,285 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Close ResourceManager connection 891b310d11292bd53187431d3242c7c9: Stopping JobMaster for job 'human-tasks-job' (e7a1b5ea667817b13388efa7b3095834).
2023-01-30 03:57:38,288 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Disconnect job manager 00000000000000000000000000000000@akka.tcp://flink@localhost:6123/user/rpc/jobmanager_2 for job e7a1b5ea667817b13388efa7b3095834 from the resource manager.
2023-01-30 03:57:38,291 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source coordinator for source Source: humanTaskCreation:Flink_Source closed.
2023-01-30 03:57:38,296 INFO  org.apache.kafka.common.utils.AppInfoParser                  [] - App info kafka.admin.client for humanTaskCompletionEvent:Flink_Source:9119c2e8-5550-4a1b-8248-555fd7a00086-enumerator-admin-client unregistered
2023-01-30 03:57:38,307 INFO  org.apache.kafka.common.metrics.Metrics                      [] - Metrics scheduler closed
2023-01-30 03:57:38,307 INFO  org.apache.kafka.common.metrics.Metrics                      [] - Closing reporter org.apache.kafka.common.metrics.JmxReporter
2023-01-30 03:57:38,307 INFO  org.apache.kafka.common.metrics.Metrics                      [] - Metrics reporters closed
2023-01-30 03:57:38,361 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source coordinator for source Source: humanTaskCompletion:Flink_Source closed.

Open in Slack

Previous Next