https://flink.apache.org/ logo
Join Slack
Powered by
Please can you help me troubleshoot this error mes...
# troubleshooting
a
Please can you help me troubleshoot this error message in our taskmanager logs?
Copy code
2023-07-13 22:07:27 2023-07-13 12:07:27,393 INFO  org.apache.flink.runtime.io.network.netty.NettyClient        [] - Transport type 'auto': using EPOLL.
2023-07-13 22:07:27 2023-07-13 12:07:27,394 INFO  org.apache.flink.runtime.io.network.netty.NettyClient        [] - Successful initialization (took 34 ms).
2023-07-13 22:07:27 2023-07-13 12:07:27,397 INFO  org.apache.flink.runtime.io.network.netty.NettyServer        [] - Transport type 'auto': using EPOLL.
2023-07-13 22:07:27 2023-07-13 12:07:27,419 INFO  org.apache.flink.runtime.io.network.netty.NettyServer        [] - Successful initialization (took 23 ms). Listening on SocketAddress /0.0.0.0:42927.
2023-07-13 22:07:27 2023-07-13 12:07:27,420 INFO  org.apache.flink.runtime.taskexecutor.KvStateService         [] - Starting the kvState service and its components.
2023-07-13 22:07:27 2023-07-13 12:07:27,451 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at <akka://flink/user/rpc/taskmanager_0> .
2023-07-13 22:07:27 2023-07-13 12:07:27,461 INFO  org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Start job leader service.
2023-07-13 22:07:27 2023-07-13 12:07:27,464 INFO  org.apache.flink.runtime.filecache.FileCache                 [] - User file cache uses directory /tmp/flink-dist-cache-6ae1b4c5-61be-4d22-a17e-7044a879417f
2023-07-13 22:07:27 2023-07-13 12:07:27,465 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Connecting to ResourceManager <akka.tcp://flink@localhost:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)>.
2023-07-13 22:07:27 2023-07-13 12:07:27,510 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [<akka.tcp://flink@localhost:6123>] has failed, address is now gated for [50] ms. Reason: [Association failed with [<akka.tcp://flink@localhost:6123>]] Caused by: [java.net.ConnectException: Connection refused: localhost/127.0.0.1:6123]
2023-07-13 22:07:27 2023-07-13 12:07:27,511 WARN  akka.remote.transport.netty.NettyTransport                   [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: localhost/127.0.0.1:6123
2023-07-13 22:07:27 2023-07-13 12:07:27,516 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Could not resolve ResourceManager address <akka.tcp://flink@localhost:6123/user/rpc/resourcemanager_*>, retrying in 10000 ms: Could not connect to rpc endpoint under address <akka.tcp://flink@localhost:6123/user/rpc/resourcemanager_*>.
2023-07-13 22:07:37 2023-07-13 12:07:37,541 WARN  akka.remote.transport.netty.NettyTransport                   [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: localhost/127.0.0.1:6123
2023-07-13 22:07:37 2023-07-13 12:07:37,542 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [<akka.tcp://flink@localhost:6123>] has failed, address is now gated for [50] ms. Reason: [Association failed with [<akka.tcp://flink@localhost:6123>]] Caused by: [java.net.ConnectException: Connection refused: localhost/127.0.0.1:6123]
2023-07-13 22:07:37 2023-07-13 12:07:37,543 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Could not resolve ResourceManager address <akka.tcp://flink@localhost:6123/user/rpc/resourcemanager_*>, retrying in 10000 ms: Could not connect to rpc endpoint under address <akka.tcp://flink@localhost:6123/user/rpc/resourcemanager_*>.
The jobmanger seems to be start fine and doing what we expect, listening to a Kafka Topic:
Copy code
...
2023-07-13 22:17:40 2023-07-13 12:17:40,409 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@6d61af7a for Kafka Streaming Job for topic: dummy_topic (cddea62acc79eb3bfe533bb4e14642f2).
2023-07-13 22:17:40 2023-07-13 12:17:40,416 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Starting execution of job 'Kafka Streaming Job for topic: dummy_topic' (cddea62acc79eb3bfe533bb4e14642f2) under job master id 00000000000000000000000000000000.
2023-07-13 22:17:40 2023-07-13 12:17:40,418 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.PipelinedRegionSchedulingStrategy]
2023-07-13 22:17:40 2023-07-13 12:17:40,418 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job Kafka Streaming Job for topic: dummy_topic (cddea62acc79eb3bfe533bb4e14642f2) switched from state CREATED to RUNNING.
2023-07-13 22:17:40 2023-07-13 12:17:40,420 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: Kafka Source -> Sink: Print to Std. Out (1/1) (8a7317ac5692db1fd2fcedd39324f25a_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from CREATED to SCHEDULED.
2023-07-13 22:17:40 2023-07-13 12:17:40,428 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Connecting to ResourceManager <akka.tcp://flink@localhost:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)>
2023-07-13 22:17:40 2023-07-13 12:17:40,431 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Resolved ResourceManager address, beginning registration
2023-07-13 22:17:40 2023-07-13 12:17:40,432 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager 00000000000000000000000000000000@akka.tcp://flink@localhost:6123/user/rpc/jobmanager_2 for job cddea62acc79eb3bfe533bb4e14642f2.
2023-07-13 22:17:40 2023-07-13 12:17:40,434 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager 00000000000000000000000000000000@akka.tcp://flink@localhost:6123/user/rpc/jobmanager_2 for job cddea62acc79eb3bfe533bb4e14642f2.
2023-07-13 22:17:40 2023-07-13 12:17:40,438 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000.
2023-07-13 22:17:40 2023-07-13 12:17:40,439 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job cddea62acc79eb3bfe533bb4e14642f2: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2023-07-13 22:17:40 2023-07-13 12:17:40,474 INFO  org.apache.flink.StreamingJob                                [] - Flink streaming job for Kafka topic 'dummy_topic' started
Kakfa is running fine as I can see messages coming into that topic fine. Our docker-compose for Flink is as follows:
Copy code
version: "3.3"
services:
  jobmanager:
    restart: always
    image: <http://deeprecognition.azurecr.io/runtime-flink_jobmanager1:${BUILD_NUMBER}|deeprecognition.azurecr.io/runtime-flink_jobmanager1:${BUILD_NUMBER}>
    container_name: runtime-flink_jobmanager1
    expose:
      - "${JOB_MANAGER_PORT}"
    ports:
      - "${JOB_MANAGER_PORT}:${JOB_MANAGER_PORT}"
    command: jobmanager
    dns:
      - ${ENVIRONMENT_DNS}
    environment:
      - FQDN_SUFFIX=${FQDN_SUFFIX}
      - JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS}
      - JOB_MANAGER_RPC_PORT=${JOB_MANAGER_RPC_PORT}
      - KAFKA_HOST=${KAFKA_HOST}
      - KAFKA_PORT=${KAFKA_PORT}
      - KAFKA_TOPIC=${KAFKA_TOPIC}
      - VM_AGENT_NAME=${VM_AGENT_NAME}

  taskmanager:
    restart: always
    image: <http://deeprecognition.azurecr.io/runtime-flink_taskmanager1:${BUILD_NUMBER}|deeprecognition.azurecr.io/runtime-flink_taskmanager1:${BUILD_NUMBER}>
    container_name: runtime-flink_taskmanager1
    depends_on:
      - jobmanager
    expose:
      - "${TASK_MANAGER_PORT}"
    ports:
      - "${TASK_MANAGER_PORT}:${TASK_MANAGER_PORT}"
    command: taskmanager
    dns:
      - ${ENVIRONMENT_DNS}
    environment:
      - JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS}
      - JOB_MANAGER_RPC_PORT=${JOB_MANAGER_RPC_PORT}
      - RESTART_STRATEGY_FAILURE_RATE_DELAY_BETWEEN_ATTEMPTS=30000
      - RESTART_STRATEGY_FAILURE_RATE_FAILURE_RATE_INTERVAL=60000
      - RESTART_STRATEGY_FAILURE_RATE_MAX_FAILURES_PER_INTERVAL=3
      - RESTART_STRATEGY=failureRate
      - TASK_MANAGER_MEMORY_PROCESS_SIZE=4096m # TODO: adjust this value according to requirements per environment
      - TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
4 Views
Open in Slack
PreviousNext
Powered by