david radley
06/06/2024, 10:08 AM"$FLINK_BIN_DIR"/jobmanager.sh start-foreground so I can see what is happening. I have amended flink-console.sh so the last lines are DEBUG_OPTS=" -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=localhost:9000 "
`exec "$JAVA_RUN" $DEBUG_OPTS $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH:$SQL_GATEWAY_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" ${CLASS_TO_RUN} "${ARGS[@]}"` . I start the sql-client, create a table and then do a select which hangs, the select normally works without the script alterations. I see in the output of the start-cluster.sh window 2024-06-06 09:58:48,084 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Matching resource requirements against available resources.
Missing resources:
Job 4aebd3a4723cb69c643c85ab85688763
ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}
Current resources:
(none)
2024-06-06 09:58:48,163 INFO org.apache.flink.connector.kafka.source.enumerator.KafkaSourceEnumerator [] - Discovered new partitions: [vivekavro-0]
`2024-06-06 095921,722 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job SELECT source_1.order_id, source_1.order_time, source_1.`buyer``
`FROM default_catalog.default_database.source_1 AS source_1 (4aebd3a4723cb69c643c85ab85688763) switched from state RUNNING to CANCELLING.` It looks like the addition of the debug options to the job manager java command somehow prevents the a task manager from being initialised! I have tried with in the background and cannot get it to work either. I would be grateful for any pointers.david radley
06/06/2024, 12:20 PMyux
06/07/2024, 1:52 AMstart-foreground to jobmanager.sh, it will be blocked by running flink-console.sh which never returns, so the following TMWorkers start code never got executed. If you'd like to watch the output of jobmanager.sh start and keeps executing the following shell script, just append an & to the startup line.
Here's my patch:
#!/usr/bin/env bash
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
bin=`dirname "$0"`
bin=`cd "$bin"; pwd`
. "$bin"/config.sh
# Start the JobManager instance(s)
shopt -s nocasematch
if [[ $HIGH_AVAILABILITY == "zookeeper" ]]; then
# HA Mode
readMasters
echo "Starting HA cluster with ${#MASTERS[@]} masters."
for ((i=0;i<${#MASTERS[@]};++i)); do
master=${MASTERS[i]}
webuiport=${WEBUIPORTS[i]}
if [ ${MASTERS_ALL_LOCALHOST} = true ] ; then
"${FLINK_BIN_DIR}"/jobmanager.sh start-foreground "${master}" "${webuiport}" & # <= Here!
else
ssh -n $FLINK_SSH_OPTS $master -- "nohup /bin/bash -l \"${FLINK_BIN_DIR}/jobmanager.sh\" start ${master} ${webuiport} &"
fi
done
else
echo "Starting cluster."
# Start single JobManager on this machine
"$FLINK_BIN_DIR"/jobmanager.sh start-foreground & # <= Here!
fi
shopt -u nocasematch
# Start TaskManager instance(s)
TMWorkers start