I am having a hell of a time getting the sql clien...
# troubleshooting
m
I am having a hell of a time getting the sql client to load my args correctly. If I run the
sql-client.sh
one way my init file doesn't load but I am able to manually run commands to register a python UDF function and that all works... if I run it with the init file arg first then the init file loads but then my udf function doesn't get registered correctly.. seeming like my python path is off or the pyfiles arg is borked... this is how I am running things: Init SQL File doesn't load
Copy code
#!/nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26/bin/bash
set -o errexit
set -o nounset
set -o pipefail

export PATH="/nix/store/8p75w2cs6lgzakvz6q9xns7j876d0i8m-gnugrep-3.11/bin:/nix/store/h2gq8hngnsnwphzpq7992cip77lwrphm-gnused-4.9/bin:/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2/bin:/nix/store/lhns6bwqlwfs4z6hd8jf08v4di08qqdy-glibc-2.39-52-bin/bin:/nix/store/p1jwaambbgk5wg963dpn7xq0v200c18v-su-exec-0.2/bin:/nix/store/8sgpjfbmlalr0xvybfj68540kz57rx6c-gosu-1.17/bin:/nix/store/r08ns1vm8vvvz996frwql0c52vwgqaw2-hostname-net-tools-2.10/bin:/nix/store/k0haaab80wycif0k8f5xm8ykdxpq21jy-jemalloc-5.3.0/bin:/nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26/bin:/nix/store/v8sjbyscx6r58xngbhf0rsdf5czfyf8q-findutils-4.9.0/bin:/nix/store/2wvb4326f069mz8zan43yx6nak6lsjqk-util-linux-2.39.4-bin/bin:/nix/store/xfm4mg874w5n39zbqx24yiw7hmka94n7-coreutils-9.5/bin:/nix/store/hsd3ppzdlkz31qdlnrc2dizylxp7wp9f-openjdk-11.0.23+9/bin:$PATH"

# Ensure FLINK_CONF_DIR is set
if [ -z "${FLINK_CONF_DIR:-}" ]; then
  export FLINK_CONF_DIR="/nix/store/hv2biq5cpmy4iri7lph12q85lfav91h6-flink-conf-drv/conf"
  echo "FLINK_CONF_DIR set to $FLINK_CONF_DIR"
else
  echo "FLINK_CONF_DIR already set to $FLINK_CONF_DIR"
fi

# Additional script logic
export PYTHONPATH="/nix/store/x74v1acazbqcr1yhfidyyx452xnyvih4-python3-3.11.9-env/lib/python3.11/site-packages:/nix/store/dkj9ygpkkz2iw9sa7jrnb5khs2q3k4w0-example-flink-job"
PYFILES="$(echo "$PYTHONPATH" | tr ':' ',')"
export PATH="/nix/store/x74v1acazbqcr1yhfidyyx452xnyvih4-python3-3.11.9-env/bin/:$PATH"
export PYFLINK_PYTHON="/nix/store/x74v1acazbqcr1yhfidyyx452xnyvih4-python3-3.11.9-env/bin/python"
export JAVA_HOME="/nix/store/hsd3ppzdlkz31qdlnrc2dizylxp7wp9f-openjdk-11.0.23+9"
export FLINK_HOME="/nix/store/sn2s860l0dxhf8n5dl4sz46falnilwjg-flink-1.19.1/opt/flink"

echo "PYFILES: $PYFILES"
echo "PYTHONPATH: $PYTHONPATH"

/nix/store/jxpi70rg23rlppdn3i1qhg91bvrg792y-flink-1.19.1/opt/flink/bin/sql-client.sh -j=/nix/store/dj0w4a1m2lvx7w55bfagibn67qxglhkc-flink-sql-connector-kafka-3.2.0-1.19.jar -pyclientexec=/nix/store/x74v1acazbqcr1yhfidyyx452xnyvih4-python3-3.11.9-env/bin/python --pyFiles="$PYFILES" "$@"
SQL Init File loads but Python is messed up
Copy code
#!/nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26/bin/bash
set -o errexit
set -o nounset
set -o pipefail

export PATH="/nix/store/8p75w2cs6lgzakvz6q9xns7j876d0i8m-gnugrep-3.11/bin:/nix/store/h2gq8hngnsnwphzpq7992cip77lwrphm-gnused-4.9/bin:/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2/bin:/nix/store/lhns6bwqlwfs4z6hd8jf08v4di08qqdy-glibc-2.39-52-bin/bin:/nix/store/p1jwaambbgk5wg963dpn7xq0v200c18v-su-exec-0.2/bin:/nix/store/8sgpjfbmlalr0xvybfj68540kz57rx6c-gosu-1.17/bin:/nix/store/r08ns1vm8vvvz996frwql0c52vwgqaw2-hostname-net-tools-2.10/bin:/nix/store/k0haaab80wycif0k8f5xm8ykdxpq21jy-jemalloc-5.3.0/bin:/nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26/bin:/nix/store/v8sjbyscx6r58xngbhf0rsdf5czfyf8q-findutils-4.9.0/bin:/nix/store/2wvb4326f069mz8zan43yx6nak6lsjqk-util-linux-2.39.4-bin/bin:/nix/store/xfm4mg874w5n39zbqx24yiw7hmka94n7-coreutils-9.5/bin:/nix/store/hsd3ppzdlkz31qdlnrc2dizylxp7wp9f-openjdk-11.0.23+9/bin:$PATH"

# Ensure FLINK_CONF_DIR is set
if [ -z "${FLINK_CONF_DIR:-}" ]; then
  export FLINK_CONF_DIR="/nix/store/hv2biq5cpmy4iri7lph12q85lfav91h6-flink-conf-drv/conf"
  echo "FLINK_CONF_DIR set to $FLINK_CONF_DIR"
else
  echo "FLINK_CONF_DIR already set to $FLINK_CONF_DIR"
fi

# Additional script logic
export PYTHONPATH="/nix/store/x74v1acazbqcr1yhfidyyx452xnyvih4-python3-3.11.9-env/lib/python3.11/site-packages:/nix/store/dkj9ygpkkz2iw9sa7jrnb5khs2q3k4w0-example-flink-job"
PYFILES="$(echo "$PYTHONPATH" | tr ':' ',')"
export PATH="/nix/store/x74v1acazbqcr1yhfidyyx452xnyvih4-python3-3.11.9-env/bin/:$PATH"
export PYFLINK_PYTHON="/nix/store/x74v1acazbqcr1yhfidyyx452xnyvih4-python3-3.11.9-env/bin/python"
export JAVA_HOME="/nix/store/hsd3ppzdlkz31qdlnrc2dizylxp7wp9f-openjdk-11.0.23+9"
export FLINK_HOME="/nix/store/sn2s860l0dxhf8n5dl4sz46falnilwjg-flink-1.19.1/opt/flink"

echo "PYFILES: $PYFILES"
echo "PYTHONPATH: $PYTHONPATH"

/nix/store/jxpi70rg23rlppdn3i1qhg91bvrg792y-flink-1.19.1/opt/flink/bin/sql-client.sh \
  -i=/config/packages/examples/example-flink-job/flink.sql \
  -j=/nix/store/dj0w4a1m2lvx7w55bfagibn67qxglhkc-flink-sql-connector-kafka-3.2.0-1.19.jar \
  -pyclientexec=/nix/store/x74v1acazbqcr1yhfidyyx452xnyvih4-python3-3.11.9-env/bin/python \
  -pyfs=/nix/store/dkj9ygpkkz2iw9sa7jrnb5khs2q3k4w0-example-flink-job
This is the error when Python isn't working
Copy code
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.api.python.shaded.py4j.Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last):
  File "/nix/store/jxpi70rg23rlppdn3i1qhg91bvrg792y-flink-1.19.1/opt/flink/opt/python/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 2466, in _call_proxy
    return_value = getattr(self.pool[obj_id], method)(*params)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jxpi70rg23rlppdn3i1qhg91bvrg792y-flink-1.19.1/opt/flink/opt/python/pyflink.zip/pyflink/java_gateway.py", line 179, in getPythonFunction
    udf_wrapper = getattr(importlib.import_module(moduleName), objectName)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/5w07wfs288qpmnvjywk24f3ak5k1np7r-python3-3.11.9/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named '\n    jobs'
This happens if I try to call the udf function
seems I have the python issues fixed but now i can not for the life of me get the damn init file to be read
this works
Copy code
/nix/store/jxpi70rg23rlppdn3i1qhg91bvrg792y-flink-1.19.1/opt/flink/bin/sql-client.sh --init=/home/mcamp/work-code/cyclops-work/ussf-hypertime/HYPR-93-96-Additional-Metrics/packages/flink-jobs/tle-metrics-flink-job/combined-init.sql -j=/nix/store/dj0w4a1m2lvx7w55bfagibn67qxglhkc-flink-sql-connector-kafka-3.2.0-1.19.jar -pyclientexec=/nix/store/0yb905dnl1n45l6y7r44gd60ma9hlsz2-beehivesupply-cyclops-docker.jfrog.io-cyclops-flink-tle-metrics-flink-job/bin/python --pyFiles=/nix/store/kbvlnk9lymfzhm9r0i8gmcwb4i1bh31c-tle-metrics-flink-job/
this doesnt
Copy code
/nix/store/jxpi70rg23rlppdn3i1qhg91bvrg792y-flink-1.19.1/opt/flink/bin/sql-client.sh -pyfs=/nix/store/kbvlnk9lymfzhm9r0i8gmcwb4i1bh31c-tle-metrics-flink-job -j=/nix/store/dj0w4a1m2lvx7w55bfagibn67qxglhkc-flink-sql-connector-kafka-3.2.0-1.19.jar -pyclientexec=/nix/store/0yb905dnl1n45l6y7r44gd60ma9hlsz2-beehivesupply-cyclops-docker.jfrog.io-cyclops-flink-tle-metrics-flink-job/bin/python --init=/home/mcamp/work-code/cyclops-work/ussf-hypertime/HYPR-93-96-Additional-Metrics/packages/flink-jobs/tle-metrics-flink-job/combined-init.sql
what the hell??!
it seems like
--init
has to be at the beginning?! Is this correct?