hmmm, battling to keep pyflink flow stable, or is that flink cluster and output to paimon on S3. https://github.com/georgelza/PyFlink_Embedder.git slowed the data push into postgress right down, 3 records a second. the embedidng is like 0.15 seconds, so faster than the inbound flow. i've upped the various memory values for flink in config file. to make more memory avail i stopped the 2nd taskmanager as flink cdc is single/thread consuming via the WAL anyhow. would appreciate if someone can help getting the environment stable and the tps up... configuration file for flink is <project root>/devlab/conf/config.yaml the pyflink job is in <project root>/devlab/pyflink the insert into select from statement that kicks the flink job off is in: the pyflink job is in <project root>/devlab/creFlinkFlows/scripts/4.1.creInsertsAhSingle.sql the above uses the generate_ah_embedding udf defined in devlab/pyflink/udfs the idea is once this single threaded works to maybe see if i can get 4.1.creInsertsAhMulti.sql also to work, with multiple workers. just further on this, at the moment i'm battling to embed and store accountholder profiles, at hmm 10/s, i want to be able to embed/store transactions in parallel at 100/s ()for lab can reduce to 5 spread out across workers to proof the configuration.... as it stands i'm calculating the vector in 0.1txn/sec. so should be able to do say 7/sec / worker realistically, I can def run 10 workers on my laptop, each with 1.5GB dedicated, I have 32GB to play work and 10 cores. which should put that 50/sec well in the do-able zone. i understand I need a supporting configuration at the flink server jobmanager/taskmanager level, but also thinking some per job specific tweeking is needed as not all jobs will/or even should inherit all settings from a global config.