Amir Hossein Sharifzadeh
02/12/2023, 9:50 PMt_env.sql_query(query).execute().print()
but can’t iterate over results and it doesn’t print anything:
table_result = t_env.sql_query(query).execute().collect()
with table_result as results:
for result in results:
print(result)
This is how I start the code:
env = StreamExecutionEnvironment.get_execution_environment()
t_env = StreamTableEnvironment.create(stream_execution_environment=env)
t_env.get_config().get_configuration().set_boolean("python.fn-execution.memory.managed", True)
t1 = f"""
CREATE TABLE raw_table(
raw_id INT,
raw_data STRING
) WITH (
'connector' = 'kafka',
'topic' = 'empad_raw',
'properties.bootstrap.servers' = 'kafka:9092',
'properties.group.id' = 'MY_GRP',
'scan.startup.mode' = 'latest-offset',
'format' = 'json'
)
"""
t2 = f"""
CREATE TABLE bkgd_table(
raw_id INT,
raw_data STRING
) WITH (
'connector' = 'kafka',
'topic' = 'empad_bkgd',
'properties.bootstrap.servers' = 'kafka:9092',
'properties.group.id' = 'MY_GRP',
'scan.startup.mode' = 'latest-offset',
'format' = 'json'
)
"""
t_env.execute_sql(t1)
t_env.execute_sql(t2)
query = "SELECT raw_table.raw_id, raw_table.raw_data, bkgd_table.raw_data " \
"FROM raw_table JOIN bkgd_table ON raw_table.raw_id = bkgd_table.raw_id"
t_env.sql_query(query).execute().print()
I will need to get the results from my query and pass the results in another function to process them. Thanks for any suggestions.Amir Hossein Sharifzadeh
02/13/2023, 3:22 AMdef collect(self) -> 'CloseableIterator':
"""
Get the result contents as a closeable row iterator.
Note:
For SELECT operation, the job will not be finished unless all result data has been
collected. So we should actively close the job to avoid resource leak through
CloseableIterator#close method. Calling CloseableIterator#close method will cancel the job
and release related resources.
For DML operation, Flink does not support getting the real affected row count now. So the
affected row count is always -1 (unknown) for every sink, and them will be returned until
the job is finished.
Calling CloseableIterator#close method will cancel the job.
For other operations, no flink job will be submitted (get_job_client() is always empty), and
the result is bounded. Do noting when calling CloseableIterator#close method.
Recommended code to call CloseableIterator#close method looks like:
>>> table_result = t_env.execute("select ...")
>>> with table_result.collect() as results:
>>> for result in results:
>>> ...
In order to fetch result to local, you can call either collect() and print(). But, they can
not be called both on the same TableResult instance.
:return: A CloseableIterator.
.. versionadded:: 1.12.0
"""Dian Fu
02/13/2023, 3:24 AMflink run , flink run -d , python xxx.py ?Amir Hossein Sharifzadeh
02/13/2023, 3:26 AMAmir Hossein Sharifzadeh
02/13/2023, 3:26 AMAmir Hossein Sharifzadeh
02/13/2023, 3:29 AMAmir Hossein Sharifzadeh
02/13/2023, 3:32 AMversion: "2.2"
services:
jobmanager:
build: .
image: varimatflink:0.1
hostname: jobmanager
container_name: jobmanager
ports:
- "8081:8081"
command: jobmanager
volumes:
- ./data:/data
- ./artifacts:/opt/flink/usrlib
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
parallelism.default: 2
taskmanager:
build: .
image: varimatflink:0.1
hostname: taskmanager
container_name: taskmanager
depends_on:
- jobmanager
command: taskmanager
volumes:
- ./data:/data
- ./artifacts:/opt/flink/usrlib
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 4
parallelism.default: 2
zookeeper:
image: confluentinc/cp-zookeeper:6.1.1
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
restart: always
broker:
image: confluentinc/cp-kafka:6.1.1
hostname: kafka
container_name: kafka
depends_on:
- zookeeper
ports:
- "29092:29092"
- "9092:9092"
- "9101:9101"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: <PLAINTEXT://kafka:9092>,PLAINTEXT_<HOST://localhost:29092>
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0Amir Hossein Sharifzadeh
02/13/2023, 3:43 AMDian Fu
02/13/2023, 3:47 AM-d in ./bin/flink run -py /data/process_data.py -d , it will disconnect the client with the Flink cluster once the job is submitted and so there is no way to collect the results back to client. You can try to remove -d and try again~Amir Hossein Sharifzadeh
02/13/2023, 3:48 AMAmir Hossein Sharifzadeh
02/13/2023, 3:50 AMAmir Hossein Sharifzadeh
02/13/2023, 4:07 AMDian Fu
02/13/2023, 6:35 AMprint method will buffer the output in non-interactive mode. You can change print(result) to print(result, flush=True) to disable the buffer.
See https://stackoverflow.com/questions/107705/disable-output-buffering for more details about the output buffer behavior of print .
PS: This issue should has nothing to do with the -d option.Dian Fu
02/13/2023, 8:13 AM