I am able to print results from my query but can’t...
# troubleshooting
a
I am able to print results from my query but can’t extract and iterate over them:
Copy code
t_env.sql_query(query).execute().print()
but can’t iterate over results and it doesn’t print anything:
Copy code
table_result = t_env.sql_query(query).execute().collect()

with table_result as results:
    for result in results:
        print(result)
This is how I start the code:
Copy code
env = StreamExecutionEnvironment.get_execution_environment()
t_env = StreamTableEnvironment.create(stream_execution_environment=env)
t_env.get_config().get_configuration().set_boolean("python.fn-execution.memory.managed", True)

t1 = f"""
        CREATE TABLE raw_table(
            raw_id INT,
            raw_data STRING
        ) WITH (
          'connector' = 'kafka',
          'topic' = 'empad_raw',
          'properties.bootstrap.servers' = 'kafka:9092',
          'properties.group.id' = 'MY_GRP',
          'scan.startup.mode' = 'latest-offset',
          'format' = 'json'
        )
        """

t2 = f"""
        CREATE TABLE bkgd_table(
            raw_id INT,
            raw_data STRING
        ) WITH (
          'connector' = 'kafka',
          'topic' = 'empad_bkgd',
          'properties.bootstrap.servers' = 'kafka:9092',
          'properties.group.id' = 'MY_GRP',
          'scan.startup.mode' = 'latest-offset',
          'format' = 'json'
        )
        """

t_env.execute_sql(t1)
t_env.execute_sql(t2)

query = "SELECT raw_table.raw_id, raw_table.raw_data, bkgd_table.raw_data " \
        "FROM raw_table JOIN bkgd_table ON raw_table.raw_id = bkgd_table.raw_id"

t_env.sql_query(query).execute().print()
I will need to get the results from my query and pass the results in another function to process them. Thanks for any suggestions.
This is collect() function from table_result.py and it suggests how to iterate over the collection but it really does’t work:
Copy code
def collect(self) -> 'CloseableIterator':
    """
    Get the result contents as a closeable row iterator.

    Note:

    For SELECT operation, the job will not be finished unless all result data has been
    collected. So we should actively close the job to avoid resource leak through
    CloseableIterator#close method. Calling CloseableIterator#close method will cancel the job
    and release related resources.

    For DML operation, Flink does not support getting the real affected row count now. So the
    affected row count is always -1 (unknown) for every sink, and them will be returned until
    the job is finished.
    Calling CloseableIterator#close method will cancel the job.

    For other operations, no flink job will be submitted (get_job_client() is always empty), and
    the result is bounded. Do noting when calling CloseableIterator#close method.

    Recommended code to call CloseableIterator#close method looks like:

    >>> table_result = t_env.execute("select ...")
    >>> with table_result.collect() as results:
    >>>    for result in results:
    >>>        ...

    In order to fetch result to local, you can call either collect() and print(). But, they can
    not be called both on the same TableResult instance.

    :return: A CloseableIterator.

    .. versionadded:: 1.12.0
    """
d
@Amir Hossein Sharifzadeh How you execute the job? E.g.
flink run
,
flink run -d
,
python xxx.py
?
a
@Dian Fu thanks for your reply. I run my python file through docker:
docker exec jobmanager ./bin/flink run -py /data/process_data.py -d
my docker contains kafka and flink containers:
and this is my docker-compose.yml file:
Copy code
version: "2.2"
services:
  jobmanager:
    build: .
    image: varimatflink:0.1
    hostname: jobmanager
    container_name: jobmanager
    ports:
      - "8081:8081"
    command: jobmanager
    volumes:
      - ./data:/data
      - ./artifacts:/opt/flink/usrlib
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        parallelism.default: 2        

  taskmanager:
    build: .
    image: varimatflink:0.1
    hostname: taskmanager
    container_name: taskmanager
    depends_on:
      - jobmanager
    command: taskmanager
    volumes:
      - ./data:/data
      - ./artifacts:/opt/flink/usrlib
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 4
        parallelism.default: 2

  zookeeper:
    image: confluentinc/cp-zookeeper:6.1.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    restart: always

  broker:
    image: confluentinc/cp-kafka:6.1.1
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: <PLAINTEXT://kafka:9092>,PLAINTEXT_<HOST://localhost:29092>
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
This is a screenshot from *print*() function and jobmanager command:
d
As you have used detach option
-d
in
./bin/flink run -py /data/process_data.py -d
, it will disconnect the client with the Flink cluster once the job is submitted and so there is no way to collect the results back to client. You can try to remove
-d
and try again~
a
Ahhhhhhh. OK. I will try that! Thank you!
So why does it work for print function? Also would you mind tell me why pyflink is so slow? Which one is faster Scala or Java? Thanks.
@Dian Fu It worked. Thanks a lot!
d
@Amir Hossein Sharifzadeh Hey, I figured out the root cause of this issue. The
print
method will buffer the output in non-interactive mode. You can change
print(result)
to
print(result, flush=True)
to disable the buffer. See https://stackoverflow.com/questions/107705/disable-output-buffering for more details about the output buffer behavior of
print
. PS: This issue should has nothing to do with the
-d
option.
👍 1