Hi all, I am trying deploy pyfink application usin...
# troubleshooting
r
Hi all, I am trying deploy pyfink application using Kubernetes operator. I am constantly hitting a error after which my Task Manager Pod goes unresponsive and dies after certain interval.
Copy code
File "/opt/flink/usrlib/data_processing_v9/stream_processing/process/prepare_data.py", line 71, in process
    filtered_elements = self.filter_elements(raw_data)
  File "/opt/flink/usrlib/data_processing_v9/stream_processing/process/prepare_data.py", line 104, in filter_elements
    raw_mapping_dict = set(self.raw_io_mapping_dict.keys())
  File "/opt/flink/usrlib/data_processing_v9/stream_processing/utils/state_utils/map_state_utils.py", line 66, in keys
    return self.map_state.keys()
  File "/usr/local/lib/python3.10/dist-packages/pyflink/fn_execution/state_impl.py", line 934, in keys
    return self.get_internal_state().keys()
  File "/usr/local/lib/python3.10/dist-packages/pyflink/fn_execution/state_impl.py", line 840, in keys
    self.remote_data_iterator(IterateType.KEYS))
  File "/usr/local/lib/python3.10/dist-packages/pyflink/fn_execution/state_impl.py", line 879, in remote_data_iterator
    return self._map_state_handler.lazy_iterator(
  File "/usr/local/lib/python3.10/dist-packages/pyflink/fn_execution/state_impl.py", line 437, in lazy_iterator
    current_batch, iterator_token = self._iterate_raw(
  File "/usr/local/lib/python3.10/dist-packages/pyflink/fn_execution/state_impl.py", line 621, in _iterate_raw
    data, response_token = self._underlying.get_raw(state_key, continuation_token)
  File "/usr/local/lib/python3.10/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 1064, in get_raw
    response = self._blocking_request(
  File "/usr/local/lib/python3.10/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 1104, in _blocking_request
    raise self._exception
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 1039, in pull_responses
    for response in responses:
  File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 543, in __next__
    return self._next()
  File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 969, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "recvmsg:Connection reset by peer"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"recvmsg:Connection reset by peer", grpc_status:14}"
>
This is the error that I am facing. I can see some changes in metrics around the same time but I am not able to find the root cause behind this issue. Can someone guide me with the tuning of configuration parameters. I am using RocksDB backend.