https://flink.apache.org/ logo
Join Slack
Powered by
Hey! A logging issue: I’m running PyFlink locally ...
# troubleshooting
f
Hey! A logging issue: I’m running PyFlink locally simply by building a graph and calling 
execute
, but for debugging I’d like to change logging levels. Currently I only see warning and up, and I haven’t found a way to show more. In a proper deployment on K8s all logs are visible thanks to configuring 
logConfiguration:log4j-console.properties
. Is there a way to change the level with some configuration either in code or in a config file? The flink deployment is pointing to the same exact entry point in Python, so surely there must be, but merely having a 
log4j-console.properties
file didn’t do the job. Where am I going wrong?
c
f
Thank you for the link 🙏 I take it that I should try to create one of these files in the flink source directory then. For PyFlink, how can I find this directory?
c
I've never used PyFlink, but if your using k8s I'd mount this as a volume from a configmap.
f
I am, but the deployment logs are fine. It’s only locally that I cannot override the default config 🤔
c
well, in java I would put that in the log4j.properties in the resources folder. In python im not sure
f
Thank you, I’ll try to look for the folder and try that
👍 1
I found the directory, which was 
conf
under the Flink install directory (not a huge surprise), but all of the logging template files were already set to level INFO. While debugging, I noticed that all logging coming from child processes - flink operators - was suppressed, warnings included. Here’s a minimal example:
Copy code
from logging import DEBUG, getLogger

from pyflink.common import Types
from pyflink.datastream import StreamExecutionEnvironment

logger = getLogger(__name__)
logger.setLevel(DEBUG)

print("PRINT is visible")
<http://logger.info|logger.info>("INFO should be visible but is not")
logger.warning("WARNING is visible")

def test_operator(n: int) -> int:
    print("PRINT is visible")
    <http://logger.info|logger.info>("INFO should be visible but is not")
    logger.warning("WARNING is not visible either")
    return n

env = StreamExecutionEnvironment.get_execution_environment()
source = env.from_collection(range(5), type_info=<http://Types.INT|Types.INT>())
source.map(test_operator).print()
env.execute()
Open in Slack
PreviousNext
Powered by