Adding an update here in case anyone is following this. I've been through the Java code and in the PythonEnvUtils the temp directory is created in

addToPythonPath

. There's this code:

if (!pythonFile.getFileSystem().isDistributedFS()) {
                // if the path is local file, try to create symbolic link.
                new File(targetPath.getParent().toString()).mkdir();
                createSymbolicLink(
                        Paths.get(new File(pythonFile.getPath()).getAbsolutePath()),
                        Paths.get(targetPath.toString()));
            } else {
                try {
                    FileUtils.copy(pythonFile, targetPath, true);

Now I fully expect that since I have python.files: s3a://code/flink/test.py the URI is s3a, hence it

isDistributedFS()

and the result should be that it's just copied to the targetPath. That's unless the URI doesn't indicate that it is a distributed file system - and hence the createSymbolicLink is being triggered. What seems weird though is that there seem to be both of these being triggered because the file is in /tmp/pyflink/uuid1/uuid2/test.py (the 626kb file is there) and a second link to /opt/flink/test.py from the /tmp/pyflink/uuid1/uuid3/test.py (incidently, I don't understand how we have 2 UUIDs here and not just a single UUID - since the creation of the tmpDir seems to only have a single UUID)

Path targetPath =
             new Path(
                      tmpDirPath,
                      String.join(
                                 File.separator, UUID.randomUUID().toString(), sourceFileName));

Could this be a bug? I'm no Java programmer, but this does seem very weird.

So this makes me wonder whether any of this is even possible. Can we have our scrips on S3 and get the SessionJob to pull the script from S3 instead of baking it into the container?

More info. So s3a: is considered a local file system, so I was wrong. Seems it does deal with the

if

to create the symlink. I can confirm that: (a) the flink operator is NOT downloading the test.py from s3 (b) the flink session cluster isn't either so it's only when the session job starts that the file is download from S3. I've gone back to first principles and removed all JARs from the image that relate to AWS/hadoop to ensure there's only s3-fs-hadoop/ there in the correct place (plugins/) and I can confirm that the jar is loaded when the session cluster starts.