Apache Pinot

Another question (related to the previous one): after I trigger a `LaunchDataIngestionJob` job with a file spec, for how long do I need to keep that job spec file around? Can it be deleted right after the job finishes, if I downloaded it from somewhere before I triggered the job?

you can just make a tmp file, no need to keep it around. It gets logged I believe. In scala something like
```val tmpFile = File.createTempFile("jobSpec", ".yaml")
tmpFile.deleteOnExit()```
works well

i know you said you want to use s3 in your previous thread, but 1) i’m not sure if you can, and 2) the host should at least have a tmp dir available

Ah, you gave me an idea: I can have a script that generates the job spec file, triggers the job and then deletes the file right after the job finishes. Thanks!

That’s basically what we do, via a (Java) tool that creates the job spec file based on CLI parameters. But we just put it in temp storage (same as what <@U0264HWF9FS> suggested), so no need to delete it.

Got it. I'll delete the spec later because I'll put it in a tmpfs mount, to avoid losing sight of disk space usage.