Who knows about docker container options? I had i...
# nostupidquestions
r
Who knows about docker container options? I had issues (again) on some of the spark based GATK tools with errors such as
Copy code
NFCORE_SAREK:SAREK:FASTQ_PREPROCESS_GATK:BAM_MARKDUPLICATES_SPARK:GATK4SPARK_MARKDUPLICATES (test2)
        > ERROR ~ Error executing process > 'NFCORE_SAREK:SAREK:FASTQ_PREPROCESS_GATK:BAM_MARKDUPLICATES_SPARK:GATK4SPARK_MARKDUPLICATES (test2)'
        >
        > Caused by:
        >   Process `NFCORE_SAREK:SAREK:FASTQ_PREPROCESS_GATK:BAM_MARKDUPLICATES_SPARK:GATK4SPARK_MARKDUPLICATES (test2)` terminated with an error exit status (3)
        >
        >
        > Command executed:
        >
        >   gatk --java-options "-Xmx12288M -XX:-UsePerfData" \
        >       MarkDuplicatesSpark \
        >       --input test2.sorted.bam \
        >       --output test2.md.cram \
        >       --reference genome.fasta \
        >       --spark-master local[4] \
        >       --tmp-dir . \
        >       --remove-sequencing-duplicates false -VS LENIENT
        >
        > Command exit status:
        >   3
        >
        > Command output:
        >   (empty)
        >
        > Command error:
        >       at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2416)
        >       at org.apache.spark.SparkContext.<init>(SparkContext.scala:334)
        >       at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        >       at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.createSparkContext(SparkContextFactory.java:185)
        >       at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.getSparkContext(SparkContextFactory.java:117)
        >       at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:28)
        >       at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:150)
        >       at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:203)
        >       at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:222)
        >       at org.broadinstitute.hellbender.runCommandLineProgram(java:166)
        >       at org.broadinstitute.hellbender.mainEntry(java:209)
        >       at org.broadinstitute.hellbender.main(java:306)
        >   Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
        >       at jdk.security.auth/com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
        >       at jdk.security.auth/com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:134)
        >       at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        >       at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
        >       at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
        >       at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        >       at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
        >       at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        >       at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2091)
        >       at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2001)
        >       at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:722)
        >       at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:672)
        >       at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579)
        >       at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2416)
        >       at scala.Option.getOrElse(Option.scala:201)
        >       at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2416)
        >       at org.apache.spark.SparkContext.<init>(SparkContext.scala:334)
        >       at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        >       at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.createSparkContext(SparkContextFactory.java:185)
        >       at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.getSparkContext(SparkContextFactory.java:117)
        >       at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:28)
        >       at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:150)
        >       at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:203)
        >       at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:222)
        >       at org.broadinstitute.hellbender.runCommandLineProgram(java:166)
        >       at org.broadinstitute.hellbender.mainEntry(java:209)
        >       at org.broadinstitute.hellbender.main(java:306)
claude lurk suggested this github.com/nf-core/sarek/pull/2248/…/06c68e2cc673…
'-v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro'
and it works. But I am not very knowledgable on these container configurations and would appreciate someone taking a look that does. My main concern is anything around users not being able to interact with their files after (which has happened in the past)
m
That was what was needed to get the container to work when the old docker.emulateUser flag was still around. It added those files as well the user info and such so the usernamespace matched inside and outside.
p
The error is because of java not being able to look up the running user and get user information for that. Providing those files from the surrounding system may help (if you have the user you're running as existing in the local files). This only relates tangentially to whatever users can read those files (which is both about what user creates them in the container environment, but also how it runs, e.g. with user namespaces "root" in the container can be the right user outside. If adding those file mappings work in your enviornment, it might be good enough.
r
github.com/nf-core/modules/pull/12637 PR to move this into the module and out of sarek
could I get a re-review
✅ 1