Hi! I know the pipeline currently expects GFFs in Prokka format, but I've been using Bakta annotation and ran into some column/attribute mismatches in the annotation module. I modified

NFCORE_MAGMAP:MAGMAP:PROKKAGFF2TSV

locally to be robust to missing columns (like

ec_number

) and to check for Bakta's

Dbxref

format. It works for Bakta now and stays compatible with Prokka. Do you think this is of general interest? If so I could create a pull request

Absolutely. I have actually been thinking of incorporating Bakta as an alternative. The only problem is that it's not supposed to be used for archaeal genomes as far as I understand, so one would have to split genomes and use Prokka for Archaea and Bakta for Bacteria.

Okay cool, will try to submit it in the next days.

Thanks!

I have annotated the genome

GCA_002688505.1_ASM268850v1.fna.gz

from the test-dataset repo using bakta to test the new script. I noticed that when bakta is run with the

--compliant

flag, the only difference of the gff file from prokka is that it's

dbxref

instead of

db_xref

. Anyways, as the

PROKKAGFF2TSV

script in my PR is set up, it works in either case. Should I submit a PR for the test-dataset to add the gff annotated with bakta? Thanks for your help!

Sounds great, please add the genome and update the samplesheet. I think you can use

.gff3.gz

as the suffix as we have a too strict schema in

assets/schema_genomeinfo.json

that also needs to be updated (just add that to the same PR since it's not merged yet). Since Bakta uses this suffix, we're going to make people annoyed if they have to rename all their files to use

.gff.gz

when files are produced as

.gff3.gz

.

I'm a little lost currently: When I'm running

nf-test test -profile docker,test

locally, some assertions fail, also with the master branch. Some snapshots are different (seems like largely lists of tools and their versions) and then there are these assertions:

assert path("${params.outdir}/summary_tables/magmap.genomes2orfs.tsv.gz").linesGzip.size() > 800

and `assert path("${params.outdir}/summary_tables/magmap.prokka-annotations.tsv.gz").linesGzip.size() > 800`which fail because the output files only have 425 lines. But when I created the PR the same tests seemed to pass when they were run automatically on github. I tried with the master and dev branches, without my modifications.

Is this after updating the spread sheet with genomes? If the new annotation doesn't work as it should, it would be consistent with fewer rows in those files.

No it happens also when I clone the release version from the master branch, no changes made at all

You didn't change the test-dataset?

No I have not changed it, but then there must be some issue on my machine, I will look into it. Thanks for checking!

I think I know: Change to

--profile +docker,test

, note the

+

.

Ha okay, so when I run

nf-test test --profile +docker,test

from the second test on it fails again, but the assertion with the line numbers seems to work. The only assertion failing is the snapshot comparing all created output files, the differences are from bbduk. But when I run

nf-test test --profile +docker tests/*.nf.test

all tests pass 🙂 So I would now change the testdata, should I add one bakta gff3 to the

duplicates_genomes.csv

? Because

genometest.csv

contains only one genome and

genomes_full_test.csv

links to aws

Strange with the test. I don't know why the pipeline would produce different files with the different commands. However, a colleague recently cancelled the file list test from #CEA7TBJGJ, so maybe there's something. But, since tests pass for you now, and they seem to also on GitHub, all is well I suppose.

Okay weird, but as long as they pass now locally and on github I also guess it's okay. I realized if I update

duplicates_genomes.csv

the tests of the magmap master branch would fail since the parsing of the

gff3

does not work. I would propose instead to add a new csv

duplicates_genomes_bakta.csv

to the testdata repo which has the

gff

changed to the

gff3

from bakta and change the

duplicates_genomes

test in my PR to

duplicates_genomes_bakta

and then use the new

duplicates_genomes_bakta.csv

for that modified test. That way the tests of the master branch will still work since all the original files in the test-data repo remain unchanged and the new version of the test checks the bakta compatibility. If you agree I would create the PR for the test-data repo, as soon as changes are merged I would update the PR for the magmap dev repo, then the new duplicates test should pass

I'm not completely following. Have you merged the original PR? I thought you could add to that that

.gff3

with or without

.gz

should be accepted in the genome sheet. Any PRs to master would then work when it's time for them. Would that fix what you describe?

I have not merged the original PR no. So what I mean is that the tests use the

duplicates_genomes.csv

from the test-data repo. If I would modify that file, the tests of the master branch would fail since they also use it. So instead I would add new files,

duplicates_genomes_bakta.csv

and the gff3 to the test-data repo and leave the original unchanged. Then I would modify the duplicates test in my PR to use the new genome file from the test-data repo to test the feature.

But there are no CIs run for the master branch until we make a master branch PR, right?

Hm yes true, but I thought if anyone runs tests for the master branch repo until the fix makes it to the master branch, it would be better if they pass. But I'm completely new to this, if you think this is okay it's also fine for me to change the original

duplicates_genomes.csv

It is complicated when one has to update the test data so I'm not 100% sure either, but I can't see why it wouldn't work as tests only runs when there's a PR.

Okay but don't I need to update the test data anyways to implement this?

#C02L5UB4Y9G is the way to go. It's just complicated when one has to do changes there since one has to be sure they're correct before one can start fixing the pipeline code. I usually work with a local file first that I pass to the pipeline with a command line param until it seems to work. This doesn't work for nf-test testing though, so then one has to merge a test-data PR. It's not unusual that errors creep in in this step so I have to make repeated PRs... In any case, it's normal to just merge PRs to pipeline branches in test-data without review. Feel free to do that so you don't have to wait.

Okay thanks for the explanation. I will first test this with my fork of the magmap branch of #C02L5UB4Y9G , then create a PR for the test-data and as soon as this is merged go on to update the original PR I created for the magmap dev branch.

Thanks a lot for your efforts! A relatively straight forward contribution that ended up a bit convoluted.

Yes but was interesting for me, definitely learned something, thanks the guidance. Just to be sure we're on the same page: I update the existing

duplicates_genomes.csv

with the link to the new bakta

gff3

correct?

Correct.

All done and merged. As a more general follow-up to something you mentioned earlier: I checked the Prokka documentation again and I think while, as opposed to bakta, prokka can also be used for achaeal genomes this would need to be specified. The

--kingdom

parameter defaults to

Bacteria

. So as far as I can tell the way the pipeline is set up now it would not support using both archaeal and bacterial genomes. Theoretically as you said it could be possible to split them and annotate the genomes separately. We could either make this a user input or infer the kingdom from the GTDB-tk output for which we would need to slightly change the workflow order.

Thanks again for your contribution! What I had in mind was, as you're into, to decide which annotation program to use based on taxonomy provided either with

--gtdbtk_metadata

for user-provide genomes or

--gtdb_metadata

for public, NCBI-genomes. If

--kingdom

(unfortunate name; should be

--domain

) supports

Archaea

, one would just have to supply the domain with the call. This would probably mean some changes to the pipeline logic besides adding Bakta. The first thing I'd try is to parse the taxonomy sources into a channel that can be `filter`ed and `join`ed appropriately to select genomes for either annotation program or set the domain.

Sure, thank you! That sounds like a good approach, I hope I find some time in the coming weeks to look into it. I will keep you updated.

That would be great! I'll make an issue and put your name on it.

Okay sounds good!

I forgot to say that the channel way might explode since the GTDB metadata files are so large nowadays. Even if one reduces the actual size to only columns 1, 19 and 20 (accession, species-representative and taxonomy) it's likely large and will probably have to be full size for a moment before one has reduced the columns.

hmm okay good to know, thanks for the warning