Does anyone have an opinion on buildt-in benchmarking? First step in https://github.com/nf-core/ampliseq/pull/1021

I'm not sure I understand. What's the difference from a test case?

For now its just that one can specify expected sequences (and abundances) and the pipeline will take over the comparison to the ASV sequences & abundances. Applications could be to run a mock sample among other samples as routine control or to explore the impact of settings on results.

Late reply, sorry; lots of meetings today. This was not what I was thinking of when I saw the word benchmarking. Instead that suggests, to me, a comparison of performance more along the lines of is A better/faster than B. What you're after is more an inbuilt function to evaluate known positives e.g. in the form of a mock community. As I'm not a native speaker I honestly don't know if my interpretation of the word is correct. From the PR it's not clear to me how you aim to achieve this. Have you thought of a significance test between expected outcome and realized, or something else?

Apologies, I wasnt thinking about the correct wording too much. I agree, benchmarking is something more comprehensive than what the PR does. Its essentially comparing observed ASV sequences against expected reference sequence(s) and report sequence mismatches/match rates. I'll modify the wording throughout.

I'm not sure what's the best word. So, back to what you aim to do: Would you mainly look for the presence of ASVs in the realized outcome compared to the expected plus report how different they are (nucleotide substitutions) or do you aim to weigh in their relative abundance too? Plus report any sequences present in one but not the other, I suppose? Have you done any tests on mock communities? What differences can one expect?

Would you mainly look for the presence of ASVs in the realized outcome compared to the expected plus report how different they are (nucleotide substitutions) or do you aim to weigh in their relative abundance too?

The current PR only checks how different sequences are. Relative abundance would be done in separate PR, I am wary to add too much at once in one PR. Several smaller PRs might be easier to handle, the first one is the most difficult one I assume.

Plus report any sequences present in one but not the other, I suppose?

Yes, but for now in numbers, not sequence IDs.

Have you done any tests on mock communities? What differences can one expect?

Some time ago yes, not recently. But would be nice to have it in the pipeline to do that more frequently. Similarities of analysis results to expected outcomes for mock communities are typically high, but not perfect. I was thinking about adding nanopore analysis to the pipeline (ASV generation as in https://github.com/nf-core/ampliseq/issues/741) and to check how good that is working I did some comparisons, and I thought it would be best to have some scripts for comparisons/assessment of results in the pipeline directly.

Sounds convincing. I'm all for step-wise approaches. I'll approve the pr

I'm still a bit modifying 😄

OK, let me know.

Alright, if you want to have a look at github.com/nf-core/ampliseq/pull/1021, now its ready for review I think 🙂

I probably won't have time until tomorrow

Hi Daniel, I have another one: github.com/nf-core/ampliseq/pull/1023 The CI test is just our usual test data with made-up expected data, so results are not meaningful. Alternatively a mock dataset could be added or could replace the current test data (starting with Illumina PE). Do you think that would be helpful? I worked with Zymo's standard & log distribution in the past, do you have any preferences?

It's been a long while since I had a project with mock data, so I don't have any preferences. Will look at the PR right away.

I sort of postponed the change to a zymo dataset. Other stuff feels more important 😞 I have a new PR though: github.com/nf-core/ampliseq/pull/1025 I am unsure about one sort of important point I think: I added several metrics based on abundances. The abundances I provide is just the complete abundance table, i.e. includes FN & FP. For some metrics that seems the correct choice, e.g. bray-curtis (and other) distances of observed to expected abundance table. However, for others, like spearman correlation or RMSE, that might not be appropriate: it seemed to me that by adding FN & FP these values improved while they should intuitively rather get worse. Shall I exclude FN and/or FP from the data that is used for some metrics? I checked on mock data, an example barplot with FN & Fp and a scatter plot with TP only (produced by PR1025):

I'm not sure I follow. (You've thought much more than I.) When you talk about correlation, you mean how truth (i.e. expected) correlates with what denoising gives? When I mentioned Bray-Curties earlier I was thinking of comparing distances that are expected with those that Ampliseq gives. (One beta diversity measurement I'm missing is centered log ratios (CLR) and euclidian distances based on them. There's a variant in Vegan called "robust CLR" that deals differently with the zeroes that I'm leaning towards because of its simplicity.)

Sorry, I am recently a little too deep into that I think and I have trouble explaining. > When you talk about correlation, you mean how truth (i.e. expected) correlates with what denoising gives? In that case I mean sprearman correlation of observed to expected abundances per sample. This is now only applied to TP, i.e. not on anything with 0 abundance (FN+FP). Essentially on data as shown in the scatter plot above. > When I mentioned Bray-Curties earlier I was thinking of comparing distances that are expected with those that Ampliseq gives. Currently I have Bray-Curties between observed and expected abundance (incl the complete abundance table, i.e. TP,FP,FN), per sample. On data as shown in the bar plot above. > I'm missing is centered log ratios (CLR) Until now I used base R. That can be changed at any time I guess and any new metric can be added later.

Sorry, I am recently a little too deep into that I think and I have trouble explaining.

> When you talk about correlation, you mean how truth (i.e. expected) correlates with what denoising gives?

In that case I mean sprearman correlation of observed to expected abundances per sample. This is now only applied to TP, i.e. not on anything with 0 abundance (FN+FP). Essentially on data as shown in the scatter plot above.

I see. TP makes sense as that means only basing the correlation on what's in the mock.

> When I mentioned Bray-Curties earlier I was thinking of comparing distances that are expected with those that Ampliseq gives.

Currently I have Bray-Curties between observed and expected abundance (incl the complete abundance table, i.e. TP,FP,FN), per sample. On data as shown in the bar plot above.

Right, so the BC distance between each sample's expected ASVs and its observed? I think using all observed makes sense.

> I'm missing is centered log ratios (CLR)

Until now I used base R. That can be changed at any time I guess and any new metric can be added later.

OK. Including the Vegan package would probably make it easier to calculate also Bray-Curtis. OTOH, I think BC would do the job perfectly well here since it's just comparing observed with expected for each sample.

PR is ready to review 😅