https://nf-co.re logo
Join Slack
Powered by
<@U015PB15C5U> we are having a problem with taxpas...
# taxprofiler
j
@moritz we are having a problem with taxpasta and conda in the release PR. Docker works fine, but in the sister conda test, taxpasta appears to be failing for all files it is receiving due to the error I show in this thread. Esssentially we are failing with a large pandera deprecation warning and then a critical error with a KeyError fialure_case, even though as far as I can tell it looks fine. Do you think this can be fixed by pinning pandas/pandera?
👍 1
The error:
Copy code
ERROR ~ Error executing process > 'NFCORE_TAXPROFILER:TAXPROFILER:STANDARDISATION_PROFILES:TAXPASTA_MERGE (metaphlan|metaphlan3)'

Caused by:
  Process `NFCORE_TAXPROFILER:TAXPROFILER:STANDARDISATION_PROFILES:TAXPASTA_MERGE (metaphlan|metaphlan3)` terminated with an error exit status (1)


Command executed:

  taxpasta merge \
      --profiler metaphlan \
      --output metaphlan_metaphlan3.tsv \
       \
       \
       \
      2611_metaphlan3.metaphlan_profile.txt 2613_metaphlan3.metaphlan_profile.txt 2612_metaphlan3.metaphlan_profile.txt
  
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_TAXPROFILER:TAXPROFILER:STANDARDISATION_PROFILES:TAXPASTA_MERGE":
      taxpasta: $(taxpasta --version)
  END_VERSIONS

Command exit status:
  1

Command output:
  [13:02:07] CRITICAL Error in sample '2611_metaphlan3.metaphlan_profile' with profile '2611_metaphlan3.metaphlan_profile.txt'.                                                       merge.py:419
             CRITICAL                column                                     failure_case index schema_context                          check  check_number                        merge.py:424
                      0  relative_abundance  KeyError("<class 'pandas.core.series.Series'>")  None         Column  greater_than_or_equal_to(0.0)             0                                    
                      1  relative_abundance  KeyError("<class 'pandas.core.series.Series'>")  None         Column   less_than_or_equal_to(100.0)             1                                    

Command error:
  /home/james/cache/conda/env-b942a33bd54bec81-c9db2a34faaff75b8d8bb7cfbe848899/lib/python3.14/site-packages/pandera/_pandas_deprecated.py:160: FutureWarning: Importing pandas-specific classes and functions from the
  top-level pandera module will be **removed in a future version of pandera**.
  If you're using pandera to validate pandas objects, we highly recommend updating
  your import:
# old import import pandera as pa # new import import pandera.pandas as pa
Copy code
If you're using pandera to validate objects from other compatible libraries
  like pyspark or polars, see the supported libraries section of the documentation
  for more information on how to import pandera:
  
  <https://pandera.readthedocs.io/en/stable/supported_libraries.html>
  
  To disable this warning, set the environment variable:
export DISABLE_PANDERA_IMPORT_WARNING=True
Copy code
warnings.warn(_future_warning, FutureWarning)
  [13:02:07] CRITICAL Error in sample '2611_metaphlan3.metaphlan_profile' with profile '2611_metaphlan3.metaphlan_profile.txt'.                                                       merge.py:419
             CRITICAL                column                                     failure_case index schema_context                          check  check_number                        merge.py:424
                      0  relative_abundance  KeyError("<class 'pandas.core.series.Series'>")  None         Column  greater_than_or_equal_to(0.0)             0                                    
                      1  relative_abundance  KeyError("<class 'pandas.core.series.Series'>")  None         Column   less_than_or_equal_to(100.0)             1                                    

Work dir:
  /home/james/git/nf-core/taxprofiler/testing/work/6c/b16324f7473f6ccff107ef6234bab4

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details
(working!) Docker image version info:
Copy code
sh-5.2# python
Python 3.12.3 | packaged by conda-forge | (main, Apr 15 2024, 18:38:13) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> import pandera
>>> print(pandas.__version__)
2.2.2
>>> print(pandera.__version__)
0.19.3
>>>
And in the conda env:
Copy code
Python 3.14.0 | packaged by conda-forge | (main, Oct 22 2025, 23:24:08) [GCC 14.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
i>>> import pandera
>>> print(pandas.__version__)
2.3.3
>>> print(pandera.__version__)
0.26.1
m
Yes, this looks like pinning pandera should do the trick.
j
Should we do that in taxpasta itself or on the bioconda recipe?
m
Would be best in both places 😬
j
What notiation should I use here?
Copy code
dependencies = [
    "depinfo ~=2.2",
    "numpy ~=1.26",
    "pandas ~=2.1",
    "pandera ~=0.26",
    "taxopy ~=0.13",
    "typer ~=0.16",
]
?
I'm not fmailiar wit ~=
m
That means for pandera >= 0.26 && <1.0
Question is which version introduced the change.
j
How dos it get the 
<1.0
from 
~=
? 🤔 But here is the lazy PR for now 😬
m
j
Ahhh ok
So requires semantic-like versioning
👍 1
@moritz what's your capacities atm to look at this? I could just go and pin the conda recipe for the time being... Just asking as our release is sort of blocked because of it
(so I'm wondering what the most efficiency way to fixing this would be)
m
I'm happy for you to apply the fix. Otherwise I have time tomorrow to look into it.
j
Tomorrow is fine 🙂
👍 1
It would make sense to fix it in taxpasta anyway in the long run !
Thank you!
m
Cool and sorry for not having been available 😞
j
No worries!
Been there 🙂
Update: So i tried pinning python to 
3.13
, and then it ran through. In this case we had 
pandera
0.26.1 and 
pandas
2.3.3
Copy code
pandas                              2.3.3            py313h08cd8bf_1          conda-forge
pandera                             0.26.1           hd8ed1ab_0               conda-forge
pandera-base                        0.26.1           pyhd8ed1ab_0             conda-forge
pip                                 25.3             pyh145f28c_0             conda-forge
prometheus-cpp                      1.3.0            ha5d0236_0               conda-forge
pyarrow                             22.0.0           py313h78bf25f_0          conda-forge
pyarrow-core                        22.0.0           py313he109ebe_0_cpu      conda-forge
pydantic                            2.12.4           pyh3cfb1c2_0             conda-forge
pydantic-core                       2.41.5           py313h843e2db_1          conda-forge
pygments                            2.19.2           pyhd8ed1ab_0             conda-forge
python                              3.13.9           hc97d973_101_cp313       conda-forge
python-dateutil                     2.9.0.post0      pyhe01879c_2             conda-forge
python-tzdata                       2025.2           pyhd8ed1ab_0             conda-forge
python_abi                          3.13             6_cp313                  conda-forge
pytz                                2025.2           pyhd8ed1ab_0             conda-forge
I'm pretty sure it's the same as in python 3.14 recipe, so maybe it's a python issue?!
Copy code
$ diff python_313_conda_env.txt python_314_conda_env.txt 
1c1
< # packages in environment at /home/james/cache/conda/env-4952a345cb869b199563723d2c6b444b:
---
> # packages in environment at /home/james/cache/conda/env-c9db2a34faaff75b8d8bb7cfbe848899:
25c25
< biom-format                         2.1.17           py313h54dd161_1          conda-forge
---
> biom-format                         2.1.17           py314h31f8a6b_1          conda-forge
35c35
< h5py                                3.15.1           nompi_py313h253c126_100  conda-forge
---
> h5py                                3.15.1           nompi_py314hc32fe06_100  conda-forge
97c97
< numpy                               2.3.5            py313hf6604e3_0          conda-forge
---
> numpy                               2.3.5            py314h2b28147_0          conda-forge
101c101
< pandas                              2.3.3            py313h08cd8bf_1          conda-forge
---
> pandas                              2.3.3            py314ha0b5721_1          conda-forge
106,107c106,107
< pyarrow                             22.0.0           py313h78bf25f_0          conda-forge
< pyarrow-core                        22.0.0           py313he109ebe_0_cpu      conda-forge
---
> pyarrow                             22.0.0           py314hdafbbf9_0          conda-forge
> pyarrow-core                        22.0.0           py314h52d6ec5_0_cpu      conda-forge
109c109
< pydantic-core                       2.41.5           py313h843e2db_1          conda-forge
---
> pydantic-core                       2.41.5           py314h2e6c369_1          conda-forge
111c111
< python                              3.13.9           hc97d973_101_cp313       conda-forge
---
> python                              3.14.0           h32b2ec7_102_cp314       conda-forge
114c114
< python_abi                          3.13             6_cp313                  conda-forge
---
> python_abi                          3.14             8_cp314                  conda-forge
120c120
< scipy                               1.16.3           py313h11c21cd_1          conda-forge
---
> scipy                               1.16.3           py314he7377e1_1          conda-forge
Those are the differences... so basically it's a pyhton issue... do you want to try and fix that @moritz or should I just pin the recipe to pyhthon 13 or lower?
I've updated the PR accordingly: https://github.com/taxprofiler/taxpasta/pull/159 But in the meantime I will update the conda recipe
Open in Slack
PreviousNext
Powered by