Hahaha but it was not necesary as current nf sql implement duckdb

nooo native was better! 😄

Could we just get an apache arrow plugin? Or just use the Java Implementation? https://arrow.apache.org/docs/java/quickstartguide.html

This was what I used in my (no completed) plugin but I realised nf-sql implements duckdb and I didnt continue with my plugin. Right now I'm a little busy but will try to recover it (and push my progress at github)

Jorge the plugins making machine! 😅

https://github.com/Emiller88/duckdb-nf/tree/main/parquet-reader Yeah it's slick and solves any parquet reading things people might want to do, but a direct arrow implementation might go really far. I'm imagining you could have a Nextflow type for it like

path

and that might really open up the bear package(pandas/polars/dpylr) type of things people like to do. I think the Nextflow operators and dataflow paradigm might be fun.

Kamloops Bc GIF.gif

Jorge the plugin King

@Jorge Thought this might be of interest to you https://github.com/nextflow-io/nf-sqldb/issues/19

Reviewing the PR to implement the batch I think it'll be rejected if not improved. I mean, the executeBatch (https://github.com/duckdb/duckdb/pull/10011/files#diff-7e79d886d5e5e16e577d9e329b460f6d163cf71f8daba8c8acb46ffbd22d5408R469) is executing one by one the batch values, so very inefficient