https://nf-co.re logo
Join Slack
Powered by
So <@U03L6DM9W> (<@U034AUTDUFM> in case you're int...
# nextflow-plugins
j
So @p (@User in case you're interested) I had time and here you can find the nf-parquet plugin repo. By the moment only able to read files but you can specify which schema to use specifying them in the configuration ā€¢ https://github.com/edn-es/nf-parquet In the example folder you can find a main.nf to read a 2Gb file (you need to download it before to run the pipeline). In my machine if I use the "full" schema it tokes 3 mnts to read the 46 millions of records, but if I use the onlyX schema it tokes 1 minute I'll work in the write feature and also in a more expressive way to specify the schema instead list+map (maybe creating a mini-dsl)
šŸ¤  1
p
Hi Jorge, this looks cool.
What does 
type: ["null","double"]
represent? I mean, why 
null
and 
double
(sorry for the late reply, iā€™m out of office until next week)
šŸ¹ 1
j
This is a WIP but by the moment is from the "fields" specification of avro Basic types can be "double", "string", "boolean", and so on "null" specify the field is optional https://avro.apache.org/docs/1.4.0/spec.html#schema_primitive
āœ… 1
I'm working in a mini-dsl to specify fields+schema, something as
Copy code
parquet {
    schemas{
        catalog {
            field "X" type "double" defaultValue null
            field "Y" type "double", "null" defaultValue null
            field "Z" type "double", "null" defaultValue null
        }
        schema "full"
        schema "onlyX" fields "X"
        schema "XAndY" fields "X", "Y"
    }
p
Sounds smart!
j
thnks, still is an idea but as is a simple dsl it's easy to implement/play with the idea
šŸ‘ 1
This is a first implementation of the miniDSL. I think it is also interesting because it "introduces" a closure into the configuration to be interpreted later by the plugin instead to be parsed at startup by nf-core (I would like to improve it with more complex types but by the moment it will work only with primitives types) https://github.com/edn-es/nf-parquet/blob/schema-dsl/example/nextflow.config#L6
I've continued working in the plugin and trying to figure how to read an S3 parquet file I founded a public bucket from Amazon with Bitcoin information, so .. let's try it:
Copy code
include { fromParquetFile } from 'plugin/nf-parquet'

url = "<s3a://aws-public-blockchain/v1.0/btc/blocks/date=2024-01-06/824538.snappy.parquet>"

Channel.fromParquetFile(url)
        | view {"""
            Hash: $it.hash
            bits: $it.bits
            size: $it.size
            coinbase param: $it.coinbase_param
            number: $it.number  
            transaction count: $it.transaction_count
            date: $it.date
            """.stripIndent()
}
(I haven't any idea about what every field means)
Copy code
nextflow run <http://bitcoin.nf|bitcoin.nf> 
N E X T F L O W  ~  version 23.10.0
Launching `<http://bitcoin.nf|bitcoin.nf>` [desperate_raman] DSL2 - revision: 8755b8d98a

Hash: 00000000000000000001fa3246b27e63c6aa9ee666c6b1009f6399db945eedf2
bits: 1703e8b3
size: 2075635
coinbase param: 03da940c194d696e656420627920416e74506f6f6c207a001306fa016061fabe6d6d2bb8ef668ab3cf9a781f76488c6c8d87ce7c45607825b3599266e0fd0b089efa10000000000000000000d7c4e967010000000000
number: 824538  
transaction count: 2814
date: 2024-01-06
Main problem is hadoop library I'm using for s3 has a lot of dependencies so I need to review all of them and exclude not required libs
Open in Slack
PreviousNext
Powered by