https://ploomber.io/ logo
Join Slack
Powered by
This message was deleted.
# ask-anything
s
This message was deleted.
e
Is this a variable generated by another task or a parameter that's coming from outside the pipeline?
j
a variable used in a cell in a notebook The key name used to upload to s3
1st step = Fetch from psql database --> upload to s3 under a key_name 2nd step = Download from s3 with the key_name clean data 3rd Step = Upload to another s3 Bucket
the key name is generated from
Copy code
s3_file = f"ploomber/{date_pipeline_run}/{table}/{filename}"

filename = f"{extract_date}_{table_newest_update_date}_{table_oldest_update_date}_{table}.parquet"
I tried using the %store <variable>
e
Ok, so if the key name is known before the pipeline starts execution. you can pass it via “params” https://docs.ploomber.io/en/latest/api/spec.html You can pass the same key as a param to as many tasks as you want. But to keep things simple, you may defined it once in an env.yaml file and reference it https://docs.ploomber.io/en/latest/user-guide/parametrized.html
I'm guessing they key name is known before the execution right? If it's generated via the previous task, then you need to add another product to the previous task. For example you can store the key in a JSON file and load it in the next task
j
the key name is generated during 1st step
build from the date the pipeline would be run and if the data change at the source
e
Alright then the way to do is to store it in a JSON or a text file. We actually have an open issue to simplify this
j
alright
e
Let me know if you need more help!
j
I was able to get to this with the %store magic
e
Ah. Interesting. I'm not familiar with the stores magic, but I'll dig into that. Maybe we can incorporate that into the documentation. Thanks for sharing!
j
Good !
writing to a json and retrieving it works !
👍 1
e
great!
3 Views
Open in Slack
PreviousNext
Powered by