https://ploomber.io/ logo
Join Slack
Powered by
This message was deleted.
# ask-anything
s
This message was deleted.
e
you can achieve this by converting your file client into a placeholder:
Copy code
clients:
  File: '{{my_client}}'
then, in your env.prod.yaml:
Copy code
my_client: clients.s3_client
and your env.yaml:
Copy code
my_client: none
then you can switch between these two with an env variable: https://docs.ploomber.io/en/latest/user-guide/configuration.html#configuration-dev-prod
👍 1
f
Thanks so generally a 'none' will be treated as a null value, since all placeholders require to be strings I wasnt sure because i cant inject an actual null yaml value it always becomes 'null'
e
np, hope this helps!
f
do you recommend reusing those clients in tasks or should they be kept only for pipeline internal purposes and init separate client for work that the task needs to do that includes IO. For example my task downloads a bunch of S3 files stiches them together and then has 1 product file.
def helps thanks!
e
good point. I think it's ok to re-use them. since the pipeline client already has the config, it'll make it simpler to embed it into your code
f
@Eduardo it doesn't seem to work. Passing a value of none triggers 
TypeError: Expected dotted path spec to be a str or Mapping, got None (type: NoneType)
Copy code
File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/cli/parsers.py", line 495, in load_dag_from_entry_point_and_parser
    dag, args = _process_file_dir_or_glob(parser)
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/cli/parsers.py", line 447, in _process_file_dir_or_glob
    dag = DAGSpec(dagspec_arg, env=env).to_dag()
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/spec/dagspec.py", line 484, in to_dag
    dag = self._to_dag()
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/spec/dagspec.py", line 521, in _to_dag
    dps = dotted_path.DottedPath(dotted_path_spec,
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/util/dotted_path.py", line 52, in __init__
    self._spec = DottedPathSpecModel.from_spec(dotted_path, strict=strict)
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/util/dotted_path.py", line 444, in from_spec
e
are you passing the client to each task?
Copy code
tasks:
  - source ...
    client: ...
or do you have a pipeline-level client?
Copy code
clients:
  File: clients.get_s3_client
I think the task-level won't work but the pipeline one will
f
It's a pipeline level and the following error is when the value is set to none or "none" as advised to run local filesystem in the dev env. Seems like it iterates over the set of clients and expects a string but receives null value as defined by the none which explodes.
e
ah, I think I might have introduced this in a recent update. can you open an issue?
👍 1
f
Open in Slack
PreviousNext
Powered by