check out this tutorial! https://docs.ploomber.io/en/latest/deployment/online.html
currently, there are some restrictions: your pipeline must contain python function tasks. are you using scripts/notebooks?
we recently released a new notebook executor that contains all the bits and pieces to run a pipeline of notebooks while keeping everything in memory but we haven't started working on it yet
h
Himanshu Singh
02/09/2023, 11:52 PM
ok. let me go through. we are using notebooks. By design, we want to be able to use scripts, functions as well as functions within class objects as part of the pipeline.
e
Eduardo
02/09/2023, 11:55 PM
sure. so if I understand correctly, you have a pipeline.yaml with some notebook steps and you want to run pass the output of these notebooks but keep them in memory?
Eduardo
02/10/2023, 2:43 PM
update: added a short example showing how to pass objects between notebooks. you can write a wrapper that turns your ploomber pipeline into an in-memory one. happy to guide you if you wanna give it a try! https://engine.ploomber.io/en/latest/howto/pass-objects.html
meerkat 1
i
Ido (Ploomber)
03/17/2023, 8:00 PM
@Himanshu Singh were you able to get it work? Still need guidance?