No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

Untitled.txt

If I `⌃c` after it freezes, I get the following traceback

<@U0164GZJ5EK> tell him about our launched engine
<@U0327AHTCUU> great timing!

Check <https://github.com/ploomber/ploomber-engine|this one out>
Feel free to star it :star-struck:

so ploomber does not restrict cpu/memory, so there isn't any control to bump it up - ploomber adds some cpu/memory overhead but it's minimal. I'd start by <https://github.com/ploomber/ploomber-engine#notebook-profiling-cpu-gpu-ram|profiling> the notebook with papermill (we will integrate this into ploomber we haven't finished yet)

you can also try running the notebook in isolation with `ploomber task`

one of the problems with papermill is that it runs the notebook in a separate process and it has to communicate to the kernel, so it might happen that the kernel freezes and papermill enters in a weird state - we're working in a full replacement for papermill to fix those issues so stay tuned (actually, the profiling engine is an early version of what the new ploomber engine will be so give it a try)

Interesting, so it seems to work using `ploomber task`. For my own edification, why is that?

&gt; so it might happen that the kernel freezes and papermill enters in a weird state
this sounds likely.

Side note: you may want to add `memory-profiler` as a dep of `ploomber-engine` (I had to `pip install` it separately).

Running `mprof run papermill /path/to/notebook.ipynb &amp;&amp; mprof plot`  works as expected:

are you using the parallel executor? if so, it might be that

i wouldn't expect differences in behavior between `ploomber task` and `ploomber build` unless you're using the parallel executor

I'm not... maybe it's just a matter of how much other work my computer is doing at the time? It did happen a few times over the past couple days :thinking_face:

anyway, thanks for the pointers. I'll try to keep an eye out for anything more causal going on

Maybe if it's compute heavy the cores get busy context switching.. It'd be interesting to hear if that happens again