Hello community! We've been building our distributed data pipelines w/ Daft for some time now - One thing that could really bring our experience with the library to another level is low-level access to progress loggings of our runs. We're managing the orchestration of Daft runs via Dagster. We've used

dagster pipes

to manage communication of metadata between dagster and other external services, so alot of the plumbing to bring in logs from other services is already in place. While Daft provides a really clean log at runtime in the CLI, accessing this info programmatically (ie; via an API) during the actual run remains difficult (and seems not possible yet based on the public forums ive been reading through) If anyone has come across a similar issue and has any useful insights on how we can access runtime logs / progress bars, please feel free to share or comment here!

Thank you @John Rahmon for the feedback. There are a few initiatives that are ongoing focused on observability so this comment is quite timely for us. cc @Srinivas Lade