Hi <@U08RG6UAA0L> I am interested in implementing ...
# daft-devv
VOID 001
11/06/2025, 3:51 AMHi @Srinivas Lade I am interested in implementing distributed flotilla daft dashboard in order to make it work on Ray cluster environment. Is there any suggestion or documentation/guide I can follow? Thanks
BTW, I've checked the roadmap https://github.com/Eventual-Inc/Daft/discussions/5155 already, looking for some good starting point for the distributed version.
š 1
s
Srinivas Lade
11/06/2025, 9:39 PMHey @VOID 001, it's awesome to hear that you want to help with the distributed dashboard support! I updated the roadmap to track our current plan for the dashboard, but for context:
ā¢ We are current focused on adding support for emitting metrics through OpenTelemetry (OTEL), as we know many users have existing metrics setups like Prometheus that they want to dump metrics into
ā¢ In addition, we are working on taking Swordfish-level worker metrics and aggregating them on the Flotilla scheduler node for global metrics
ā¢ Lastly, we are working on getting the subscriber framework, which the dashboard uses, working with Flotilla. That should enable the dashboard
Srinivas Lade
11/06/2025, 9:47 PMThere are a couple of loose ends that we currently need to deal with that I'd love some help with:
ā¢ Right now, when a query fails or is canceled, the dashboard gets stuck. We should track the failed and canceled states and emit them to the dashboard
ā¢ Once we aggregate metrics on the scheduler, we need to add some special handling for some operators. For example, with Filter, we compute a "selectivity" metric, which is the rows_in / rows_out. I've listed all operators that need special handling in the roadmap. It would be really helpful if you could take on some of them
ā¢ We need to track the runner type (native vs distributed) for queries on the dashboard)
If any of those sound interesting, let me know!
v
VOID 001
11/08/2025, 5:14 AMSure! I would first take the issue "dashboard shows running forever when a query fails or canceled" and see how to fix it. Then discuss in further detail
š 1
VOID 001
11/10/2025, 4:48 AMJust submitted an issue for the daft-dashboard https://github.com/Eventual-Inc/Daft/issues/5518
š 1
VOID 001
11/16/2025, 2:21 PMHi @Srinivas Lade, I tried to add "Canceled" and "Failed" state to the dashboard, it shows like this, does this look good to you? btw the draft PR is here: https://github.com/Eventual-Inc/Daft/pull/5576
You can take a look and provide some comments if you have time, I will clean it up recently
VOID 001
11/16/2025, 3:02 PMI also have a question, when I run
make precommit Copy code
error: cannot find attribute `pyclass` in this scope
--> src/common/metrics/src/lib.rs:27:3
|
27 | #[pyclass(module = "daft.daft", eq)]
| ^^^^^^^
error: could not compile `common-metrics` (lib) due to 1 previous errorc
Colin Ho
11/19/2025, 3:01 AM@VOID 001 srinu is out for next 2 week, I can help instead
Colin Ho
11/19/2025, 3:03 AMI also have a question, when I runPyo3 is a default feature in the project, and when we run checks we run with and without it. You probably need to add, it shows common-metrics cannot pass compilemake precommit
#[cfg(feature = "python")]v
VOID 001
11/19/2025, 3:38 AMThanks @Colin Ho!
I have solved that problem "marked as *solved*" š
VOID 001
11/19/2025, 3:38 AMCould you please also take a look at my PR? link: https://github.com/Eventual-Inc/Daft/pull/5576
VOID 001
11/19/2025, 8:37 AMKindly ping @Colin Ho just in case if you miss my message š . https://github.com/Eventual-Inc/Daft/pull/5576 If you have time, could you please take a look at this PR, thanks
c
Colin Ho
11/19/2025, 8:41 AMYeah I saw, i'll take a look soon!
daft party 1