👋 Hello, team! I’m looking at integrating great expectations with datahub, is there any news on the support for different execution engines? (spark/pandas)

Hello William, I don’t think we have any concrete timeline for different execution engines at the moment. Could you check https://feature-requests.datahubproject.io if there is a similar request and if not add it there? It’s easier for the core team to track 🙂

Hey Pedro, We are currently running Great Expectations on in-memory batches through spark. We would like to get the expectation results into DataHub, but we're not sure what the best way would be to proceed. We know which datahub dataset we're validating, but using the GE operator would require us to pass this information through great expectations itself, and then modifying the validation action code to accept our new batch type. Maybe some batch metadata fields could be standardized for this, so that similar cases can work with datahub? Would you have any pointers? Of course, we're willing to upstream some code :)

Hello @quaint-branch-37931, I’m not very familiar with GE but I can try and find someone who is to help you out. that said if we don’t have that flexibility then a contribution would be welcome!

That would be great @incalculable-ocean-74010! Thanks!