Hello. I wondered whether it is possible to exclude functions from using the RAY backend. I had a single pipeline with elastic queries and transformation code. I had to separate out the elasticsearch queries because it didn't work correctly with RAY. Now I have two services: elastic running with default driver and the transformation code running with the RAY driver. I believe Hamilton works on the backend by adding a RAY decorator to each function. Is is possible to exclude functions from adding this decorator? This is more for future reference.

Hey! So currently this is the case (that they’re all running on ray), however, it should be an easy(ish) fix. First, do you mind sharing how you’re calling your code? I assume you’re using the Ray Graph Adapter?

Hi Elijah, this is the relevant part of my run script:

@main.command()
def run():

    if config["ray_backend"]:
        output_type = h_ray.RayGraphAdapter(result_builder=base.PandasDataFrameResult())
    else:
        output_type = base.SimplePythonDataFrameGraphAdapter()

    logger_hook = lifecycle.default.PrintLn(print_fn=<http://logger.info|logger.info>)

    dr = (
        driver.Builder()
        .with_modules(
            pipe_load_data,
            pipe_prep_data,
            pipe_entity_features,
            pipe_add_features,
            pipe_risk_metric,
        )
        .with_config(config)
        .with_adapters(logger_hook, output_type)
        .build()
    )

Got it! OK, so yeah, this should be an easy change. If you’re interested in contributing we’d love contributions, but we can also take a stab at some point soon. FWIW I think this is a common pattern, and extends nicely what @Fran Boon did. For task-based orchestration its much more powerful but less well documented TBH.

Thanks Elijah, I will definitely take a look at task-based docs soon. Id like to contribute if I find some time. I am currently deep into some integration testing but will return to this when I have some headspace.

Great! Just reach out when you need it and we can help you out.