Hello, we are currently trying to implement the following transformer https://datahubproject.io/docs/metadata-ingestion/docs/transformer/dataset_transformer/#add-dataset-datasetproperties. The transformer part of my yaml looks as follows:

transformers:
    -
        type: add_dataset_properties
        config:
            add_properties_resolver_class: 'cdp-datahub-actions.snowflake_properties:SnowflakePropertiesResolver'

and I can see that cdp-datahub-actions is installed in the datahub-actions pod using pip freeze. Despite this I’m getting the error

Failed to configure transformers: No module named 'cdp-datahub-actions'

. Have I installed the module in the wrong place? Any advice would be super appreciated!

Did you tried to create volume in your pod with your custom modules and added the path to the modules in the PYTHONPATH environment variable?

I ended up doing this:

-- create venv that snowflake ingestor uses and enter into that venv
RUN python3 -m venv /tmp/datahub/ingest/venv-snowflake-0.9.6
ENV PATH="/tmp/datahub/ingest/venv-snowflake-0.9.6/bin:$PATH"
-- install local modules and executor
RUN pip install .
RUN pip install 'acryl-datahub[executor]'

-- leave the venv and go back to base
ENV PATH=(OLD_PATH)

That seemed to resolve my issues, but doesn’t feel very clean. Is that what you meant?

I thought in a simillar way, but without using distinct venvs. Basically, install all acryl-data dependencies and your custom classes dependencies and after add the path to your custom classes package in your PYTHONPATH. This will resolve your import error problems.

hi everyone! I happen to tackle the same problem right now, but my level of comprehension is somewhat lower. Could you please explain, where should I put my custom PropertiesResolver class for datahub to see it?

Hello @busy-train-56443, In the environment where you run your ingestion, you have to change the PYTHONPATH environment variable to contains the absolute path to your custom classes.

Thanks, I’ll try it

Hi @busy-train-56443, I'm facing the same issue at the moment. Could you explain where/how you ended up adding your custom package, so that it's available to the virtual environment used by the UI ingestion?

@average-cartoon-6213 I’ll reply today, no worries

Thanks for taking the time @busy-train-56443! I just realised that this thread is specific to the

add_dataset_properties

transformer. We are actually trying to install a custom transformer like here. Apologies for the confusion. In our case we are using the

acryldata/datahub-actions

docker image and are running UI ingestions on a schedule. We are trying to understand where in the container we have to copy our transformer code into, so that it will be installed into the venv created for the ingestion runs.

Oh, I understood. We had this problem too. You need to copy it to the /tmp/datahub/ingest/venv-<WE_HAVE_REDSHIFT_HERE>/lib/python3.10/site-packages

@busy-train-56443 we tried this before, but after reading your messages carefully I found that we had an issue in the

entry_points

of the setup.py we're using to install the package. It's working now and the transformer is used during ingestion, thank you! Nevertheless we now manually copied and installed the custom package inside the actions container after the venv in

/tmp/datahub/ingest/

is created by a first (failing) ingestion run. Did you find a way to install the code from the beginning (as the venv does not exist before the first ingestion)?

@average-cartoon-6213 no, we didn't have a need to do this, so we never really dug into that direction