Hi all, I am trying to make changes to an ingestion source in the metadata-ingestion module, and trying to create a Docker image to test the changes but running the Dockerfile

docker/datahub-ingestion/Dockerfile

throws the error below. Can someone help me out with this? Command:

docker build \
--platform linux/amd64 \
-f ./docker/datahub-ingestion/Dockerfile \
-t datahub-iceberg-ingestion:latest .

Error:

=> ERROR [full-install 1/1] RUN pip install --no-cache --user ".[base]" &&     pip install --no-cache --user "./airflow-plugin[acryl-datahub-airflow-plugin]" &&     pip install --no-cache --user ".[a  16.7s
------
 > [full-install 1/1] RUN pip install --no-cache --user ".[base]" &&     pip install --no-cache --user "./airflow-plugin[acryl-datahub-airflow-plugin]" &&     pip install --no-cache --user ".[all]":
0.928 Processing /datahub-ingestion
0.932   Installing build dependencies: started
15.26   Installing build dependencies: finished with status 'done'
15.27   Getting requirements to build wheel: started
15.52   Getting requirements to build wheel: finished with status 'error'
15.53   error: subprocess-exited-with-error
15.53   
15.53   × Getting requirements to build wheel did not run successfully.
15.53   │ exit code: 1
15.53   ╰─> [1 lines of output]
15.53       error in acryl-datahub setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.
15.53       [end of output]
15.53   
15.53   note: This error originates from a subprocess, and is likely not a problem with pip.
15.53 error: subprocess-exited-with-error
15.53 
15.53 × Getting requirements to build wheel did not run successfully.
15.53 │ exit code: 1
15.53 ╰─> See above for output.
15.53 
15.53 note: This error originates from a subprocess, and is likely not a problem with pip.
16.52 
16.52 [notice] A new release of pip is available: 23.3.1 -> 23.3.2
16.52 [notice] To update, run: pip install --upgrade pip
------
ERROR: failed to solve: executor failed running [/bin/sh -c pip install --no-cache --user ".[base]" &&     pip install --no-cache --user "./airflow-plugin[acryl-datahub-airflow-plugin]" &&     pip install --no-cache --user ".[all]"]: exit code: 1

I believe our main supported python version is 3.10, but not sure if that's the problem you're having here. @gray-shoe-75895 do you know if this is still a problem people can run into and how to get around it?

It's the same with python version 3.10

With the docker/datahub-ingestion/Dockerfile dockerfile, you'll need to set the RELEASE_VERSION build arg

Hi @gray-shoe-75895 I was able to create a docker image for datahub-ingestion. I am trying to use this docker image in my k8s setup container to ingest but running into this error below

Traceback (most recent call last):
  File "/datahub-ingestion/.local/bin/datahub", line 5, in <module>
    from datahub.entrypoints import main
ModuleNotFoundError: No module named 'datahub'