A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Is there anybody who has a custom docker image for metadata-ingestion. Since, I need to be able to point to the recipe files in different locations rather than the current WorkingDir, I cant use the out of the box container/image. If I have to build custom docker image, can I just use a python3.6+ base image and then include the following (assuming other dependencies are ok)  :
```RUN python3 -m pip install --upgrade pip wheel setuptools
RUN python3 -m pip install --upgrade acryl-datahub
RUN datahub version```

Hi <@U01Q3UZBD38> is it possible to address these requirements by making any changes to the existing image? Would like to understand if we can support your requirements. 

Thanks <@UV0M2EB8Q>, what i understand from the documentation is that currently, if I use the docker image for ingestion, my recipe my be on the working dir. However, this causes a limitation to us. We wanna run that image in K8 (argo workflow) and basically I just want to be able to point to a recipe in a volume/artifact location.

<@U01Q3UZBD38>: so if you were able to mount that k8s volume into the ingestion docker container, things should work?

e.g. let's say you mounted it under /recipes/

and changed the entrypoint command to be : `datahub ingest -c /recipes/&lt;your_recipe_file&gt;` this should work?

Yeah this shouldn't be that hard to accomplish using the current container

Let me follow that approach and report back

Hi <@UV0M2EB8Q> finally managed to at least call the ingest command, however, I ended up with this huge error dump. here is a snipet of it:
```er.RestRestLiServer.handleResourceRequest(RestRestLiServer.java:173)\n\t... 62 more\n", 'message': "No root resource defined for path '/datasets'", 'status': 404}
gama-datahub-dlkdk: Source report:
gama-datahub-dlkdk: {'failures': {},
gama-datahub-dlkdk:  'filtered': ['cron', 'information_schema', 'tiger', 'topology'],
gama-datahub-dlkdk:  'tables_scanned': 164,
gama-datahub-dlkdk:  'warnings': {'gama-pg-db.ethiopia.etnahil1': ['unable to map type Raster() to metadata schema'],
gama-datahub-dlkdk:               'gama-pg-db.ethiopia.etnarnf3': ["unable to map type Geometry(geometry_type='POINT', srid=4326, from_text='ST_GeomFromEWKT', "
gama-datahub-dlkdk:                                                "name='geometry') to metadata schema"],```

I used directly the linkedin/datahub-ingestion image. Also I wonder why the spatial columns are throwing that error, i checked for the plugins and saw that both postgres and sqlalchemy are included

Do we need something else ontop of these:

```Sources:
athena
bigquery
dbt
druid
file
glue
hive
kafka
ldap
mongodb
mssql
mysql
oracle
postgres
snowflake
sqlalchemy

Sinks:
console
datahub-kafka
datahub-rest
file```