Hi everyone, I’ve been messing about with the postgres ingest and so far so good! One question I have: we have 100+ databases on one postgres cluster. Is it possible to prefix the database to the WorkUnit? For example, looking at the datasets I have dozens of tables in the ‘public’ schema - but they’re actually in several different databases.

Hey Matt - clarifying question: are you not seeing the database names at all, only schema + table names?

that’s correct, just the schema.table.

got it - just opened this PR to fix https://github.com/linkedin/datahub/pull/2401

oh perfect! thanks (ps, screenshot of what I see)

when you run ingestion, do you specify the "database" option in the config?

@broad-flag-97458: I merged in @gray-shoe-75895’s PR, so you can try it now

Hi @gray-shoe-75895 I do specify the database option, e.g.

source:
  type: postgres
  config:
    username: user
    password: password
    host_port: dbs1db02:5432
    database: retail-content-service

got it - do you run

datahub ingest

once per database?

Well, that’s TBD honestly (but I’m open to suggestions/better ways)

@broad-flag-97458: we do push docker images from our github actions pipeline and I see the latest one was published 23 mins ago. Could you re-pull

docker pull linkedin/datahub-ingestion:latest

?

Hmm, I think I’ve been going about this in the wrong way… I just did a

docker run -t --rm --network datahub_network -v /home/matt/datahub/recipes/:/recipes linkedin/datahub-ingestion:latest ingest -c /recipes/postgres.yaml

and I can see the database prefixed to the schema.table. Thanks @mammoth-bear-12532