Is there any difference between the output produced by `glue DataHub #ingestion

Is there any difference between the output produce...

square-activity-64562

07/22/2021, 11:05 AM

Is there any difference between the output produced by

glue

and

athena

sources? We have athena tables which are managed by

glue

catalog. The ingestion plugins for athena does not support views. So I was think

glue

file

-> replace

glue

with

athena

in file ->

athena

ingestion. Will bypass needing to add support for athena views

square-activity-64562

07/22/2021, 11:23 AM

Ran these commands and it seems to have worked

Copy code

datahub ingest -c ~/glue_to_file.yml;
sed 's/:glue/:athena/g' glue-output.json > athena-input.json;
datahub ingest -c ~/glue_file_ingest.yml;

mammoth-bear-12532

07/22/2021, 2:36 PM

Cool. Thanks for letting us know. In the case where glue is managing more than Athena tables, we would have to separate out the ingestion to perform the platform swap.

mammoth-bear-12532

07/22/2021, 2:37 PM

@square-activity-64562 could you check in this case of glue managing Athena tables, whether there is some info in the properties that describes that this is backed by Athena?

square-activity-64562

07/22/2021, 2:54 PM

I know all the metadata of athena is being managed by glue. Or are you are asking for some use case?

mammoth-bear-12532

07/22/2021, 3:03 PM

Asking whether there is some information in the returned metadata from glue that describes the Athena backend. So we could perform the swap automatically.

square-activity-64562

07/23/2021, 8:36 AM

I checked and there does not seem to be anything there which suggests this is athena metadata

Open in Slack

Previous Next