Hello.
When using the Spark Lineage in AWS, if we...
# troubleshoota
adamant-honey-44884
06/05/2023, 9:52 PMHello.
When using the Spark Lineage in AWS, if we are using glue as our Hive Metastore do we have to use the Glue source for the datasets or does using an Athena source? I have been trying to get it working with Athena but the pipeline/tasks are not connected to the Athena imported datasets.
Thanks in advance for the help.
g
gentle-hamburger-31302
06/06/2023, 4:09 AMDid you tried with Glue
gentle-hamburger-31302
06/06/2023, 4:09 AMgentle-hamburger-31302
06/06/2023, 4:09 AM Copy code
This plugin extracts the following:
Tables in the Glue catalog
Column types associated with each table
Table metadata, such as owner, description and parameters
Jobs and their component transformations, data sources, and data sinksa
adamant-honey-44884
06/06/2023, 1:07 PMHi @gentle-hamburger-31302, no. My question was if I had to use the Glue source rather than the Athena source for the link to be made.
g
gentle-hamburger-31302
06/07/2023, 11:57 AM@dazzling-judge-80093 might help you on this question
2 Views