I’m facing some issues with the Node Limit exceeded. I noticed a few things: •

PublishedDatasourcesConnection

&

CustomSQLTablesConnection

doesn’t have

page_size

implemented like workbook. https://github.com/datahub-project/datahub/blob/7e15947a372f6f627f29f5a1c783383d49[…]daf6/metadata-ingestion/src/datahub/ingestion/source/tableau.py • The workbooksConnection is little complex ( i have some complex workbook and even with

page_size

=1, it still exceed the node limit), I think we can refactor the

EmbeddedDatasourcesConnection

to a seperate call like

PublishedDatasourcesConnection

(at least it seems to help with my issue, although i still have some issue that i haven’t worked out yet). https://github.com/datahub-project/datahub/blob/7e15947a372f6f627f29f5a1c783383d49[…]tadata-ingestion/src/datahub/ingestion/source/tableau_common.py

For point 1, page_size was accidently removed in this commit. I have already created the PR that'll restore it.

@hundreds-photographer-13496 also I noticed some tables in lineage are missing because

emit_upstream_tables

was called too early. It should be called after all the

emit_published_datasources

and

emit_custom_sql_datasources

because those 2 functions can still populate the

self.upstream_tables

https://github.com/datahub-project/datahub/blob/7e15947a372f6f627f29f5a1c783383d49[…]daf6/metadata-ingestion/src/datahub/ingestion/source/tableau.py

you are right. updated above PR to fix for upstream_tables

@hundreds-photographer-13496 thanks for the PR and the refactoring embedded datasource out of the main graphql. i tried that and it works much better. We can see a lot more workbook now because in the earlier version, the process failed half way with 20,000 Node limit so the other pages couldn’t be reached as

has_next_page

is null in that payload.

Awesome 🙂

@hundreds-photographer-13496 i put up a github issue https://github.com/datahub-project/datahub/issues/5854 Happy to discuss further if it doesn’t make sense