I hate to copy this big of a trace into the chat but suddenly the last week since upgrading our tableau ingestion has been failing... has anyone seen a similar error?

[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO - [2023-08-17, 05:06:24 PDT] ERROR    {datahub.entrypoints:199} - Command failed: 'NoneType' object has no attribute 'get'
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO - Traceback (most recent call last):
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/entrypoints.py", line 186, in main
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     sys.exit(datahub(standalone_mode=False, **kwargs))
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     return self.main(*args, **kwargs)
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     rv = self.invoke(ctx)
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     return ctx.invoke(self.callback, **ctx.params)
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     return __callback(*args, **kwargs)
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     return f(get_current_context(), *args, **kwargs)
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 448, in wrapper
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     raise e
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 397, in wrapper
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     res = func(*args, **kwargs)
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/utilities/memory_leak_detector.py", line 95, in wrapper
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     return func(ctx, *args, **kwargs)
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 198, in run
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     ret = loop.run_until_complete(run_ingestion_and_check_upgrade())
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     return future.result()
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 182, in run_ingestion_and_check_upgrade
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     ret = await ingestion_future
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 140, in run_pipeline_to_completion
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     raise e
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 132, in run_pipeline_to_completion
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     pipeline.run()
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 367, in run
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     for wu in itertools.islice(
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 119, in auto_stale_entity_removal
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     for wu in stream:
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 143, in auto_workunit_reporter
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     for wu in stream:
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 208, in auto_browse_path_v2
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     for urn, batch in _batch_workunits_by_urn(stream):
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 346, in _batch_workunits_by_urn
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     for wu in stream:
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 156, in auto_materialize_referenced_tags
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     for wu in stream:
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 70, in auto_status_aspect
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     for wu in stream:
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/tableau.py", line 2590, in get_workunits_internal
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     yield from self.emit_sheets()
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/tableau.py", line 2028, in emit_sheets
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     yield from self.emit_sheets_as_charts(
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/tableau.py", line 2107, in emit_sheets_as_charts
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     project_luid: Optional[str] = self._get_workbook_project_luid(workbook)
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -   File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/tableau.py", line 1438, in _get_workbook_project_luid
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO -     if wb.get(tableau_constant.LUID) and self.workbook_project_map.get(
[2023-08-17, 05:06:24 PDT] {{pod_manager.py:235}} INFO - AttributeError: 'NoneType' object has no attribute 'get

I don't think the issue is with

workbook_project_map

(its well initialized to a non-null object) but rather

wb

is being passed as null. Somehow the sheets graphql query must've returned null for workbook field. do you have any warnings in the ingestion report from tableau api ? Does reducing

page_size

config help ? Nonetheless, this issue should be handled well in connector. Do you mind creating a github issue for this ? Dev note: this also highlights that we need to add python type annotations to improve mypy coverage in tableau connector to catch such issues earlier.