Hi everyone! I'm trying to debug a

redshift_usage

ingestion, but I'm getting hardly any logs. I tried both python sdk in airflow and cli. In airflow I only see this log, both in successful and failed ingestions:

INFO - Setting gms config

When I run the cli I also see this:

INFO     {datahub.cli.ingest_cli:91} - Starting metadata ingestion

The problem I'm facing is that the process ends because it consumes all the available ram, so I'd like to find which tables are the ones that cause these memory requirements. Is it possible that this callback isn't working as expected?

Hi @incalculable-branch-51967, It is hard to tell without any stack trace. Could you try restricting it to a single table initially using the

table_pattern

config param and setting

start_time

and

end_time

to a small window to see if the recipe needs some tuning?

Hi @lemon-engine-23512, I investigated a bit further. Looks like I'm not getting any logs because the process runs out of memory while getting workunits, so it never gets to the point in which the callback shoud be triggered, I tried with a bigger machine and it looks like the process takes roughly 12gb ram and it's been running for 4 hours now (our Redshift instance is pretty big). Also, note that I'm using an outdated version (v0.8.31). Is memory allocation improved in newer versions? Do you recommend a certain strategy to make better use of resources (such as partitioning the ingestion somehow or maybe using different time deltas)?

Hi @incalculable-branch-51967, partitioning the ingestion recipe by datasets(via patterns) should definitely help. We will definitely look into the memory consumption at some point.

great, thanks!