Hello :slightly_smiling_face:
I was trying to inge...
# troubleshootw
worried-zebra-47870
10/05/2022, 2:25 PMHello 🙂
I was trying to ingest metadata from Snowflake using the version 0.8.45.1. It took me more than five hours (instead of ~10 mins with the 0.8.44.4 version) to run and at the end I got this error:
Copy code
snowflake.connector.errors.ProgrammingError: 390114 (08001): Authentication token has expired. The user must authenticate again.e
echoing-airport-49548
10/05/2022, 4:06 PMHmm it looks like this happens if there is an idle connection
echoing-airport-49548
10/05/2022, 4:07 PM@gray-shoe-75895 can you chime in here?
echoing-airport-49548
10/05/2022, 4:07 PMg
gray-shoe-75895
10/05/2022, 5:27 PMI haven’t seen this issue before, but an idle connection seems like a likely culprit. Could you enable debug logs in the ingestion (under the advanced section in ingestion config step 4) and paste the logs here?
h
hundreds-photographer-13496
10/06/2022, 8:01 AMHi @worried-zebra-47870 From logs, it looks like table lineage snowflake query itself took 4.2 hours. Is it always taking this long on newer datahub version ?
w
worried-zebra-47870
10/06/2022, 8:51 AMOn the snowflake ui it took less than one minute to run has it always did
h
hundreds-photographer-13496
10/06/2022, 9:52 AMwhich query did you run using snowflake ui ? In 0.8.45.1, there have been changes in table lineage query as compared to 0.8.44.4.
w
worried-zebra-47870
10/06/2022, 10:00 AMI meant that I can see on the Snowflake History tab the time that a query took. The ones executed by datahub only took less than a minute to run as usual.
What changes do you refer to? When upgrading from 0.8.44.4 to 0.8.45.1 does those changes apply automatically on the next ingestion run?
h
hundreds-photographer-13496
10/06/2022, 10:03 AMyes, if datahub CLI version is updated from 0.8.44.4 to 0.8.45.1.
hundreds-photographer-13496
10/06/2022, 10:09 AMI refer to changes in query done for snowflake column level lineage . Could you run ingestion with latest version 0.8.45.1 and confirm that the queries still take less than 1 minute.
w
worried-zebra-47870
10/06/2022, 10:11 AMI'll try to do that today and get back to you. Thanks for your help on this matter
h
hundreds-photographer-13496
10/10/2022, 3:55 PMHey @worried-zebra-47870 did you get chance to try out the latest version , how long does the lineage query take ?
w
worried-zebra-47870
10/10/2022, 3:57 PMSorry about the that, I've not been able to test the latest version. I'll try to test tomorrow morning
worried-zebra-47870
10/11/2022, 8:57 AMI've launched the test with the 0.8.45.2 version of the CLI. I'll let you know how it goes
worried-zebra-47870
10/11/2022, 10:12 AMIt's been one hour and my logs are stuck here:
Copy code
[2022-10-11 08:53:02,763] INFO {datahub.ingestion.source.snowflake.snowflake_v2:982} - Checking current version
[2022-10-11 08:53:02,821] INFO {datahub.ingestion.source.snowflake.snowflake_v2:988} - Checking current role
[2022-10-11 08:53:02,870] INFO [2022-10-11 08:53:00,264] INFO {datahub.cli.ingest_cli:182} - DataHub CLI version: 0.8.45.2
[2022-10-11 08:53:00,293] INFO {datahub.ingestion.run.pipeline:175} - Sink configured successfully.
[2022-10-11 08:53:02,089] INFO {datahub.ingestion.source.sql.sql_common:279} - Applying table_pattern {'deny': ['^.*_sdc.*$', '^.*_airbyte.*$', '^.*_loader_snow.*$', '^.*_dbt_tmp.*$']} to view_pattern.
[2022-10-11 08:53:02,090] INFO {datahub.ingestion.source_config.sql.snowflake:231} - using authenticator type 'DEFAULT_AUTHENTICATOR'
[2022-10-11 08:53:02,101] INFO {datahub.ingestion.run.pipeline:200} - Source configured successfully.
/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py:211: DeprecationWarning: There is no current event loop
loop = asyncio.get_event_loop()
[2022-10-11 08:53:02,103] INFO {datahub.cli.ingest_cli:129} - Starting metadata ingestion
No ~/.datahubenv file found, generating one for you...
[2022-10-11 08:53:02,763] INFO {datahub.ingestion.source.snowflake.snowflake_v2:982} - Checking current version
[2022-10-11 08:53:02,821] INFO {datahub.ingestion.source.snowflake.snowflake_v2:988} - Checking current role
[2022-10-11 08:53:02,870] INFO {datahub.ingestion.source.snowflake.snowflake_v2:994} - Checking current warehouse{datahub.ingestion.source.snowflake.snowflake_v2:994} - Checking current warehouseworried-zebra-47870
10/11/2022, 10:13 AMI have checked and on the Snowflake side the queries worked perfectly
h
hundreds-photographer-13496
10/11/2022, 10:36 AMProbably some query still running ?
This line from earlier ingestion log, gives the impression that table lineage query is taking very long time -and that could also explain why ingestion logs are stuck even now -
'table_lineage_query_secs': '15297.268323742002',w
worried-zebra-47870
10/11/2022, 10:37 AMI believe they should appear in Snowflake.
My container CPU is at 100% is it possible that datahub is processing data and takes a long time?
h
hundreds-photographer-13496
10/11/2022, 10:39 AMgot it. That makes sense
w
worried-zebra-47870
10/11/2022, 12:34 PMI'm actually running the ingestion through a separate container not from the datahub UI, how can I setup the debug mode? I'll try the options you gave me for the connexion.
I'm not sure to understand what you meant about column lineage feature on snowflake..
h
hundreds-photographer-13496
10/11/2022, 12:38 PMIf you are running ingestion using datahub cli, you can enable debug logs using command:
datahub --debug ingest -c recipe.ymlw
worried-zebra-47870
10/11/2022, 12:40 PMI'm adding it right now, I'll let you know what comes out of it 🙂
worried-zebra-47870
10/12/2022, 12:57 PMAs you said it appears it is the column lineage that is taking a long time. I didnt try the option to keep the connexion alive, I'll try it by monday and come back to you.
Thanks for your time it was really nice of you
h
hundreds-photographer-13496
10/21/2022, 12:30 PMJust checking in if you were able to get successful snowflake ingestion run with new cli version.
14 Views