Hi Team Looks like there is a miss in version 12 1 When the Elementary Community #community-support

Hi Team, Looks like there is a miss in version 12....

Shantanu Panwar

11/09/2023, 8:45 AM

Hi Team, Looks like there is a miss in version 12.1 When the run-end hook is triggered, I am running into following issue:

Copy code

001795 (42601): SQL compilation error: error line 2 at position 25
08:16:01  maximum number of expressions in a list exceeded, expected at most 16,384, got 24,002

Snowflake has a limitation of at most 16,384 expressions at a time. For reference: https://community.snowflake.com/s/article/maxi-expressions-exceeded But the insert statement into dbt_columns doesn’t handle it pretty well. Can you please fix it or let me know if I am missing something? cc: @Or Avidov @Maayan Salom

Shantanu Panwar

11/09/2023, 8:51 AM

Small logger snippet for reference:

Copy code

Elementary: Inserting 24002 rows to table "ETHOS_PRODUCTION"."DBT_CORE_TRANSFORMATION_ELEMENTARY"."DBT_COLUMNS__tmp_20231109081533313323"
2023-11-09 08:15:55.918723 (MainThread): 08:15:55  Elementary: [1/1] Running insert query.
2023-11-09 08:16:00.480962 (MainThread): 08:16:00  Using snowflake connection "master"
2023-11-09 08:16:00.516622 (MainThread): 08:16:00  On master: insert into "ETHOS_PRODUCTION"."DBT_CORE_TRANSFORMATION_ELEMENTARY"."DBT_COLUMNS__tmp_20231109081533313323"
         (METADATA_HASH) values
    ('xxxxxx509'),('xxxxxc13'),('xxxx38d')... 24002 records.

Let me know if you want complete debug logs.

Idan Shavit

11/09/2023, 9:12 AM

Hey @Shantanu Panwar, thank you for raising that up! Can you share with me the full log? It will help me debug it 🙂

Shantanu Panwar

11/09/2023, 9:32 AM

Hey @Idan Shavit, Shared the log file in personal chat.

01/24/2024, 3:19 AM

Hi, did anyone manage to resolve this issue?

Shantanu Panwar

01/24/2024, 5:50 AM

Even I am following up with Idan for the fix. No updates yet. cc: @Or Avidov What can we do to prioritize it?

01/24/2024, 5:51 AM

Have you stopped using elementary bcos of this? Seems like I'll have to remove it if i cant resolve the issue

Shantanu Panwar

01/24/2024, 5:51 AM

I have downgraded the version of elementary. It is stable on older versinos.

01/24/2024, 5:52 AM

Oh interesting, which version are you using right now? Maybe i'll downgrade as well

Shantanu Panwar

01/24/2024, 5:53 AM

One of the projects where the issue persists has the older version : 7.8 as it was stable for the longest time on our platform.

01/24/2024, 6:50 AM

I see, okay thank you for your input! will try downgrading to 7.8

Ofek Weiss

02/04/2024, 2:37 PM

Hey @Shantanu Panwar @B you can set the following vars in

dbt_project.yml

to resolve this issue:

Copy code

vars:
  insert_rows_method: chunk
  dbt_artifacts_chunk_size: 5000

you can play around with

dbt_artifacts_chunk_size

to see what works best for you, but 5000 should be a great start

🙌 2

02/05/2024, 10:55 AM

Hi Ofek, thanks for the fix. Are you able to elaborate on the issue and what this fix does? 🙂

Ofek Weiss

02/05/2024, 11:07 AM

Sure @B it instructs elementary to insert records to the artifact tables in chunks of

record each, so if, for example, there are 25,000 records to be uploaded, instead of attempting to insert all of them in a single

insert

command, it will run 5 separate

insert

commands of 5,000 records

02/05/2024, 11:08 AM

I see, thanks Ofek, this is useful. Is this only available in the latest version of elementary? I didnt see this in the latest docs

Ofek Weiss

02/05/2024, 12:36 PM

It is not new, but it does sadly seem to be missing from the docs... I'll try to add it in the near future

🙌 1

31 Views

Open in Slack

Previous Next