Hey team, I’m posting here just to check if I’m doing the setup correctly. I have a connection *Postgres <> Redshift, 1 table, 100k record*s. The extraction took 11 minutes on the first run, the subsequent ran took 16 minutes (0 records). It seems Airbyte is having a hard time during the DBT phase. Do you have any performance tips? Deployment: k8s Version: 0.40.15 Worker(ram/cpu limits): 2Gi/2500m Worker replicas: 3 Using S3 as staging for redshift COPY

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2981 to ensure timely resolution!

Hey @Jhonatas Kleinkauff, have you read through this article? https://docs.airbyte.com/operator-guides/scaling-airbyte/ May provide some insight on how to configure Airbyte efficiently for the number of records you’re trying to sync.

Hey, thank you so much for your answer! I’ve tweaked my worker (6Gi/4500m), it seems Airbyte is cleverer (?), but it took almost 10 minutes to start running the first dbt model. Do you think if it could be related to my resources still? (this execution have 0 records, the total time was 15 minutes…)

@Jhonatas Kleinkauff, I think you may need to increase the number of threads. The example in the docs uses 32 threads whereas you’re using 4 threads. I believe this can be set in the profiles.yml

Thank you for your answer. Where can i change it within a k8s deployment?

@Jhonatas Kleinkauff, we don't currently support custom dbt transformations on kubernetes: https://github.com/airbytehq/airbyte/issues/5091

Gotcha. According to this doc 4 is the default and recommended value for the thread parameter. - and indeed, is the value airbyte is using. But i believe this settings just tells dbt how many scripts he will run in parallel ONCE the process is started. I’m not sure how this affects the gap between the start itself: 7 minutes to start the first model 2022-10-21 170147 - Concurrency: 4 threads (target=‘prod’) 2022-10-21 170803 - 1 of 3 START view model _airbyte_airbyte.carriers_carrier_mini_original_stg............................................. [RUN]

There's more info about dbt threads here: https://docs.getdbt.com/docs/get-started/connection-profiles#understanding-threads Specifically, I think this part is relevant:

For example, if you specify threads: 1, dbt will start building only one model, and finish it, before moving onto the next. Specifying threads: 8 means that dbt will work on up to 8 models at once without violating dependencies – the actual number of models it can work on will likely be constrained by the available paths through the dependency graph.

Also perhaps this is also relevant? https://github.com/dbt-labs/dbt-core/issues/1002#issuecomment-430033278

i believe the problem could be related to dbt doing a full-parse, not sure how to get through this tho

It says the partial parsing is turned on by default, what does your profile.yml look like?

Airbyte logs:

This is an interesting problem. This is the default profile.yml for Airbyte, which sets partial parse to true: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/base-normalization/normalization/transform_config/profile_base.yml. This is how Airbyte configures the dbt config: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/base-normalization/normalization/transform_config/transform.py Maybe there is something here?

Yeah, but I do think - and it is stated in the logs - Airbyte/dbt is doing a full parse. Do you mind if I take this conversation to a github issue?

Not at all, please make an issue on Github!

Not sure if we can do a partial parse the way it is today. If dbt really depends off a manifest.json file we would need to dig a little bit..

If you made a github issue for this, could you also link it here for others to follow?

Here it is, https://github.com/airbytehq/airbyte/issues/18317