https://linen.dev logo
Join Slack
Powered by
# advice-data-transformation
  • a

    Antonia Villa

    08/22/2022, 8:06 PM
    Cross posting here as I am struggling to understand why dates are not being transformed correctly in the salesforce connector
    • 1
    • 1
  • m

    Marielby Soares

    08/23/2022, 10:09 AM
    Hi Team! Is there any update regarding running custom DBT in kubernetes?
    m
    • 2
    • 1
  • v

    Vadym Samsonovych

    08/23/2022, 3:45 PM
    Hi, could anyone help with my problem? I configured two transform models in DBT and connected it to GitHub, but when syncing the custom transform configured in Airbyte, there was a problem without exact description in the logs. Please I do need help
    s
    • 2
    • 1
  • a

    Anatole Callies

    08/29/2022, 12:26 PM
    Hey everyone, I am curious to know what IDE you are using to develop your DBT projects. DBT positioning being quite unique, considering SQL as a full fledge programming language, I haven’t found the perfect IDE yet. Coming from the Python ecosystem, I am used to Jetbrain’s Pycharm so I am currently editing my DBT projects using the DataGrip IDE, but it’s not perfect as it fails to reformat my longest models + it is not free.
    c
    • 2
    • 2
  • d

    Dusty Shapiro

    08/29/2022, 2:46 PM
    Question: Our Source database (Postgres) is converting date/timestamps to strings during basic normalization. How do we make sure this doesn’t happen?
  • p

    Pavan Charan Dharmavaram Hari Rao

    08/29/2022, 6:21 PM
    Question: i am facing issue with converting CSV to postgres, all numbers are getting created as Float data type
  • p

    Pavan Charan Dharmavaram Hari Rao

    08/29/2022, 6:22 PM
    any pointers how to control schema
  • i

    Isabella Peraertz

    08/30/2022, 7:09 PM
    Hi! Has anyone else had the issue when integrating with the Google Ads API that the Ad ID’s do not match the format for the Ad ID’s in the Google Ads Platform at all ? How did you resolve that issue?
  • p

    Pavan Charan Dharmavaram Hari Rao

    08/31/2022, 6:49 PM
    Hi Team, any body know answer this problem
  • p

    Pavan Charan Dharmavaram Hari Rao

    08/31/2022, 6:49 PM
    https://discuss.airbyte.io/t/csv-file-to-postgres-connection-number-formats-saving-as-float-data-type/2426
    m
    • 2
    • 5
  • p

    Pavan Charan Dharmavaram Hari Rao

    08/31/2022, 6:49 PM
    i am surprised even if we create table it is dropping again and making float at destination
  • m

    Marlon Gómez

    09/02/2022, 3:16 PM
    Hello everybody. Does anyone know what the current state is for the goland SDK? A few months ago I was excited with the announcement but then I had to pause my project with Airbyte. Any connectors with this SDK? good reviews so far?
  • m

    Matias Ordoñez

    09/02/2022, 11:58 PM
    Hello , im having a problem , when i try to sync two tables , origin in mongodb dest in mysql a simple field with numbers in format : 2.15 are saved as strings and the column name is automatically changed to $name_airbyte_transform and shows all null values on the destination why is this happening?
  • p

    Pavan Charan Dharmavaram Hari Rao

    09/07/2022, 4:17 PM
    is there way to integrate spark with Airbyte for transformations instead of dbt
  • p

    Pavan Charan Dharmavaram Hari Rao

    09/07/2022, 4:17 PM
    dbt is not approved product then difficult to introduce two tools in org at a time
  • s

    Saymon_says1985

    09/08/2022, 10:23 AM
    Hi, is there a way for me to make Airbyte run basic normalization using local installation of dbt or a different docker container of dbt? Reason is we need custom plugin property within dbt for it to work correctly with our database.
  • s

    Slackbot

    09/09/2022, 2:18 PM
    This message was deleted.
  • m

    Marcos Marx (Airbyte)

    09/15/2022, 5:36 PM
    If the destination supports dbt yes
  • s

    Slackbot

    09/16/2022, 3:18 PM
    This message was deleted.
  • p

    Patrick Leahey

    09/19/2022, 12:53 PM
    How can I change the number DBT threads from the default 4 that basic normalization uses?
    e
    • 2
    • 1
  • j

    Jonathan Crawford

    09/21/2022, 6:55 PM
    Hey gang! I’ve been working on an initial import for a week & a half and haven’t been able to get it to complete. This process dies at the very end of the DBT normalization step while the Postgres target database is doing hours of vacuuming. It eventually results in a timeout. Here’s a screenshot of the output. The table it’s attempting to normalize is 68GB at the source. Any guidance or experience here? Thanks in advance!
  • g

    god830

    09/23/2022, 4:18 PM
    https://github.com/airbytehq/airbyte/pull/16629 one more review please, this pr saves my life, thanks to krisjan
    m
    s
    • 3
    • 7
  • v

    Vijay

    09/29/2022, 10:15 PM
    Hi All, this should be quick, can you tell us why there seems to be no activity between these two log messages?
  • v

    Vijay

    09/29/2022, 10:15 PM
    2022-09-29 213032 normalization > Found 212 models, 0 tests, 0 snapshots, 0 analyses, 558 macros, 0 operations, 0 seed files, 6 sources, 0 exposures, 0 metrics 2022-09-29 214616 normalization > Concurrency: 8 threads (target='prod') 2022-09-29 214628 normalization > 1 of 53 START incremental model public.ip5pxydtrvafvsh_ads.............................................................. [RUN] 2022-09-29 214630 normalization > 2 of 53 START incremental model public.ip5pxydtrvafvsh_campaigns....
  • v

    Vijay

    09/29/2022, 10:15 PM
    It waited for 15 minutes to start normalization
  • v

    Vijay

    09/29/2022, 10:19 PM
    Also, is it normal for incremental models to take 10-20 seconds even when there is no insertions and we have less than 1MB of data?
  • v

    Vijay

    09/29/2022, 10:19 PM
    2022-09-29 221436 normalization > 24 of 53 OK created incremental model public.ip5pxydtrvafvsh_ad_c__rl_spec_windows_phone................................ [INSERT 0 0 in 21.98s] 2022-09-29 221438 normalization > 25 of 53 OK created incremental model public.ip5pxydtrvafvsh_ad_c__template_url_spec_web................................ [INSERT 0 0 in 22.99s] 2022-09-29 221438 normalization > 26 of 53 OK created incremental model public.ip5pxydtrvafvsh_ad_c__template_url_spec_ios................................ [INSERT 0 0 in 21.01s] 2022-09-29 221439 normalization > 27 of 53 OK created incremental model public.ip5pxydtrvafvsh_ad_c___story_spec_link_data................................ [INSERT 0 38 in 22.31s] 2022-09-29 221446 normalization > 28 of 53 OK created incremental model public.ip5pxydtrvafvsh_ad_c__ry_spec_template_data................................ [INSERT 0 0 in 16.61s] 2022-09-29 221448 normalization > 29 of 53 OK created incremental model public.ip5pxydtrvafvsh_ad_c___story_spec_text_data................................ [INSERT 0 0 in 15.79s] 2022-0
  • v

    Vijay

    09/29/2022, 10:25 PM
    https://github.com/dbt-labs/docs.getdbt.com/issues/335
  • v

    Vijay

    09/29/2022, 10:25 PM
    Does postgres not work well with incremental models from DBT?
  • o

    Olivier AGUDO PEREZ

    09/30/2022, 1:10 PM
    Hey, I would like to add a custom dbt transformation for my replication to Bigquery. I also need to specify custom destination dataset in Bigquery (default one writes to
    airbyte_raw
    ). To do that I had to put my file
    profiles.yml
    into my dbt project github repo in order to specify my dataset. Following Airbyte doc, I have now to specify the
    profiles.yml
    path into Airbyte GUI
    Entrypoint arguments for dbt cli
    : that field is as follow
    run --profiles-dir .
    And this is where problems begin. When I run my Airbyte sync, I have the following error during dbt phase :
    Could not find profile named 'normalize'
    Problem is my profile name is not "normalize", neither in
    profiles.yml
    or in
    dbt_project.yml
    . What is the way to specify that
    profiles.yml
    path and why is it trying to use normalize project ? Also when I try to run the sync + transformation without specifying custom dataset, it works great.
    c
    • 2
    • 8