Naphat Theerawat
01/20/2022, 4:37 AMFull refresh
on final destination table.
At the moment, Airbyte runs a full-refresh to recreate the final tables. This can cause more costs in some destinations like Snowflake, Redshift, and Bigquery. To understand better what sync mode and frequency you should select, read this doc. There is a FAQ topic on our Discourse that more extensively explains the cost issue here.
Can someone tell me if this is still true for Incremental Sync - Deduped History too?Moshe Mazuz
01/20/2022, 10:26 AMSoufiane Odf
01/20/2022, 10:09 PMJacob Mass
01/21/2022, 12:55 AMSoufiane Odf
01/21/2022, 9:27 AMSoufiane Odf
01/21/2022, 12:32 PMDaniel Eduardo Portugal Revilla
01/21/2022, 1:31 PM{
"type": "LOG",
"log": {
"level": "FATAL",
"message": "HttpStream.__init__() got an unexpected keyword argument 'auth'\nTraceback (most recent call last):\n File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/main.py\", line 13, in <module>\n launch(source, sys.argv[1:])\n File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/.env/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 127, in launch\n for message in source_entrypoint.run(parsed_args):\n File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/.env/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 118, in run\n for message in generator:\n File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/.env/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 92, in read\n stream_instances = {s.name: s for s in self.streams(config)}\n File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/source_servicesnow_api/source.py\", line 55, in streams\n return [ServicesnowApi(auth=(config[\"psswrd\"], config[\"psswrd\"]), user=config[\"user\"], psswrd=config[\"psswrd\"])]\n File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/source_servicesnow_api/source.py\", line 67, in __init__\n super().__init__(**kwargs)\nTypeError: HttpStream.__init__() got an unexpected keyword argument 'auth'"
}
}
I was trying this script
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
"""
TODO: Replace the streams below with your own streams.
:param config: A Mapping of the user input configuration as defined in the connector spec.
"""
# TODO remove the authenticator if not required.
auth = TokenAuthenticator(token="api_key") # Oauth2Authenticator is also available if you need oauth support
return [ServicesnowApi(authenticator=(config["psswrd"], config["psswrd"]))]
Justin Cole
01/21/2022, 2:59 PMshunde zhao
01/21/2022, 10:16 PMSoufiane Odf
01/22/2022, 3:36 PMOluwapelumi Adeosun
01/22/2022, 3:58 PMSteve
01/22/2022, 5:09 PMSoufiane Odf
01/23/2022, 2:52 PMFridtjof Petersen
01/23/2022, 6:27 PMElias Djurfeldt
01/24/2022, 10:39 AMEugene Krall
01/24/2022, 11:01 AMLihan
01/24/2022, 11:23 AMVictor Francess
01/24/2022, 12:09 PMSteve
01/24/2022, 1:23 PMAlex Meadows
01/24/2022, 7:32 PMAmir Davidoff
01/24/2022, 8:27 PMYiyang (Heap.io)
01/24/2022, 8:58 PMFridtjof Petersen
01/24/2022, 9:01 PMfull refresh - append
but this already broke because someone added a new column and the sync schema didn't stay the same so we need to stick with the overwrite
option.
Does anyone have a suggestion on how to properly archive/store all data with dbt? My idea was to use the snapshot
feature of dbt as it can also add new columns that don't fit with the created schema.
Thank you!Amol Walanj
01/25/2022, 3:46 AMDekel R
01/25/2022, 8:41 AMSteve
01/25/2022, 11:05 AMTech Account
01/25/2022, 8:41 PMVikram Medishetty
01/26/2022, 1:32 AMMahdir Ishmam
01/26/2022, 2:24 AMSteve
01/26/2022, 11:56 AM