Question, Is it possible to write the unique identifier of the airbyte sync run to the datawarehouse?

does anyone have experience setting up a hubspot connector? I am trying to pull the

marketing_emails

object, however it seems to be limited at 250 records (we have a total of 720 in hubspot). I am assuming this is because of some sort of rate limit through the API. Is there a way to backfill all data if the only way around this is through an incremental load setup? For further context, I was able to get all records via the API using python directly using pagination. I am just not sure how to configure this via the airbyte UI. Looking at the source code, it looks like there is a 250 record limit setup for each pull. This is definitely a nooby question, but how do you get around the pagination limit here?

hi 👋 I’ve setup a connection from Postgres to Amazon Redshift, the Redshift has a mechanism to Improve Query Performance and Optimize Storage with Compression Encoding It observed

An analysis of the cluster’s workload and database schema identified columns that will significantly benefit from using a different compression encoding.

All suggested tables are those configured with Postgres CDC (Deduped History) And its suggestion

ALTER TABLE "public"."tatable_1_scd" ALTER COLUMN "_airbyte_unique_key_scd" ENCODE lzo;
ALTER TABLE "public"."table_2_scd" ALTER COLUMN "_airbyte_unique_key_scd" ENCODE lzo;
ALTER TABLE "public"."table_3_scd" ALTER COLUMN "_airbyte_unique_key_scd" ENCODE lzo;
ALTER TABLE "public"."tatable_1_scd" ALTER COLUMN "_airbyte_emitted_at" ENCODE az64;
ALTER TABLE "public"."table_3_scd" ALTER COLUMN "_airbyte_emitted_at" ENCODE az64;
ALTER TABLE "public"."table_4_scd" ALTER COLUMN "_airbyte_emitted_at" ENCODE az64;

Is it a relevant suggestion? Would it break the airbyte sync logic if encoding updated? Thanks

Hi all, We are currently building a system that needs a data warehouse. We are exploring on the possible open source options. We are planning to use Airbyte for ETL/ELT. We have narrowed down to HDFS but soon found out that Airbyte has no direct support for the same, referring to the destinations list provided by Airbyte - https://airbyte.com/connectors?connector-type=Destinations . Is my understanding correct that there is no support for HDFS as of now ? I would like to seek advise on following, for the research we are doing - 1. Irrespective of the Airbyte support what are some of the open source techs we should look forward to for data warehousing 2. Suggestion on some of the open source techs we could use for data warehousing from the ones that are currently supported by Airbyte

Hi. Has anyone had success with key-pair authentication for Snowflake destination on OSS? I'm unsure of the issue just yet and I've tried a handful of variations on the keys and jdbc params. In any case it's throwing the following trace:

Could not connect with provided configuration. net.snowflake.client.jdbc.SnowflakeSQLLoggedException: Private key provided is invalid or not supported: rsa_key.p8: Cannot invoke "net.snowflake.client.jdbc.internal.org.bouncycastle.util.io.pem.PemObject.getContent()" because the return value of "net.snowflake.client.jdbc.internal.org.bouncycastle.util.io.pem.PemReader.readPemObject()" is null

<!subteam^S0250R4S8RY|@support> 🤠 https://discuss.airbyte.io/t/error-while-syncing-the-clickhouse-to-google-bigquery/2663 #clickhouse -> #bigquery #sync_issue

Hello folks, anyone having trouble with airbyte running with bigquery ? Since yesterday night on incremental job I get the below error:

500 An internal error occurred and the request could not be completed. This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem: <https://cloud.google.com/bigquery/sla>. If the error continues to occur please contact support at <https://cloud.google.com/support>. Error: 5423415

Google status page is not reporting any issue regarding Bigquery service 🤔 any idea ?