Airbyte #ask-community-for-troubleshooting

James Salmon

03/31/2023, 10:36 PM

I all, I am trying to set up a connection from a postgres database as an S3 load to Redshift on 0.42.0. I get the tables created in the Redshift schema, but no data gets loaded. There are no errors in the Airbyte sync logs, so I guess it is something to do with the COPY process from S3 to Redshift? I can see files get created in S3 just fine as well. Are there specific configurations needed for Airbyte to allow the user to run this command? I have given the user S3 and Redshift access, and still no luck.

Shreepad Khandve

03/31/2023, 11:59 PM

Hi team , is this really bug or we are doing something wrong while data sync ? https://github.com/airbytehq/airbyte/issues/16641 cursor = column_names[self.cursor_field[0]][0] 2023-03-31 122750 normalization > KeyError: 'since_updated_at'

Sam Stoelinga

04/01/2023, 1:58 AM

Any python destination connector examples that have integration tests that use actual sources e.g. postgres to ensure source > destination works? I want to write an integration test case for Weaviate destination, that loads data into Postgres and then use the destination to write to Weaviate. What I'm doing now is creating AirbyteCatalogs, AirbyteMessages to do integration testing, however I rather create source database and use those for integration tests.

Scott Chua

04/01/2023, 1:59 AM

Do any connectors allow incremental sync by calendar month? Hi folks, I’m working on this issue for the Facebook Marketing connector. The ask is to support Facebook Marketing’s special time increments

monthly

and

all_days

, which return aggregates by calendar month and all time, respectively. Updating the API connection in

spec

is straightforward enough, but the tricky part is updating

stream_slices

. The cursor seems to rely on Pendulum’s

range

function all the way down, and

range

doesn’t allow iterating over calendar months. Wondering if other connectors have successfully implemented incremental sync by calendar months, or otherwise any thoughts on this issue! Much appreciated 🙏

Sam Stoelinga

04/01/2023, 2:14 AM

How to delete a custom connector? Seems this indicates I would have to go into the database and manually delete some entries? Any copy pasteable commands on what to do delete exactly?

Anupam Kumar

04/01/2023, 5:31 AM

Hi everyone, I'm running airbyte locally using docker trying to test a few connections, I want to have google ads as source and redshift as destination, when I test the source and destination individually they come out as successful but while creating the connection from google ads > redshift it fails for some reason not sure why, who would be the right person with whom I can share the logs and maybe get this fixed?

Pankaj

04/01/2023, 5:14 PM

Does airbyte supports change-data-capture where source is RDS Postgres ReadReplica?

Rishikesh Srinivas

04/02/2023, 12:15 PM

Can anyone help me out for this?

vaibhav rai

04/03/2023, 5:55 AM

We are using the open-source Airbyte & have hosted it on and EC2 instance. Whenever we restart the EC2 instance, the Airbyte Source versions are being downgraded. Any idea why this could be hapenning? Example: We upgraded FB marketing to the latest version, restarted the EC2 instance 2 days letter, the FB Marketing source version was downgraded automatically.

Theo Sjöstedt

04/03/2023, 8:16 AM

Hi! New user here! 🙂 First sync I ever run in Airbyte Cloud, a Pipedrive to BigQuery connection. The spinner is just spinning, I cant see what went wrong. What can I do?

Abdullah Alhabshan

04/03/2023, 8:21 AM

Hello, I want create connection oracle DB by JDBC I chose Oracle as source but didn’t work, how I do it?

Sergei Kapochkin

04/03/2023, 9:28 AM

Hi! What’s is the most common and safe way to pass environment variables to DBT container by this UI

Simon Thelin

04/03/2023, 9:32 AM

Is there any way to trigger a manual sync via

octavia

or the

airbyte api

? I can’t seem to find it in the docs. Is this what I should use? https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/sync When I try this I get

Copy code

<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

Is there a requirement on

airbyte

version for this to work?

Anusha

04/03/2023, 9:53 AM

Hi! While cloning the Airbyte Git Repo to the local system, unable to find the run-ab-platform.sh And most of the files & folders are missing while cloning. Any help is appreciated.

Sanjeev

04/03/2023, 11:18 AM

Hi, I am trying to set a pipeline between Mysql and Redshift

Copy code

2023-04-03 10:31:22  [32mINFO [m i.a.w.p.DockerProcessFactory(create):175 - Preparing command: docker run --rm --init -i -w /data/4/0/normalize --log-driver none --name normalization-redshift-normalize-4-0-shzvg --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e AIRBYTE_VERSION=0.42.1 airbyte/normalization-redshift:0.4.0 run --integration-type redshift --config destination_config.json --catalog destination_catalog.json
2023-04-03 10:31:23  [32mINFO [m i.a.w.n.DefaultNormalizationRunner(close):194 - Terminating normalization process...
2023-04-03 10:31:23  [42mnormalization [0m > [FATAL tini (7)] exec /airbyte/entrypoint.sh failed: Exec format error
2023-04-03 10:31:23  [1;31mERROR [m i.a.w.g.DefaultNormalizationWorker(run):86 - Normalization failed for job 4.

There is an error I am facing when airbyte tries to do the normalization. I am running on ARM64 if that helps

Jana Kniel

04/03/2023, 12:08 PM

Hey everybody, I tried to connect LinkedIn Pages, but everytime I want to set up the connection to big query it says "unkown error". Does somebody have the same problem and maybe a solution?

Algis Setkus

04/03/2023, 1:26 PM

Hello All, I am new to the channel and Airbyte but very excited to use the tool! I set up Airbyte open source on a VM and trying to connect to a mySQL instance on AWS. The service requires us to whitelist IP addresses. Can anyone tell me which IP address I need to include?

Pablo Morales

04/03/2023, 1:54 PM

Hi Everyone, I opened this Pinterest Issue on Github because I need help to get ROAS data from Pinterest. I get the correct spend but not the correct revenue. Could anyone help me with this? Thanks all!

Vrushali Ghodake

04/03/2023, 2:19 PM

Is it possible to set the priority for syncing streams in ? For example, if I have two streams, can I configure to sync the second stream only after finishing the first stream? If yes, how can I implement this in ?

Dharshan Viswanathan

04/03/2023, 2:37 PM

does anyone use postgres cdc ?

Justen Walker

04/03/2023, 3:14 PM

From the Metrics Reporter -- What is the difference between Running, Pending, and Orphaned jobs? Is there a specific metric I should look at when deciding if I should scale up the number of Worker pods?

Dennis Setiawan

04/03/2023, 3:38 PM

up for visibility postgres-cdc https://airbytehq.slack.com/archives/C021JANJ6TY/p1671837396118439

subhani shaik

04/03/2023, 3:52 PM

Hey Airbyte Team, How we can create a catalogId and Schema??? Any API's for that ???

Asafe Almeida De Oliveira

04/03/2023, 5:53 PM

Hey guys I have a problem configuring Source for Zendesk Chat. Failed to fetch schema. Please try again Error: Internal Server Error: For input string: "1000000000000000000000000000000000" I noticed that there is a fix https://github.com/airbytehq/airbyte/pull/24190 Could anyone help me with this?

👀 2

[DEPRECATED] Marcos Marx

04/03/2023, 7:11 PM

@kapa.ai Hello, I want create connection oracle DB by JDBC I chose Oracle as source but didn’t work, how I do it?

Marcos Marx (Airbyte)

04/03/2023, 7:17 PM

last year when i tried postgres to postgres cdc (logical replication ) it didn’t work well. Does anyone use postgres logical replication now in prod environment ?

Vivek PG

04/03/2023, 8:29 PM

Hi #C021JANJ6TY Im trying to sync data from JIRA to POSTGRES getting this error..

Copy code

2023-04-03 14:30:50 INFO i.a.w.g.DefaultNormalizationWorker(run):97 - Normalization executed in 1 minute 52 seconds for job 74.
2023-04-03 14:30:50 INFO i.a.w.g.DefaultNormalizationWorker(run):109 - Normalization summary: io.airbyte.config.NormalizationSummary@68451c53[startTime=1680532138900,endTime=1680532250986,failures=[io.airbyte.config.FailureReason@10f977c2[failureOrigin=normalization,failureType=system_error,internalMessage=syntax error at or near "user"
LINE 94:                 drop column user,,externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@7c6f746a[additionalProperties={attemptNumber=0, jobId=74, from_trace_message=true}],stacktrace=AirbyteDbtError: 
59 of 366 ERROR creating incremental model public.issues................................................................ [ERROR in 1.21s]
Database Error in model issues (models/generated/airbyte_incremental/public/issues.sql)
  syntax error at or near "user"
  LINE 94:                 drop column user,
                                       ^

Anyone knows what the issue is? im not able to find this file too "models/generated/airbyte_incremental/public/issues.sql"

Lucas Wiley

04/03/2023, 8:32 PM

What's the best way to delete old logs on the minio? We're provisioned on kubernetes and it seems something about the image settings is preventing me from manually deleting logs inside the box. I believe log deletion is automatic ~30 days. I am planning to update

TEMPORAL_HISTORY_RETENTION_IN_DAYS

in the

.env

but unclear to me if this will impact the minio...

Luis Gomez

04/03/2023, 9:16 PM

Is there any way to deploy Airbyte and keep the current versions of the installed connectors? We have experienced issues when we upgrade Airbyte to a new version and existing connectors are upgraded as well.

Fred Manley

04/04/2023, 1:18 AM

Is there any plan to allow ssl in airbyte open source? Seems like it's been a "problem" for almost 2 years. I don't fault you guys for creating pointless overhead for the open sourced product to drive cloud payments -- you gotta eat -- but it's a really strange business model to then not allow custom connectors in the cloud service. I'd love to pay you but I literally can't do anything meaningful without custom connectors so instead I'm spending a bunch of extra money (my time and Azure resources) making insecure connections "safe." Maybe you guys should have another offering like Plural.sh -- again, not sure what the business model is where the only people making money off your product are customers and re-implementors.

👍 1