Rodrigo Menezes
07/12/2021, 8:42 PM2021-07-12 20:10:45 INFO () JsonSchemaValidator(test):76 - JSON schema validation failed.
errors: $: unknown found, object expected
2021-07-12 20:10:45 ERROR () DefaultAirbyteStreamFactory(lambda$create$1):83 - Validation failed: null
Xing Fang
10/04/2021, 5:44 PMTom Gordon
12/16/2021, 6:57 PM[34msource[0m - 2021-12-16 18:38:36 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-16 18:38:36 [32mINFO[m i.a.i.s.m.MySqlSource(getIncrementalIterators):181 - {} - using CDC: false
[34msource[0m - 2021-12-16 18:38:36 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-16 18:38:36 [32mINFO[m i.a.i.s.r.AbstractRelationalDbSource(queryTableFullRefresh):35 - {} - Queueing query for table: establishments
[34msource[0m - 2021-12-16 18:43:48 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-16 18:43:48 [32mINFO[m i.a.i.s.r.AbstractDbSource(lambda$read$2):123 - {} - Closing database connection pool.
[34msource[0m - 2021-12-16 18:43:48 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-16 18:43:48 [32mINFO[m i.a.i.s.r.AbstractDbSource(lambda$read$2):125 - {} - Closed database connection pool.
[34msource[0m - 2021-12-16 18:43:48 ERROR () LineGobbler(voidCall):82 - Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
[34msource[0m - 2021-12-16 18:43:48 ERROR () LineGobbler(voidCall):82 -
[34msource[0m - 2021-12-16 18:43:48 ERROR () LineGobbler(voidCall):82 - The last packet successfully received from the server was 310,727 milliseconds ago. The last packet sent successfully to the server was 310,727 milliseconds ago.
Emah Bright
02/01/2022, 3:42 AMThe connection tests failed.
Exception('Error while refreshing access token: 401 Client Error: Unauthorized for url: <https://api.amazon.com/auth/o2/token>')
Followed the amazon seller set up, doubled checked the permissions and still seems to get this error. Is there any way to get more details from the logs? Would be great to see more details on ERROR check failed line.
Logs below:
2022-02-01 03:32:35 INFO () TemporalAttemptExecution(get):94 - Executing worker wrapper. Airbyte version: 0.30.25-alpha
2022-02-01 03:32:35 INFO () LineGobbler(voidCall):82 - Checking if airbyte/source-amazon-seller-partner:0.2.14 exists...
2022-02-01 03:32:35 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/06df2588-f803-473f-8ba3-3a443e8fea50/0 --network host --log-driver none airbyte/source-amazon-seller-partner:0.2.14 check --config source_config.json
2022-02-01 03:32:36 ERROR () DefaultAirbyteStreamFactory(internalLog):96 - Check failed
2022-02-01 03:32:36 INFO () TemporalAttemptExecution(get):115 - Stopping cancellation check scheduling...
Roy Peter
02/08/2022, 4:05 AMSerhii Сheredko
02/08/2022, 11:32 AMElliot Trabac
02/13/2022, 8:44 AMnomalization falied
error when syncing data
CC: @Arash Layeghi @ParhamRytis Zolubas
02/14/2022, 10:24 AMCould not connect with provided configuration. Error: Cannot create PoolableConnectionFactory (Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
Marc Heymann
02/14/2022, 11:23 AMAditya Chatterjee
02/14/2022, 1:20 PMAntonio Grass
02/14/2022, 1:39 PMAntonio Grass
02/14/2022, 2:25 PMJOB_MAIN_CONTAINER_MEMORY_REQUEST
and JOB_MAIN_CONTAINER_MEMORY_LIMIT
say that I had double the memory in my system in hopes that it would use more, but I never saw it use more than 10gb. The source and destination DB servers have 32gb memory each and plenty of storage. I also updated the number of workers from 5-15 and set new CPU request limits at near the maximum. I also restarted the server after making these changes.
from watching the log file as the job run it seems like the bottleneck is when it flushes the buffer. It reports that it only takes seconds but in reality it appears to take 2-3 minutes between this happening and running through the next group of record reads. I have attached a log from my recent run that I stopped at 6 minutes as all the adjustments I made did not seem to yield and changes.
Really my issue is how can I configure airbyte to better move this type of table between sql servers, when SSIS completed this in a minute and a half I didnt expect it to match but taking over an hour was far from the expected result.Peter Leiwakabessy
02/14/2022, 3:09 PMsource-mysql
Airbyte connector, with a CDC configuration. Incremental sync fails with this type of error log:
2022-02-14 14:15:11 source > 2022-02-14 14:15:11 ERROR i.d.r.TableSchemaBuilder(lambda$createValueGenerator$5):269 - Failed to properly convert data value for 'prestashopprod.ps_product.available_date' of type DATE for row [1, 0, 0, 13, 1, 97, 0, 0, [], [54, 54, 48, 48, 52, 50, 55, 49, 51, 54, 55, 52], 0.000000, 0, 1, 499.000000, 0.000000, [], 0.000000, 0.00, [67, 82, 88, 48, 48], [], [], 0.000000, 0.000000, 0.000000, 1.900000, 2, 0, 0, 0, 0, 0, 404, 0, 1, null, new, 1, 0, both, 0, 0, 0, 0, 2017-05-30T17:06:36.000+0000, 2020-04-23T10:32:53.000+0000, 0, 3]:
2022-02-14 14:15:11 source > org.apache.kafka.connect.errors.DataException: Invalid Java object for schema type STRING: class java.lang.Integer for field: "available_date"
2022-02-14 14:15:11 source > at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:245) ~[connect-api-2.6.1.jar:?]
2022-02-14 14:15:11 source > at org.apache.kafka.connect.data.Struct.put(Struct.java:216) ~[connect-api-2.6.1.jar:?]
2022-02-14 14:15:11 source > at io.debezium.relational.TableSchemaBuilder.lambda$createValueGenerator$5(TableSchemaBuilder.java:265) ~[debezium-core-1.4.2.Final.jar:1.4.2.Final]
even though, when looking at the table, the row actually seems to have okay values...
To sum up : https://github.com/airbytehq/airbyte/issues/6884 this issue that I was encountering previously before seems to be fixed, but now I’m having the cousin of this one https://github.com/airbytehq/airbyte/issues/9118... Any plans to upgrade Debezium at some point?Alexandre Chouraki
02/14/2022, 3:37 PM2022-02-14 14:53:10 source > Syncing stream: issues
2022-02-14 14:53:10 source > Read 0 records from issues stream
2022-02-14 14:53:10 source > Finished syncing SourceJira
2022-02-14 14:53:10 source > SourceJira runtimes:
This is strange, because :
• I know for a fact this data exists
• The token and user associated to my source are basically superadmins
• Other streams do return records
• going to https://XXX.atlassian.net/rest/api/3/search?jql= does yield the issues I want...Alexandre Chouraki
02/14/2022, 3:59 PMkubectl apply -k kube/overlays/stable
I get error: json: unknown field "envs"
error.
I haven’t really dived into the code just read the docs to do a quick start.. should I change something on one of the manifests?Peter
02/14/2022, 5:13 PMHouman Farokhzad
02/14/2022, 6:59 PM_airbyte_hubspot_companies_hashid STRING NULLABLE
_airbyte_ab_id STRING NULLABLE
_airbyte_emitted_at TIMESTAMP NULLABLE
_airbyte_normalized_at TIMESTAMP NULLABLE
_airbyte_properties_hashid STRING NULLABLE
Titas Skrebė
02/14/2022, 7:06 PMk get pods | grep 126044
airbyte-bigquery-sync-126044-0-bghap 0/5 Error 0 20h
source-postgres-sync-126044-0-zgnvm 0/4 Error 0 20h
but on the UI it shows running and doesn’t restart / retry it even after 20h.
any idea what’s going on. I am on 0.35.10-alphaCayden Brasher
02/15/2022, 8:08 PMgunu
02/16/2022, 2:54 AM2022-02-16 02:04:10 WARN i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):96 - Request made: ClientCustomerId: ####, Host: <http://googleads.googleapis.com|googleads.googleapis.com>, Method: /google.ads.googleads.v9.services.GoogleAdsService/Search, RequestId: XXX, IsFault: True, FaultMessage: User doesn't have permission to access customer. Note: If you're accessing a client customer, the manager's customer id must be set in the 'login-customer-id' header. See <https://developers.google.com/google-ads/api/docs/concepts/call-structure#cid>
it appears the ClientCustomerId
is not properly being set as the manager’s customer idgunu
02/16/2022, 4:14 AMDivya (Proximity)
02/16/2022, 9:17 AMInternal Server Error: Get Spec job failed.
I checked airbyte-scheduler logs and they say something like this: 2022-02-16 08:52:51 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-02-16 08:52:51 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
So maybe someone knows how i can resolve this issue or any advices?user
02/16/2022, 12:56 PMSyedHamza
02/16/2022, 3:32 PMPreet Singh
02/16/2022, 7:27 PMmessages
, calls
, message_media
) using the "Incremental | Append" sync mode cannot have its date columns (date_sent
, date_created
, date_updated
) queried. When we try to query date_created
directly, we receive a Spectrum Scan Error
. When we try date_created.member0
we receive relations "date_created" does not exist
. Moving to a "Full Refresh | Overwrite" sync mode (which does work as evidenced by some of our other tables) isn't possible because we have 4+ million records in the messages
tableNicolas Smith
02/16/2022, 9:01 PMNicolas Smith
02/16/2022, 9:18 PMMaish Maseeh
02/17/2022, 9:49 AMassociations
are specified for certain data types.
Are these associations simply missing?
But I feel more like I am doing something wrong since I imagine that anyone using the source would have a need for these links.
So I am probably simply not seeing something in the existing data.
Thanks,
JorinJorin
02/17/2022, 11:59 AMv0.32.0-alpha-patch-1
takes longer (upgrading from v0.29.4
? Doing it in GKE and its been ~30mins. CPU/Mem being utilized but not seeing anything in logs so wondering. Data is not that high . Counts from tables (configs: 230, jobs: ~8000; attempts: ~100000)Pras
02/17/2022, 2:55 PM0.32.0-alpha-patch-1 working fine
0.32.11-alpha working fine
0.34.2-alpha working fine
0.34.3-alpha cursor lost
0.34.4-alpha cursor lost
0.35.1-alpha cursor lost
0.35.18-alpha cursor lost