Parth Gangrade
03/31/2022, 1:14 PMWill Massey
03/31/2022, 1:32 PMTotal records read: 1
.
I have also tried manually connecting to the API endpoint (https://api.hubapi.com/crm-pipelines/v1/pipelines/deals) using the requests library in Python and the response includes all the details. I’m not sure what is going on and what the issue is, but perhaps it is a problem with how the schema is generated.
Any help would be much appreciated!Marcos Marx (Airbyte)
03/31/2022, 3:00 PMColton Pomeroy
03/31/2022, 9:14 PMWill Moore
03/31/2022, 9:45 PMWill Moore
03/31/2022, 10:50 PMEmily Cogsdill
03/31/2022, 10:54 PMThiago Costa
04/01/2022, 3:59 AMDejan Antonic
04/01/2022, 9:21 AMTrev Killick
04/01/2022, 11:41 AMerrors: $.client_id: is missing but it is required, $.client_secret: is missing but it is required, $.refresh_token: is missing but it is required, $.credentials_title: must be a constant value OAuth Credentials, $.credentials_title: does not have a value in the enumeration [OAuth Credentials]
Sharal Pinto
04/01/2022, 3:47 PMWill Moore
04/01/2022, 4:09 PMZlatan Ivanov
04/01/2022, 5:04 PMSELECT segments.ad_destination_type FROM campaign
It still loads 0 rows of data. When I select other tables, they load thousands of rows.
I am not sure what is the issue. Could anyone give some hints?Matheus Guinezi
04/01/2022, 5:07 PMJing Zhang
04/01/2022, 6:29 PMSECRET_STORE_GCP_PROJECT_ID
and SECRET_STORE_GCP_CREDENTIALS
. How do I provide the credentials? Right now I'm pasting the whole json in the .env
file
SECRET_STORE_GCP_CREDENTIALS={"type": "service_account","project_id":"my_project","private_key_id": "acd90445....","private_key":"...",...}
but then I got an error when setting up a Postgres source as shown in the screenshot. Is this related?Eric Santulli
04/01/2022, 6:40 PMKev Daly
04/01/2022, 8:27 PMMarcos Marx (Airbyte)
04/01/2022, 8:30 PMKev Daly
04/01/2022, 8:36 PM2022-04-01 18:26:07 normalization > File ".../transform.py", line 198, in transform_snowflake
2022-04-01 18:26:07 normalization > "password": config["password"],
2022-04-01 18:26:07 normalization > KeyError: 'password'
2022-04-01 18:26:07 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):177 - Completing future exceptionally...
io.airbyte.workers.WorkerException: Normalization Failed.
Matheus Guinezi
04/01/2022, 9:36 PMBryan Estrada
04/01/2022, 10:36 PMAshish Bansal
04/02/2022, 2:47 AM2022-04-02 02:27:49 [32mINFO[m i.a.w.p.KubePodProcess(close):695 - (pod: analytics / source-mysql-sync-3-0-xgbwt) - Closed all resources for pod
2022-04-02 02:27:49 [32mINFO[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):305 - Total records read: 0 (0 bytes)
2022-04-02 02:27:50 [32mINFO[m i.a.w.p.KubePodProcess(close):695 - (pod: analytics / destination-clickhouse-sync-3-0-uynsd) - Closed all resources for pod
2022-04-02 02:27:50 [1;31mERROR[m i.a.w.DefaultReplicationWorker(run):169 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:162) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Suppressed: io.airbyte.workers.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled.
at io.airbyte.workers.protocols.airbyte.DefaultAirbyteSource.close(DefaultAirbyteSource.java:136) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:126) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1
at io.airbyte.workers.DefaultReplicationWorker.lambda$getReplicationRunnable$5(DefaultReplicationWorker.java:312) ~[io.airbyte-airbyte-workers-0.35.64-alpha.jar:?]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
... 1 more
2022-04-02 02:27:50 [32mINFO[m i.a.w.DefaultReplicationWorker(run):228 - sync summary: io.airbyte.config.ReplicationAttemptSummary@3f52e1b1[status=failed,recordsSynced=0,bytesSynced=0,startTime=1648866152849,endTime=1648866470892,totalStats=io.airbyte.config.SyncStats@64ef342d[recordsEmitted=0,bytesEmitted=0,stateMessagesEmitted=0,recordsCommitted=0],streamStats=[]]
2022-04-02 02:27:50 [32mINFO[m i.a.w.DefaultReplicationWorker(run):250 - Source did not output any state messages
2022-04-02 02:27:50 [33mWARN[m i.a.w.DefaultReplicationWorker(run):261 - State capture: No state retained.
2022-04-02 02:27:50 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-04-02 02:27:50 [32mINFO[m i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$1):147 - sync summary: io.airbyte.config.StandardSyncOutput@58afd5e4[standardSyncSummary=io.airbyte.config.StandardSyncSummary@2be1cca4[status=failed,recordsSynced=0,bytesSynced=0,startTime=1648866152849,endTime=1648866470892,totalStats=io.airbyte.config.SyncStats@64ef342d[recordsEmitted=0,bytesEmitted=0,stateMessagesEmitted=0,recordsCommitted=0],streamStats=[]],state=<null>,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@25052332[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@28e51ac6[stream=io.airbyte.protocol.models.AirbyteStream@e43eacb[name=foo_analysis_codecount,jsonSchema={"type":"object","properties":{"id":{"type":"number"},"date":{"type":"string"},"count":{"type":"number"},"user_id":{"type":"number"},"_ab_cdc_log_pos":{"type":"number"},"_ab_cdc_log_file":{"type":"string"},"_ab_cdc_deleted_at":{"type":"string"},"_ab_cdc_updated_at":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=foo,additionalProperties={}],syncMode=incremental,cursorField=[],destinationSyncMode=append_dedup,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@2b62934b[stream=io.airbyte.protocol.models.AirbyteStream@27f239a6[name=foo_upc_upcsurveyresponse,jsonSchema={"type":"object","properties":{"id":{"type":"number"},"user_id":{"type":"number"},"response":{"type":"string"},"creation_date":{"type":"string"},"response_type":{"type":"number"},"updation_date":{"type":"string"},"_ab_cdc_log_pos":{"type":"number"},"_ab_cdc_log_file":{"type":"string"},"_ab_cdc_deleted_at":{"type":"string"},"_ab_cdc_updated_at":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=foo,additionalProperties={}],syncMode=incremental,cursorField=[],destinationSyncMode=append_dedup,primaryKey=[[id]],additionalProperties={}]],additionalProperties={}],failures=[io.airbyte.config.FailureReason@27c134b4[failureOrigin=source,failureType=<null>,internalMessage=io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1,externalMessage=Something went wrong within the source connector,metadata=io.airbyte.config.Metadata@36458735[additionalProperties={attemptNumber=0, jobId=3}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1
at io.airbyte.workers.DefaultReplicationWorker.lambda$getReplicationRunnable$5(DefaultReplicationWorker.java:312)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
... 3 more
,retryable=<null>,timestamp=1648866469670]]]
This means that the worker responsible for reading source failed, is that correct ?
Is there a way I can look into its logs how it failed and try to debug it ? I checked `source-mysql-sync`’s containers but they don’t output any information.Kyle Cheung
04/02/2022, 10:27 PMTG
04/03/2022, 11:45 AMexternalMinio:
enabled: false
host: localhost
port: 9000
s3:
enabled: false
#bucket: airbyte-dev-logs
bucket: airbyte-logs-prod
bucketRegion: ""
gcs:
bucket: "airbyte-logs-prod"
credentials: "/secrets/gcs-log-creds/airbyte-logs-sa.json"
Error
======= service endpoint: <http://airbyte-minio:9000>
======= this is new
Registering Google Cloud Storage publish helper -> com.van.logging.gcp.CloudStorageConfiguration@e6054d9
Collecting content into /tmp/toBePublished4193778332960186391.tmp before uploading.
Collecting content into /tmp/toBePublished9123473825388703691.tmp before uploading.
com.google.cloud.storage.StorageException: Anonymous caller does not have storage.buckets.get access to the Google Cloud Storage bucket.
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:233)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:425)
at com.google.cloud.storage.StorageImpl.lambda$get$4(StorageImpl.java:264)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.Retrying.run(Retrying.java:51)
at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1374)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:263)
at com.van.logging.gcp.CloudStoragePublishHelper.publishFile(CloudStoragePublishHelper.java:38)
at com.van.logging.AbstractFilePublishHelper.end(AbstractFilePublishHelper.java:61)
at com.van.logging.BufferPublisher.endPublish(BufferPublisher.java:66)
at com.van.logging.LoggingEventCache.publishEventsFromFile(LoggingEventCache.java:190)
at com.van.logging.LoggingEventCache.lambda$publishCache$0(LoggingEventCache.java:232)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized
GET <https://storage.googleapis.com/storage/v1/b/airbyte-logs-prod?projection=full>
{
"code" : 401,
"errors" : [ {
"domain" : "global",
"location" : "Authorization",
"locationType" : "header",
"message" : "Anonymous caller does not have storage.buckets.get access to the Google Cloud Storage bucket.",
"reason" : "required"
} ],
"message" : "Anonymous caller does not have storage.buckets.get access to the Google Cloud Storage bucket."
}
윤도경
04/03/2022, 11:57 AMAyyoub Maulana Hadidy
04/04/2022, 4:26 AMhashlib.sha256(primary-key).hexdigest()
Gujjalapati Raju
04/04/2022, 4:54 AMGujjalapati Raju
04/04/2022, 6:10 AMAlban van Rijsewijk
04/04/2022, 7:22 AMOluwapelumi Adeosun
04/04/2022, 11:54 AM