Hi! Has anyone here created a custom source to read data from custom objects of Hubspot? I appreciate if you can share!

Hi, denormalized BigQuery destination connector has issues: https://github.com/airbytehq/airbyte/issues/16841

Does anyone have a recommendation to allow multiple streams with the same format digested into one destination? https://github.com/airbytehq/airbyte/issues/2224

hey Getting a crash loop with recent helm chart:

___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
        : airbyte-workers :

  Micronaut (v3.6.3)

2022-09-18 13:40:31 [32mINFO[m i.m.c.e.DefaultEnvironment(<init>):159 - Established active environments: [k8s, cloud, ec2]
2022-09-18 13:40:32 [32mINFO[m i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):107 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2022-09-18 13:40:33 [1;31mERROR[m i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Failed to inject value for parameter [dslContext] of method [configDatabase] of class: io.airbyte.db.Database

Message: No bean of type [org.jooq.DSLContext] exists for the given qualifier: @Named('config'). Make sure the bean is not disabled by bean requirements (enable trace logging for 'io.micronaut.context.condition' to check) and if the bean is enabled then ensure the class is declared a bean and annotation processing is enabled (for Java and Kotlin the 'micronaut-inject-java' dependency should be configured as an annotation processor).
Path Taken: new ApplicationInitializer() --> ApplicationInitializer.checkConnectionActivities --> List.checkConnectionActivities([CheckConnectionActivity checkConnectionActivity]) --> CheckConnectionActivityImpl.secretsHydrator --> SecretsHydrator.secretsHydrator([SecretPersistence secretPersistence]) --> SecretPersistence.localTestingSecretPersistence([Database configDatabase]) --> Database.configDatabase([DSLContext dslContext])
io.micronaut.context.exceptions.DependencyInjectionException: Failed to inject value for parameter [dslContext] of method [configDatabase] of class: io.airbyte.db.Database

Message: No bean of type [org.jooq.DSLContext] exists for the given qualifier: @Named('config'). Make sure the bean is not disabled by bean requirements (enable trace logging for 'io.micronaut.context.condition' to check) and if the bean is enabled then ensure the class is declared a bean and annotation processing is enabled (for Java and Kotlin the 'micronaut-inject-java' dependency should be configured as an annotation processor).
Path Taken: new ApplicationInitializer() --> ApplicationInitializer.checkConnectionActivities --> List.checkConnectionActivities([CheckConnectionActivity checkConnectionActivity]) --> CheckConnectionActivityImpl.secretsHydrator --> SecretsHydrator.secretsHydrator([SecretPersistence secretPersistence]) --> SecretPersistence.localTestingSecretPersistence([Database configDatabase]) --> Database.configDatabase([DSLContext dslContext])
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2087) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1297) ~[micronaut-inject-3.6.3.jar:3.6.3]
....
Caused by: io.micronaut.context.exceptions.NoSuchBeanException: No bean of type [org.jooq.DSLContext] exists for the given qualifier: @Named('config'). Make sure the bean is not disabled by bean requirements (enable trace logging for 'io.micronaut.context.condition' to check) and if the bean is enabled then ensure the class is declared a bean and annotation processing is enabled (for Java and Kotlin the 'micronaut-inject-java' dependency should be configured as an annotation processor).
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2805) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:66) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2065) ~[micronaut-inject-3.6.3.jar:3.6.3]
	... 98 more

any pointers?

Hey, what's the best way to start with implementing a new connector in Airbyte? Any guides to accelerate the development?

trying to migrate data from mongogb to postgres but I get this error...... syntax error at or near "not" LINE 3: create index if not exists

LINE 3: create index if not exists,externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@1ff54c0d[additionalProperties={attemptNumber=2, jobId=25, from_trace_message=true}],stacktrace=AirbyteDbtError:

How can I change the number DBT threads from the default 4 that basic normalization uses?

Hello, i need to connect to sql server database that have in instance, does i need to add the name of instance with the Host ( exemple "localhost\myinstance" ) or in JDBC URL Params (exemple "InstanceName=myinstance") ?

Hello Airbyte community, I’m having some problems when creating my own Airbyte Connector. I got an error message saying:

The requested stream categories was not found in the source. Available streams: dict_keys(['accounts'])

. As can be seen in the picture, it happened during the

python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json

command and after it finished syncing the accounts endpoint of my source API. What does this FATAL message mean and how can I fix this problem?

Hello Airbyte community. I'm having some problems with the blob format I get when migrating data from an Oracle db source to another Oracle destination. Please, how can I normalise this blob format in the destination db?

Hello, I have a quick question: Is it possible to change the number of DBT threads from the default 4 that basic normalization uses?

When deploying airbyte on a non-managed kubernetes cluster, is there any additional steps to take for volumes etc? The docs have a lot of additional information for EKS/GKE, so I'm wondering if similar steps need to be done for self-hosted kubernetes.

One for those with an under-the-hood understanding here. Two questions: 1. Should "reset your data" empty SCD tables? 2. Does "reset your data" empty SCD tables? The documentation doesn't mention one way or the other and a Google search is proving inconclusive (suggesting that it maybe it should, but that it doesn't). Bonus third questions: 3. Does Airbyte hold any data relating to tables locally? I ask this question because I've got duplicate errors in normalization that are solved by resetting my data, but I can only rationalize this to myself as a solution if the sync is testing against values (cursor seed values possibly?) stored within Airbyte itself and not derived from the destination data. Stuart

Hey all, stumped by a deployment issue that I opened last week in Discuss. I've changed the log level to DEBUG for the server and worker pods to add more context. Not sure if there is someone here willing to take a look, but I haven't been able to find a solution. Open to suggestions on how to debug further as well. Thanks for your time! https://discuss.airbyte.io/t/airbyte-kubernetes-helm-deployment-unable-to-configure-external-postgres/2589/6?u=masonah

Hi, I’m new to Airbyte and trying to figure out how to best move configurations between Airbyte instances. We’d like to have separate QA, Stage, Prod instances as well as have developers use their own local instances. I was planning on using the export/import feature to do this and then installed a new instance of Airbyte and saw it was gone! Is there a replacement for this workflow? I couldn’t find any discussion of this. How best to move configurations so people don’t need to do a bunch of error-prone pointing and clicking? Thanks!

Hi, I'm new to airbyte and I'm blocked on deployment on kubernetes with Helm chart

airbyte:0.40.8

. My issue looks like as https://github.com/airbytehq/airbyte/issues/16908 . Any suggestion would be appriciate.

Hi team, Is there any performance matrix for airbyte?

Hi guys. What is the difference between control/data plain for airbyte-workers?

Interesting, when I downgrade from 0.40.7 to 0.40.6(dropped database & created a new one), workers could startup but servers failed with following issue:

2022-09-20 09:03:58 INFO i.a.s.ServerApp(migrateExistingConnectionsToTemporalScheduler):283 - Start migration to the new scheduler...
2022-09-20 09:03:58 ERROR i.a.s.ServerApp(main):320 - Server failed
io.grpc.StatusRuntimeException: INTERNAL: ListOpenWorkflowExecutions operation failed. Select failed: pq: relation "executions_visibility" does not exist
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.44.1.jar:1.44.1]
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.44.1.jar:1.44.1]
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.44.1.jar:1.44.1]
        at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.listOpenWorkflowExecutions(WorkflowServiceGrpc.java:2894) ~[temporal-serviceclient-1.8.1.jar:?]
        at io.airbyte.workers.temporal.TemporalClient.refreshRunningWorkflow(TemporalClient.java:223) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
        at io.airbyte.workers.temporal.TemporalClient.migrateSyncIfNeeded(TemporalClient.java:184) ~[io.airbyte-airbyte-workers-0.40.6.jar:?]
        at io.airbyte.scheduler.client.TemporalEventRunner.migrateSyncIfNeeded(TemporalEventRunner.java:52) ~[io.airbyte.airbyte-scheduler-client-0.40.6.jar:?]
        at io.airbyte.server.ServerApp.migrateExistingConnectionsToTemporalScheduler(ServerApp.java:288) ~[io.airbyte-airbyte-server-0.40.6.jar:?]
        at io.airbyte.server.ServerApp.getServer(ServerApp.java:248) ~[io.airbyte-airbyte-server-0.40.6.jar:?]
        at io.airbyte.server.ServerApp.main(ServerApp.java:317) [io.airbyte-airbyte-server-0.40.6.jar:?]

And I searched nothing related to

executions_visibility

from source code in tag v0.40.6.

I am using Airbyte to sync data (issues) from Jira to Postgres. I can see whole json being synced in a single column

_airbyte_data

. 1. Is there any way to map fields to columns (not a single column) in table? 2. Is there any way to map only a few fields to columns in table? (i.e skip a few fields). I can only check the whole endpoint, not individual fields.

Hello guys) What does a

configId

represent here? How I’m supposed to obtain it? Is there any way I can get latest logs for specific connection?

After deleting following method, my workers finally started successfully. What a great experience!

Hey, I am learning to build an http source connector. I mentioned

supported_sync_modes": ["full_refresh","incremental"]

in

configured_catalog.json

for a stream but in airbyte UI (local) when I try to set up the sync it only shows

full_refresh

as an option for source connector, any idea how to resolve this?

Hey guys, how are you? I'm trying to create a redshift destination with S3 Staging uploading method, but the set up crash and returns a log with the error 403 access denied to bucket. I have checked the credentials and the permissions to s3, and everything is ok, i don't know what is happen...someone can help me?

What would it mean if my run was successful but number of my tables didn’t replicate properly, i.e., the table replicated was empty when it shouldn’t have been?

Hello, I’m trying to execute this command:

./gradlew :airbyte-integrations:connectors:source-robin-odata:integrationTest

after making my first CDK connector. But after executing I get the error message

* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:source-robin-odata:checkPython'.
Python (*[path to virtual env]*) verion 3.7.9 does not match minimal required version: 3.9

After executing the

python --version

command in my virtual env I do get version 3.9.7 back. Does anyone know what is the problem here?

When deploying airbyte on kubernetes (non-managed), there aren't any persistent volumes being created, but there are persistent volume claims. Am I missing a step?

airbyte-bootloader-6f9qw              1/1     Running            0              44m
airbyte-cron-6d96cc88bf-2n7rv         0/1     Pending            0              44m
airbyte-db-57c84b5d69-48cv6           0/1     Pending            0              44m
airbyte-minio-fbccbf969-r9flz         0/1     Pending            0              44m
airbyte-pod-sweeper-9cd5d7c8b-85z4p   1/1     Running            0              44m
airbyte-server-55d4479c47-kzptn       0/1     Pending            0              44m
airbyte-temporal-58bbb9fb49-gx5gm     1/1     Running            0              44m
airbyte-webapp-9c5498c47-bgm8r        1/1     Running            0              44m
airbyte-worker-5d495ffc7d-zj5ng       0/1     CrashLoopBackOff   13 (84s ago)   44m

The worker pod is a crashloop waiting on the db pod, but the db pod is waiting for the pvc which doesn't have a persistent volume to claim. PVCs:

Name:          airbyte-minio-pv-claim
Namespace:     default
StorageClass:
Status:        Pending
Volume:
Labels:        app=airbyte-minio-storage-claim
Annotations:   <none>
Finalizers:    [<http://kubernetes.io/pvc-protection|kubernetes.io/pvc-protection>]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       airbyte-minio-fbccbf969-r9flz
Events:
  Type    Reason         Age                   From                         Message
  ----    ------         ----                  ----                         -------
  Normal  FailedBinding  18s (x662 over 165m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set


Name:          airbyte-volume-configs
Namespace:     default
StorageClass:
Status:        Pending
Volume:
Labels:        airbyte=volume-configs
Annotations:   <none>
Finalizers:    [<http://kubernetes.io/pvc-protection|kubernetes.io/pvc-protection>]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       airbyte-cron-6d96cc88bf-2n7rv
               airbyte-server-55d4479c47-kzptn
Events:
  Type    Reason         Age                   From                         Message
  ----    ------         ----                  ----                         -------
  Normal  FailedBinding  18s (x662 over 165m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set


Name:          airbyte-volume-db
Namespace:     default
StorageClass:
Status:        Pending
Volume:
Labels:        airbyte=volume-db
Annotations:   <none>
Finalizers:    [<http://kubernetes.io/pvc-protection|kubernetes.io/pvc-protection>]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       airbyte-db-57c84b5d69-48cv6
Events:
  Type    Reason         Age                   From                         Message
  ----    ------         ----                  ----                         -------
  Normal  FailedBinding  33s (x661 over 165m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set