🔥 Community Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1pm PDT click here to join us on Zoom!

@kapa.ai how do i change the default cpu and memory resources for the pod of my source connector

Hello! I'm trying to update an custom image from Docker Hub but I'm getting an error

Get Spec job failed

, but the previous build worked correctly. Difference between images only in libs error build

Successfully installed Deprecated-1.2.14 Jinja2-3.1.2 MarkupSafe-2.1.3 PyYAML-5.4.1 airbyte-cdk-0.39.2 airbyte-protocol-models-0.3.6 attrs-23.1.0 backoff-2.2.1 cachetools-5.3.1 cattrs-23.1.2 certifi-2023.5.7 charset-normalizer-3.1.0 dpath-2.0.8 exceptiongroup-1.1.1 genson-1.2.2 idna-3.4 isodate-0.6.1 jsonref-0.3.0 jsonschema-3.2.0 pendulum-2.1.2 platformdirs-3.5.1 pydantic-1.9.2 pyrsistent-0.19.3 python-dateutil-2.8.2 pytzdata-2020.1 requests-2.31.0 requests-cache-1.0.1 six-1.16.0  typing-extensions-4.6.3 url-normalize-1.4.3 urllib3-2.0.2 wrapt-1.15.0

working build

Successfully installed Deprecated-1.2.13 Jinja2-3.1.2 MarkupSafe-2.1.2 PyYAML-5.4.1 airbyte-cdk-0.37.0 airbyte-protocol-models-0.3.6 attrs-23.1.0 backoff-2.2.1 cachetools-5.3.0 cattrs-22.2.0 certifi-2023.5.7 charset-normalizer-3.1.0 dpath-2.0.8 exceptiongroup-1.1.1 genson-1.2.2 idna-3.4 isodate-0.6.1 jsonref-0.3.0 jsonschema-3.2.0 pendulum-2.1.2 platformdirs-3.5.1 pydantic-1.9.2 pyrsistent-0.19.3 python-dateutil-2.8.2 pytzdata-2020.1 requests-2.30.0 requests-cache-1.0.1 six-1.16.0  typing-extensions-4.5.0 url-normalize-1.4.3 urllib3-2.0.2 wrapt-1.15.0

2023-06-05 15:37:44 ERROR i.a.s.a.ApiHelper(execute):37 - Unexpected Exception
java.lang.IllegalStateException: Get Spec job failed.
	at com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[guava-31.1-jre.jar:?]
	at io.airbyte.commons.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.java:14) ~[io.airbyte-airbyte-commons-server-0.41.0.jar:?]
	at io.airbyte.commons.server.handlers.SourceDefinitionsHandler.getSpecForImage(SourceDefinitionsHandler.java:300) ~[io.airbyte-airbyte-commons-server-0.41.0.jar:?]
	at io.airbyte.commons.server.handlers.SourceDefinitionsHandler.updateSourceDefinition(SourceDefinitionsHandler.java:245) ~[io.airbyte-airbyte-commons-server-0.41.0.jar:?]
	at io.airbyte.server.apis.SourceDefinitionApiController.lambda$updateSourceDefinition$8(SourceDefinitionApiController.java:141) ~[io.airbyte-airbyte-server-0.41.0.jar:?]
	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:23) ~[io.airbyte-airbyte-server-0.41.0.jar:?]
	at io.airbyte.server.apis.SourceDefinitionApiController.updateSourceDefinition(SourceDefinitionApiController.java:141) ~[io.airbyte-airbyte-server-0.41.0.jar:?]
	at io.airbyte.server.apis.$SourceDefinitionApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.41.0.jar:?]
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371) ~[micronaut-inject-3.8.5.jar:3.8.5]
	at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594) ~[micronaut-inject-3.8.5.jar:3.8.5]
	at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303) ~[micronaut-router-3.8.5.jar:3.8.5]
	at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111) ~[micronaut-router-3.8.5.jar:3.8.5]
	at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103) ~[micronaut-http-3.8.5.jar:3.8.5]
	at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659) ~[micronaut-http-server-3.8.5.jar:3.8.5]
	at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49) ~[reactor-core-3.5.0.jar:3.5.0]
	at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62) ~[reactor-core-3.5.0.jar:3.5.0]
	at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.0.jar:3.5.0]
	at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.8.5.jar:3.8.5]
	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.0.jar:3.5.0]
	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.0.jar:3.5.0]
	at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53) ~[micronaut-context-3.8.5.jar:3.8.5]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

Hi, I have a problem with the timezone on the MS SQL source connector. CDC is not working properly. I would like to add a time zone variable (ENV TZ=Europe/Prague) to the Dockerfile. When I build the connector (./gradlew airbyte integrationsconnectorssource mssqlbuild), several tests fail. The tests failed if I left the Dockerfile original.

Can someone help me with OpenSearch custom destination @kapa.ai

Hi, is it possible to process zipped csv files from a S3 bucket?

What is the difference between a low-code connector and a connector builder?

Hi Team, When adding new connector I am getting this error. Internal Server Error: Get Spec job failed.

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom!

Hey folks! I saw that the github conector does not support the Contributors endpoint , is it on your roadmap? If not how can I add it?

Google Directory source & custom fields Is there any roadmap for adding custom fields in GW to be included in the pull of data from Google Directory?

Hi everyone, I am an engineering manager with Airbyte on the Database Sources team. We're excited to announce that we are going to release cursor-less syncs feature for Postgres. Cursorless syncs enable you to configure Standard Replication to sync new or updated rows without explicitly choosing a cursor field. We use xmin system column which is available in all Postgres databases to track inserts and updates to your source data. To participate in getting early access to Postgres cursorless incremental sync feature, please fill out this form and I'll be in touch!

Hi everyone! I have written a dbt script and is working fine in local. But when I use in airbyte, it's throwing an error "Failure Origin: dbt, Message: Something went wrong during dbt". Can anyone tell me how to debug this?

Hi folks, I'm looking into using Airbyte to capture data from an Async API. As in, I submit a job request to the API, then either poll that API to check when the job is completed, or have that API send it's result to a webhook/callback. Does it seem like a fit for Airbyte? I've found a couple of posts along similar lines they didn't get much engagement. It appears Airbyte is centered around the assumption that the web APIs it hits will return responses synchronously https://discuss.airbyte.io/t/handling-async-apis/3501 https://github.com/airbytehq/airbyte/issues/2787 Thank you

Hi everyone, I'm building my first custom connector on Airbyte for the Crisp Chat API using the manifest.yaml file but i'm blocked and haven't found the solution yet, thus asking for your help 🙏 The API i'm requesting has a path for the stream looking like :

<base_url>/conversations/<page_number>?<request_parameters>

I'm looking for a way to have an incremental paginator for the page_number which is not a request parameter but in the path, and which would increment itself as long as there are still results in the json response Thank you !

Hi everyone, I am building a custom connector with IncrementalMixin slices, but am being throttled in the children streams with too many parent ids. Could someone suggest a way to, or point me to an example of the best way to limit concurrent requests?

let's say i want to install a python library - can i use

pip install

in a venv or do i need to add to setup.py, and how do i add to setup.py any dependencies

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1PM PDT click here to join us on Zoom!

How to write the schema to dynamically add values if it's expected that an object can have more than one properties? I want to write the schema file for below snippet

"merge_fields": {
        "property1": null,
        "property2": null
      },

Hi all, I have strange situation with pagination, using connector builder. 1. page size = 1000 2. Start from page = 1 3. Inject page number into outgoing HTTP request → request parametr → page API return me 1000 records, and not tried get another 1000. It looks odd, because i have page size 1000. What can be wrong? I use last build of airbyte (0.5)

Hello I have an issue with my adhoc api connection. I have to ask the end point X with my personal token (refresh token) in order to get the access token and send it to my final endpoint Y in the header. It doesnt work, so could you tell me where im wrong... Thank you!

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves

im getting a 401 error when trying to setup OAuth for a new stream, but I know the client_id and client_secret are good because they work in postman. id like to see what airbyte is actually sending as a request to the refresh endpoint, similar to how it will show the request when you're building a stream. is there a way to see that data? currently all i see is the error message itself.

hey question about standards - would it be a good idea to write a connector in singer and wrap it with airbyte? or are there other specs that are preferable/recommended in order to have the script be separate from the actual apparatus

This message was deleted.

Hi, I'm developing a destination connector in Python. And I found my connector works wired with a CDC source (I've tested postgres and mysql). It seems airbyte failed to remember WAL position no matter how many syncs have been performed successfully:

2023-06-09 03:10:46 [44msource[0m > INFO i.d.c.p.PostgresStreamingChangeEventSource(searchWalPosition):348 WAL resume position 'null' discovered

As I continue debugging, I found something suspicious in log, it seems the state message indicating the framework to remember WAL position is not well processed:

2023-06-09 02:40:26 [32mINFO[m i.a.w.g.ReplicationWorkerHelper(processMessageFromDestination):97 - State in DefaultReplicationWorker from destination: io.airbyte.protocol.models.AirbyteMessage@5cccb520[type=STATE,log=<null>,spec=<null>,connectionStatus=<null>,catalog=<null>,record=<null>,state=io.airbyte.protocol.models.AirbyteStateMessage@520b40c9[type=GLOBAL,stream=<null>,global=<null>,data={"cdc":true,"cdc_state":{"state":{"[\"robert\",{\"server\":\"robert\"}]":"{\"transaction_id\":null,\"lsn\":29930608,\"txId\":748,\"ts_usec\":1686278411722406}"}},"streams":[{"stream_name":"person","stream_namespace":"public","cursor_field":[]}]},additionalProperties={global_={shared_state={state={["robert",{"server":"robert"}]={"transaction_id":null,"lsn":29930608,"txId":748,"ts_usec":1686278411722406}}}, stream_states=[{stream_descriptor={name=person, namespace=public}, stream_state={stream_name=person, stream_namespace=public, cursor_field=[]}}]}}],trace=<null>,control=<null>,additionalProperties={}]

2023-06-09 02:40:26 [33mWARN[m i.a.w.i.b.DefaultSyncStatsTracker(updateDestinationStateStats):195 - The message tracker encountered an issue that prevents committed record counts from being reliably computed. This only impacts metadata and does not indicate a problem with actual sync data.
io.airbyte.workers.internal.book_keeping.StateDeltaTracker$StateDeltaTrackerException: Delta was not stored for state hash 1727035794

2023-06-09 02:40:26 [33mWARN[m i.a.w.i.b.DefaultSyncStatsTracker(updateDestinationStateStats):205 - The state message tracker was unable to match the destination state message to a corresponding source state message.This only impacts metrics and does not indicate a problem with actual sync data.
io.airbyte.workers.internal.book_keeping.StateMetricsTracker$StateMetricsTrackerNoStateMatchException: Destination state message cannot be matched to corresponding Source state message.

The code that processing state message in my connector is quote simple:

def write(
        self, config: Mapping[str, Any], configured_catalog: ConfiguredAirbyteCatalog, input_messages: Iterable[AirbyteMessage]
    ) -> Iterable[AirbyteMessage]:
        ...
        for message in input_messages:
            if message.type == Type.STATE:
                <http://logger.info|logger.info>(message)
                yield message
            elif message.type == Type.RECORD:
                data = message.record.data
                stream = message.record.stream
                ...

I built a postgres to postgres cdc + dedup connection and found that the state message was different from what my connector processed:

# pg dest connector's state message
io.airbyte.protocol.models.AirbyteMessage@523f0336[
  type=STATE,log=<null>,spec=<null>,connectionStatus=<null>,catalog=<null>,record=<null>,
  state=io.airbyte.protocol.models.AirbyteStateMessage@5cf8f701[
    type=GLOBAL,stream=<null>,global=io.airbyte.protocol.models.AirbyteGlobalState@58c76846[sharedState={"state":{"[\"robert\",{\"server\":\"robert\"}]":"{\"transaction_id\":null,\"lsn\":30093368,\"txId\":752,\"ts_usec\":1686279863738929}"}},streamStates=[io.airbyte.protocol.models.AirbyteStreamState@1dc5fd3a[streamDescriptor=io.airbyte.protocol.models.StreamDescriptor@37497b07[name=person,namespace=public,additionalProperties={}],streamState={"stream_name":"person","stream_namespace":"public","cursor_field":[]},additionalProperties={}]],additionalProperties={}],
    data={"cdc":true,"cdc_state":{"state":{"[\"robert\",{\"server\":\"robert\"}]":"{\"transaction_id\":null,\"lsn\":30093368,\"txId\":752,\"ts_usec\":1686279863738929}"}},"streams":[{"stream_name":"person","stream_namespace":"public","cursor_field":[]}]},
    additionalProperties={}],trace=<null>,control=<null>,additionalProperties={}]

# my connector's state message
io.airbyte.protocol.models.AirbyteMessage@5cccb520[
  type=STATE,log=<null>,spec=<null>,connectionStatus=<null>,catalog=<null>,record=<null>,
  state=io.airbyte.protocol.models.AirbyteStateMessage@520b40c9[
    type=GLOBAL,stream=<null>,global=<null>,
    data={"cdc":true,"cdc_state":{"state":{"[\"robert\",{\"server\":\"robert\"}]":"{\"transaction_id\":null,\"lsn\":29930608,\"txId\":748,\"ts_usec\":1686278411722406}"}},"streams":[{"stream_name":"person","stream_namespace":"public","cursor_field":[]}]},
    additionalProperties={global_={shared_state={state={["robert",{"server":"robert"}]={"transaction_id":null,"lsn":29930608,"txId":748,"ts_usec":1686278411722406}}}, stream_states=[{stream_descriptor={name=person, namespace=public}, stream_state={stream_name=person, stream_namespace=public, cursor_field=[]}}]}}],trace=<null>,control=<null>,additionalProperties={}]

However I've no clue to how to go further. Any ideas?

hi! i'm working on the Convex source connector, which is an incremental HttpStream. Can someone describe how pagination is supposed to work, with the

state

property,

stream_state

argument, and

next_page_token

? We only have one cursor, which I was updating in

read_records

. But then

check_availability

started calling

read_records

and advancing the cursor but discarding results, which makes us skip records. From the code it looks like checkpoint state is determined by the (deprecated)

get_updated_state

after calling

read_records

, which means

read_records

must advance the cursor. But the availability check calls

read_records

and discards the result, which means

read_records

must not advance the cursor. This looks impossible to implement correctly.

🔥 Community Office Hours starts in 15 minutes 🔥 At 1pm PDT click here to join us on Zoom!

This message was deleted.

I have set up a source on the builder UI but don't know where all the code when I created that source is located? Please help me find the source folder that I build on Builder UI