Hi, I have implemented incremental stream. cursor field says <sourceDefined>. I have made the cursor field as date in my stream the date does not get updated but after every sync I am getting the data for the same dates.

Hi Everyone, I am trying to use the open source faros connector but getting this below and i tried this with multiple connector same error coming when i try to pull it inside airbyte source section.

ERROR i.a.s.RequestLogger(filter):110 - REQ 172.18.0.2 POST 500 /api/v1/source_definitions/create - {"name":"okta","documentationUrl":"<https://github.com/faros-ai/airbyte-connectors/tree/main/sources/okta-source>","dockerImageTag":"latest","dockerRepository":"connectprakash/okta-source"}

If you have any questions regarding connector development or how to contribute to Airbyte, we also have 2 forum sections dedicated to that: Connector Development and Contributing to Airbyte. The team has brought down the time to develop a connector to 2 hours with our new version of CDK (Connector Development Kit).

Hi All, I am using the Google Analytics source connector, I created two connections for two different GA views. Connected both sources to same destination and same tables because I want to gather these two views's data in same tables. I set the Sync modes as Incremental Append. My question is, are these two connections will be able to refresh the data correctly when they are scheduled? Because as I understand, incremental append works with ga_date column and if one of the connections appends new records, will the other connection see the other view's record at the current date and think the data is up the date for it's view as well?

If you have any questions regarding connector development or how to contribute to Airbyte, we also have 2 forum sections dedicated to that: Connector Development and Contributing to Airbyte. The team has brought down the time to develop a connector to 2 hours with our new version of CDK (Connector Development Kit).

Hello everyone!!! So, I am creating python http API connector and want to add incremental stream there. I have added incremental column for that but want to make sure is it sufficient to just edit

cursor_field

@property
def cursor_field(self) -> str:
    """
    TODO
    Override to return the cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. This is
    usually id or date based. This field's presence tells the framework this in an incremental stream. Required for incremental.
    :return str: The name of the cursor field.
    """
    return [""]

I mean if my column had 2000 rows and a new row get added then will this much will be suffient to append 1 new row or it will fetch 2001 columns from scratch . Thanks in Advanced!

If you have any questions regarding connector development or how to contribute to Airbyte, we also have 2 forum sections dedicated to that: Connector Development and Contributing to Airbyte. The team has brought down the time to develop a connector to 2 hours with our new version of CDK (Connector Development Kit).

Hello! I'm currently making a Sift source connector, specifically using the Score API to get Sift's latest score for a given customer ID (and the corresponding reasons for the score), and I do this for a list of customer IDs. I use Airbyte's Python source connector template to do this. However, sometimes the API returns a

400 Client Error: Bad Request

, because the scores are saved for only a certain amount of time, and scores from beyond that interval return that error. But, we can't pinpoint exactly which customer IDs were scored within that interval, so it's only through the 400 error code that we can tell. How can I modify the Python source connector template so that I can handle the 400 error myself, and it doesn't reflect as an error in a sync (causing the sync to be marked as 'failed')? Or what might be a better approach to handling these errors?

Hey! What is the expected release date for the SFTP/FTP connector? https://airbyte.com/connectors/sftpftp

If you have any questions regarding connector development or how to contribute to Airbyte, we also have 2 forum sections dedicated to that: Connector Development and Contributing to Airbyte. The team has brought down the time to develop a connector to 2 hours with our new version of CDK (Connector Development Kit).

Hi Team! I have implemented a custom connector using the Python CDK and am currently wondering about the persistence of state when syncs fail. Namely, if a sync fails, is the state still persistent and used in the next sync? I've been trying to find an answer myself and came across https://github.com/airbytehq/airbyte/issues/2627, where it seems mid+stream persistence has been implemented (?), but I can't find details about what changed. In my case, it's okay if the state is not to persistet if a sync fails, because I can handle duplciate records after re-triggering the sync in the destination. However, I'm not sure what is the default behavior right now. I also read about the section Checkpointing state section in the docs, which seem to be for persisting state mid-stream, but I have not implemented any of the two methods. Am I correct that state is only persistent at the end of a successful sync, if I haven't used one of the two checkpointing state methods?

Hi team! I am using the Salesforce source connector to sync accounts, contacts and users. These objects have fields that reference other objects by containing their IDs. Although we know which object to lookup for standard fields, there are also custom fields where we wouldn’t know what kind of object they reference. I am willing to update the connector, but I’d like to ask this first: is it okay to add a config parameter to include field metadata in the records? (we can get that metadata easily from Salesforce).

Any body fetched data from Woo commerce using airbyte? and have connector handy?

By using CDK can I add a column with a default or constant value to every table?

Hello, Not urgent but can someone could review my PR for the salesloft connector ? 💜 https://github.com/airbytehq/airbyte/pull/11460 Also i noticed i missed other endpoints, should I add it now or in another PR ? Thanks ! 👌

Hello! I’m having issues with the Harvest connector. When I try to set up the source, I get

Internal Server Error: java.lang.IllegalArgumentException: no JSON input found

for an error code. I’ve checked the credentials, so it’s not invalid credentials.

Regarding the specification on destinations to output airbyte state messages: https://docs.airbyte.com/understanding-airbyte/airbyte-specification#write

The destination connector should only output state messages if they were previously received as input on stdin. Outputting a state message indicates that all records which came before it have been successfully written to the destination.

Does Airbyte do anything with the state messages outputted by the destination, or is it purely for surfacing info to the user?

Hi everyone! I’m currently working on a HTTP-connector for a weather-API and I’m having one issue. Currently, I can define a configuration with one city and a date. My connector then successfully delivers the weather data for that city & day. What I want is to be able to use an array of cities in my configuration instead of one city. My idea was to change the

request_params

such that instead of returning one dictionary, I return a list of dictionaries where each entry corresponds to the same request but for a different city. However, using a list of dictionaries instead of one dictionaries only results in a

Too many values to unpack

-error. Does anyone have an idea on how I could solve this problem?

Hello 👋 I’ve hit a bug with

source-facebook-marketing

where it assumes

user_tos_accepted

to be a number but it’s actually an object, and so read operations fail. I’ve opened an issue and a PR where I’m reusing the schema definition of

tos_accepted

for

user_tos_accepted

which makes both nullable objects. Can I get a review from one of the maintainers please? It’s a small PR but blocks our use of this connector. thanku Issue: 11800 PR: 11801

How to find why/where the

discover

hangs? Hello, I am building the source and destination connectors for our own product. I had this worked before, but recently the schema discover failed. Not sure it's due to aribyte change or our product change. When I ran the UI test (read from our product and send to csv), the schema discover progress never end. Then I ran

docker run --rm -i -v ~/Dev:/secrets timeplus/airbyte-source-timeplus:dev discover --config /secrets/source_config.json

It shows tables/columns but also hang again, with many INFO messages like

2022-04-07 171926 INFO i.a.i.s.j.AbstractJdbcSource(lambda$discoverInternal$5):137 - Table bookings column cid (type string[0]) -> Json type io.airbyte.protocol.models.JsonSchemaType@492fc69e

It's a Java-based connector. Can I run this via IntelliJ and see which method it is stuck? thanku

its a fairly specialized use-case, but is anyone using airbyte with alloy? took a spin through the community connectors and didnt see anything, but figured id ask anyway!

Hi, I wnat to add pandas to manipulate the json data in streams response method, In local it was working good. But when I deploy the image(after building). It is throwing error as: 2022-04-08 085710 destination > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2022-04-08 085711 destination > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2022-04-08 085711 source > No module named 'pandas' Traceback (most recent call last): File "/airbyte/integration_code/main.py", line 9, in <module> from source_service_now_connector import SourceServiceNowConnector File "/airbyte/integration_code/source_service_now_connector/__init__.py", line 6, in <module> from .source import SourceServiceNowConnector File "/airbyte/integration_code/source_service_now_connector/source.py", line 15, in <module> import pandas as pd ModuleNotFoundError: No module named 'pandas' 2022-04-08 085712 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):305 - Total records read: 0 (0 bytes)

Is it possible to give any info on how far down the road the SFTP connector is? I´m looking to grabbing data from an SFTP to Cloud storage or Bigquery https://airbyte.com/connectors/sftpftp

Hi I am creating a custom connector . I get following error "`Failed to fetch schema. Please try again`" while connecting destination as MSSQL Server. Thanks in advance

ThoughtSpot is potentially looking to contract a few freelance data engineers for specific Airbyte connectors to work with our use case templates we call SpotApps. If this sounds interesting to you, please shoot me a DM.

is there a way to trace this error?

JSON schema validation failed. 
errors: $: null found, object expected

the whole zendesk pipeline failed because of this, and the log aren't clear enough to tell which stream and column that causing this error i want to make a fix but this leave me clueless. also is there any way to suppress this error? for example make certain column with object type as optional? second, is json schema validation run in source or run in destination part? this problem pretty much annoys me (make airbyte keeps retrying until now) and this happens after using zendesk-support connector version 2

Is it possible for a custom http stream source connector to do a post request including sending a json body? Any examples of connectors that are doing this?

Hello all, I wrote a custom http source for orbit.love but I am having trouble adding it as a connector in the UI

airbyte's authenticators are basically the

requests

package's session authenticators aren't they? the

requests_native

package, at that and the fact that they're passed into

self._session.auth

both kinda hint at that. i just want to make sure that there's nothing special about them before trying to roll my own

are there any updates on this PR?