Where can I find the instructions to build airbyte/normalization image

Is there a simple way to ingest a simple python script containing a pandas table in the Airbyte cloud version?

Hi, I have upgraded Airbyte to 0.50.4 on a test machine and am trying to create a connector for the [Userflow](https://userflow.com/docs/api#userflow-api-reference) api. I've set up the connector as stated in the tutorial, create a stream for the endpoint

/users

and made sure (as stated in their doc) to prefix the API key (upon entering my Test Values) with

Bearer

. I'm facing the following error

Could not verify streams:

"Invalid connector manifest with error: Validation against json schema defined in declarative_component_schema.yaml schema failed"

In order to test a stream, ensure that the YAML is structured as described in the docs.

Looking at the automatically imported schema it is pretty straightforward:

{
  "$schema": "<http://json-schema.org/draft-07/schema#>",
  "additionalProperties": true,
  "properties": {},
  "type": "object"
}

Would you have any insight as to what I did wrong ? Thanks~

🔥 Community Office Hours starts in 15 minutes 🔥 At 1pm PDT click here to join us on Zoom!

I have the following class which has a next_page_token and parse_response function which uses a data_field property. How do i change my GetMemberInfo class which does not have a data_field in the api response?

class MailChimpStream(HttpStream, ABC):
    primary_key = "id"
    page_size = 1000

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.current_offset = 0
        self.data_center = kwargs["authenticator"].data_center

    @property
    def url_base(self) -> str:
        return f"https://{self.data_center}.<http://api.mailchimp.com/3.0/|api.mailchimp.com/3.0/>"

    @property
    def availability_strategy(self) -> Optional["AvailabilityStrategy"]:
        return None

    def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
        decoded_response = response.json()
        api_data = decoded_response[self.data_field]
        if len(api_data) < self.page_size:
            self.current_offset = 0
            return None
        else:
            self.current_offset += self.page_size
            return {"offset": self.current_offset}

    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
        response_json = response.json()
        yield from response_json[self.data_field]


class GetMemberInfo(MailChimpStream):
    data_field = ""

where can I find the airbyte .env file?

Hi all, anybody know, when CDC will be available with the oracle connector? Thanks in advance! Cheers, Cedric

Hi, I am buidling a source which uses oauth2 authentication using python CDK(htttp api method). I have an access_token which has 30 days expiry & a refresh_token which get expired once we use it to generate a new access_token & refresh_token. I have tried implemeting with SingleUseRefreshTokenOauth2Authenticator. But I am facing an issue like once the access_token gets expired, the newly generated access_token & refresh_token is not setted properly for the next sync(i.e it is taking the tokens which I have given initially) How could I solve this?

Hi all - Is there anything special I need to do when submitting PRs to get someone to look at it? The Okta connector has a broken stream and I submitted a fix here last week: https://github.com/airbytehq/airbyte/pull/27533 It's complete modulo a question about the version bump given the breaking change, and how to run integration tests. But it doesn't seem like anyone is looking at PRs? This is a critical blocker for us so it looks like I will have to create a local fork for now. Thanks.

hi team. i'm using airbyte oss v0.40.32 deployed in ec2 . i have built a connection from google analytics (UA) to postgres with incremental deduped+history . the sync will run on the daily basis. it works fine until 21-june-2023 after that it throws duplicate key value violates unique constraint "73eb860dc700e88e96781dd9482ff850" 2023-06-26 101233 normalization > DETAIL: Key (_airbyte_unique_key)=(00033ac442e9dcb3545933f1362bfe88) already exists. 2023-06-26 101233 normalization > compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/public/traffic_sources.sql error for that traffic_sources stream

Hi all, i am trying to use the connector builder to build a connector for https://docs.custify.com/#tag/Company/paths/~1company~1all/get when i define a query parameter like this: sorting={"field": "updated_at", "direction": "desc"} This does not get encoded and sent as expected and when i urlencode myself i get an error. Is this related to this issue? https://github.com/airbytehq/airbyte/issues/18784

Hi! How long it takes for PRs to be reviewed? Or what are the requirements before someone start looking at them?

Hello. I'm trying to set up a cursor pagination for an API. Everything is working fine, but the next page in the pagination is a link with the next page ID. However, helpfully, the API will only accept the ID, it doesn't want the entire url. I need to extract the id, from, for example: https://api.example.com/payments?from=ID. Does anyone know how i would split this field to just return the ID?

Hello 👋! Here's a summary of the Connector Builder and CDK features we released over the last week (06/19 - 06/25) Connector builder: - Support single-use OAuth refresh tokens - UI suggests cursor field and date formats by inspecting test data - 🐛 OAuth: refresh request body is passed by url encoded parameters instead of json body - 🐛 Partitioned streams now track state on a per-stream basis, fixing a bug that could cause incremental sync state to be incorrect when using partition routers Python CDK: - Substreams: Better handling of 403 errors when reading records from parent streams Be sure to upgrade to the latest version to benefit from these improvements! Airbyte platform: v0.50.5 CDK: 0.43.1

Hello Team, i am trying to build a custom connector and was checking the following tutorial, https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/connection-checking at step 4, in the documentation it mentions based on the spec.yaml we must define the username, and password structure over there.

def check_connection(self, logger, config) -> Tuple[bool, any]:
        accepted_currencies = {"USD", "JPY", "BGN", "CZK", "DKK"}  # assume these are the only allowed currencies
        input_currency = config['base']
        if input_currency not in accepted_currencies:
            return False, f"Input currency {input_currency} is invalid. Please input one of the following currencies: {accepted_currencies}"
        else:
            return True, None

I overlooked the problem it was in the next step. Anyways, I figured out how to assign and test config values.

🔥 Community Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1pm PDT click here to join us on Zoom!

Hi How can i use the react code? To put in my page to connect to airbyte

While I run ./generate.sh, I got "While trying to generate a connector, an error occurred on line 38 of generate.sh and the process aborted early. This is probably a bug.". Can anyone explain what's going on? and what should I do?

Hi, I am trying to develop a custom connector (generic Python). I can defo emit data, I am just stuck here, when trying to emit a state:

for record in json_records:
    <http://logger.info|logger.info>(record)
    yield AirbyteMessage(
        type=Type.RECORD,
        record=AirbyteRecordMessage(stream=stream.name, data=record, emitted_at=int(datetime.now().timestamp()) * 1000),
    )
#yield AirbyteStateMessage(state={"data": {stream.name: {"date": max_date}}})
yield AirbyteMessage(
    type=Type.STATE,
    state=AirbyteStateMessage(state={"data": {stream.name: {"date": max_date}}})
)

that is inside the read() function, (Source class extension). I don't find an example to understand how to properly fill the last 3 lines! Any clue? thanks

Hi, I am developing a custom connector (generic Python, extending SOURCE class ), I am also emitting a state like this:

yield AirbyteMessage(

type=Type.STATE,

state=AirbyteStateMessage(data={stream.name: {"date": max_date}})

)

it works well, but as I have multiple streams, I need to emit a state per each of them. For some reasons, it looks like just the latest gets saved.

Can I have per_stream_state_enabled set to True when extending Source class and not AbstractSource?

Hi, I am developing a custom connector (generic Python, extending SOURCE class ), I am also emitting a state like this:

yield AirbyteMessage(

type=Type.STATE,

state=AirbyteStateMessage(data={stream.name: {"date": max_date}})

)

it works well, but as I have multiple streams, I need to emit a state per each of them. For some reasons, it looks like just the latest gets saved.

Having an an issue uploading a custom connector. We have had the ability enabled in the Airbyte cloud, but I get this error after it hangs for five minutes:

https://github.com/airbytehq/airbyte/pull/25599 is it ok if i pick this up

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom!

Hi, is it possible to pass a parameter to a job when syncing? like start date and number of days to backfill?

Hi all, I’ve built a custom connector using

SingleUseRefreshTokenOauth2Authenticator

and I’m running into an issue where the access token expires within the same sync. Not entirely sure what’s going on!

How I built IBM DB2 destination connector

Hi Team, Does anyone already build a Zoom connector? Unfortunately, the Alpha Zoom needs JWT token but it doesn't support by Zoom anymore.