Hello, any connector on Webengage source ?

Hey team, I'm try to create a connection with S3 as a source. For the Pattern of files to replicate I need to pass the date when the sync runs. Something like myfolder/dt=${YEAR}-${MONTH}-${DAY}. Is that the right way to do it? Ultimately I want to pass a date provided by Airflow

Hi team, my colleague @Thomas Kaliakos and I are using Matomo in our organisation and are trying to use AirByte to bring our data into BigQuery. We are mainly interested in the live endpoints (i.e. Live.getLastVisitsDetails and - Live.getVisitorProfile) wanted to ask if anyone has a yml file that they have created for the custom connector with streams for both of these two endpoints that works. We have been circling for a bit and would really appreciate any input 🙏 https://developer.matomo.org/api-reference/reporting-api#Live

does anyone know when the aircall connector will be released

I am attempting to create a custom http connector to an endpoint that utilizes pagination. Currently I am able to receive a single record and place it in my destination (Snowflake). However, when I try to utilize the next_page_token function (implementation below) I run into errors. If I run my code from VS Code, everything works as expected, but when I run my docker image in airbyte I consistently get 429s before I receive my first record. Would anyone be able to help me understand why the experience is different and what I should be doing here?

def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
        return response.json().get("@odata.nextLink")

👋 Hi all, we have an exciting announcement to share! Next week's Daily Airbyte Office Hours will feature Deep Dive Sessions hosted by the one and only @[DEPRECATED] Marcos Marx octavia muscle During the deep-dive sessions, Marcos will explain how Airbyte works, delving into each component in every session and explaining their functions. If you’re curious or want to learn more about Airbyte, these sessions will be truly valuable to you. For the first week we’re diving into the

airbyte-bootloader

and the

airbyte-db

services. The presentation will be 20 min, and we'll dedicate the remaining 25 min to questions about the daily topic or general Q&A. Check out the schedule below 👇 Reminders and updates will be posted in #C045VK5AF54 🔥 Deep Dive Sessions: airbyte-bootloader • Monday May 22 - 1pm PDT (zoom link) • Tuesday May 23 - 16:00 CEST / 10am EDT (zoom link) 🔥 Deep Dive Sessions: airbyte-db + Airbyte Database Internals • Wednesday May 24 - 1pm PDT (zoom link) • Thursday May 25 - 16:00 CEST / 10am EDT (zoom link) 🔥 Open Q&A • Friday May 26 - 1pm PDT (zoom link) Hope to see you there! octavia rocket

I solved my earlier 429 problem. Now I’m currently trying to enforce a WaitTimeFromHeaderBackoffStrategy to help deal with some additional issues. We are using the python CDK and I’m just not sure how to implement a backoff strategy using this pattern. Any advice would be appreciated.

Hi, I have a source connector set up to get the data from Github, does anyone know why the data I am getting back is not the same as the data on the website?

@kapa.ai i'm using the CDK UI to develop a new connector. the value i need to extract for my substream partition router is nested within a JSON object. how can i extract that nested value to use it as my partition router?

@kapa.ai when adding a field via transformation to my CDK connector, i get the error message

object does not support item assignment

. how can i fix this?

Hi, can Microsoft Teams source connector sync files to GCS?

Hi! Is there any documentation about how state management works? I want to modify the kafka source to manage the state at the connection level - instead of letting the kafka client manage it - but I can't find documentation about how state works in the Java CDK

While writing my first acceptance tests, I consistently fail _validate_records_structure The error I am getting back speaks to columns existing in the data that does not exist in my json schema. Some example below:

AssertionError:  Record namespace=None stream='mailbox_usage' data={'Report Refresh Date [...] emitted_at=1684509708046 from mailbox_usage stream with fields {'/Report Refresh Date' [...] should have some fields mentioned by json schema: {'/mailbox_usage/reportRefreshDate'

I originally thought that this was a formatting issue. I have since updated my implementation to reflect the format in my configured_catalog.

{
    "streams": [
        {
            "stream": {
                "name": "mailbox_usage",
                "json_schema": {
                    "$schema": "<http://json-schema.org/draft-04/schema#>",
                    "type": "object",
                    "properties": {
                        "Report Refresh Date": {
                            "type": "string"
                        }

However, the problem persists. It is leaving me with two questions…. is the configured_catalog that I declared in my acceptance.py the schema that is being interrogated for this test? Secondly, am I missing a step to get pytest to pick up the new configurations? I’m noticing in my failure that it references another field that no longer exists in my project. Any assistance is appreciated

how are folks building connectors locally, i run into python issues on m1.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':airbyte-integrations:bases:base-normalization:checkPython'.
> Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':airbyte-commons:compileJava'.
> error: invalid source release: 17

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

Hello all, I just deployed my first custom connecter from the Builder. The extraction runs fine, but the normalization step does not. So dbt is invoked but no job is given to him. When looking in the logs, I get the message :

Ignoring stream 'orders' from orders because properties list is empty

. I understand that the json I put in the "Declared schema" section might not be properly read, how can I debug this ? Is there a way to validate that the schema I give is taken into account ? Thanks a lot !

🔥 Community Office Hours starts in 15 minutes 🔥 Q&A - No topic, ask anything! At 1pm PDT click here to join us on Zoom!

is there any connector written in python against a database?. currently i see bunch of Java JDBC related ones.

Hey is there a way to use airbyte open source to do oauth authentication for sources?

Do you offer an interface my users can use to complete oauth flows

Hi, when is the source state persisted? after the destination has processed the data? before that? I'm asking this because I want to understand how errors on the destination affect source state management.

I can see multiple state types: legacy, global and stream based; which one should we use?

Normalization is failing for any snowflake destination for my connections. We have tested multiple connections and normalization fails for all of them. We have tested normalization with a postgress destination and that works. We get the following error when attempting to use basic normalization for a snowflake destination:

Normalization summary: io.airbyte.config.NormalizationSummary@7cab48aa[startTime=1684501191372,endTime=1684501247926,failures=[io.airbyte.config.FailureReason@7394d98b[failureOrigin=normalization,failureType=system_error,internalMessage=Compilation Error,externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@3bb98f9e[additionalProperties={attemptNumber=0, jobId=40, from_trace_message=true}],stacktrace=AirbyteDbtError: 
Encountered an error while reading profiles:
  ERROR: Runtime Error
  Compilation Error
    Could not render ^4{gSU+zM9{#&[@p: Missing end of comment tag
      line 1

Regarding stream state... on startup, will we get the all state transitions for a given stream, or just the last state?

I am running the following command to validate that the dbt project has been configured correctly:

docker run --rm -i -v airbyte_workspace:/data -w /data/$NORMALIZE_WORKSPACE/normalize --network host --entrypoint /usr/local/bin/dbt airbyte/normalization debug --profiles-dir=. --project-dir=.

The response I am getting tells me that the profile.yml file located in the tmp directory is invalid due to missing end of comment tag. I am under the impression that the profile.yml is generated by airbyte. Can someone guide me on either where I am going wrong in my understanding, or where the generation is going wrong?

Hi Folks, A few days ago I've raised here a question in the connection-issues channel regarding Netsuite connection taking too long to extract invoices data. 1000 rows every 30 minutes. I'm still trying to understand if it's an API thing... I mean, In my case the Stream/Table has 350 columns... 98 json columns in this group... I started reading the Netsuite source connector code... I could read without worries the overriden methods from SourceNetsuite class (auth, base_url, get_session...) I tested the check_connection method using invoice stream as the example... using limit 1000. Then I saw the API is just returning the invoice id and the url to invoice details... So my question is, Does the netsuite connector sends one request for every invoice i.e (https://5046696-sb2.suitetalk.api.netsuite.com/services/rest/record/v1/invoice/xxxx) ... or there's an endpoint to receive invoice details for n invoice... with pagination and all the required stuff... I'm asking it because, if the connector need to send one request for every invoice... I think maybe that's the reason for the connection taking too much time to run... I tried to find more details with the API documentation... but it's a little bit confusing to me... Or... the way I see, we have to send one request for every invoice. Could you help to confirm this point? please?

🔥 Community Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-bootloader At 1pm PDT click here to join us on Zoom!

Hi, there. I see in the documentation you can use the Connector Builder UI to build new connections. This works fine for me one Airbyte Cloud but it seems to be missing on local installation. Is it only available in the cloud version? If yes, is there a guide to deploy the generated YAML file to a local connector? Do I realy need to push it to dockerhub first or am I missing something?

Hey everyone, I'm currently developing a custom source connector for Podio and have run into an issue I can't quite figure out regarding pagination. As I understand the 'next_page_token' method in the 'PodioStream' class is supposed to retrieve the pagination information from the response and then use that to make the following request, but the problem is that there is no pagination information in the response... The way you paginate in Podio is by passing an 'offset' parameter to the request which then offsets the records by x amount (which would be the number of records in the former request, which I am able to retrieve). Do any of you have any suggestions on how I should approach this? This is the first time I'm working with Airbyte anything, so I'm still trying to get a grip on things, and maybe there is an obvious approach I'm just missing, but any help regarding this would be appreciated. Thank you 🙏

Hi, I ran 'python -m pytest -p integration_tests.acceptance' but encountered three errors. The first error is related to the 'conftest.py' file located in 'airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test'. The second error is related to reading two sequential readings while running 'airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py'. The third error is that the key 'Error' is missing in 'airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/connector_runner.py'. I hope you can assist me with these issues. Thank you

🔥 Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-bootloader At 16:00 CEST / 10am EDT click here to join us on Zoom!