@kapa.ai , why should the cursor be range instead of start in case of split up interval

@kapa.ai I am adding a new parameter to an exisiting airbyte source. In order to get the Acceptance Test working must I pass in parameters somewhere?

@kapa.ai why airbyte 64.1 dont have or not working pasic auth.

What does this error message mean? Failure executing: PATCH at: https://10.96.0.1:443/api/v1/namespaces/airbyte-abctl/pods/rce-postgres-check-16697a14-10f0-4bb4-a6d0-67c183837eab-0-ydybe?fieldManager=fabric8. Message: Unauthorized.

@kapa.ai

/app/connector_acceptance_test/tests/test_core.py:624: AssertionError
------------------------------ Captured log setup ------------------------------
WARNING  root:conftest.py:137 Connector config file not found at /test_input/secrets/config.json

where should I be placing my config.json for acceptance testing, and what should it contain?

@kapa.ai, i am getting this error during shopify sync

2024-09-05 12:53:35 source > During the sync, the following streams did not sync successfully: custom_collections: AirbyteTracedException("'message'")
2024-09-05 12:53:35 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):326 - During the sync, the following streams did not sync successfully: custom_collections: AirbyteTracedException("'message'")
2024-09-05 12:53:35 source > None

what is the issue?

Based on the knowledge sources provided, it seems that you are experiencing issues with unauthorized access (401 error) to

/manifest.json

and forbidden access (403 error) to

/api/v1/users/get

when using Airbyte behind an NGINX reverse proxy. The 401 Unauthorized error typically indicates that the request lacks valid authentication credentials for the target resource. In this case, it seems to be related to the

/manifest.json

file. The 403 Forbidden error, on the other hand, indicates that the server understood the request, but it refuses to authorize it. This status is similar to 401 (Unauthorized), but indicates that the client must authenticate itself to get the requested response. In this case, it seems to be related to the

/api/v1/users/get

endpoint. Unfortunately, the provided knowledge sources do not contain specific solutions to these issues. However, they do suggest that other users have experienced similar issues when using Airbyte with a reverse proxy. One user managed to get it working on port 8000 with insecure-cookies and without reverse proxy by using the

abctl local install --no-browser --insecure-cookies --host <http://xxx-vm-airbyte-04.westeurope.cloudapp.azure.com|xxx-vm-airbyte-04.westeurope.cloudapp.azure.com>

command and setting the email and organization. They were then able to log in via the credentials provided by

abctl local credentials

[source](https://github.com/airbytehq/airbyte/issues/44198). It's recommended to check your NGINX configuration and ensure that it's correctly set up to proxy requests to Airbyte. Also, ensure that your Airbyte instance is correctly configured to accept connections from your NGINX server. If the issue persists, it would be beneficial to reach out to the Airbyte community or support for more specific guidance.

Based on the knowledge sources provided, it seems like you're experiencing issues with unauthorized access when using Airbyte behind an NGINX reverse proxy. This issue has been reported by other users as well, particularly when trying to access

/manifest.json

and

/api/v1/users/get

. The 401 Unauthorized error on

/manifest.json

suggests that the server is rejecting your request due to invalid credentials. This could be due to incorrect configuration of your NGINX reverse proxy or issues with your Airbyte setup. The 403 Forbidden error on

/api/v1/users/get

indicates that the server understood the request, but it refuses to authorize it. This status is similar to 401 (Unauthorized), but indicates that authentication alone will not suffice - i.e., the client must authenticate itself to get the requested response. Unfortunately, the provided knowledge sources do not contain a specific solution to these issues. However, you might want to check your NGINX and Airbyte configurations, ensure that you're using the correct credentials, and that your reverse proxy is correctly set up to forward requests to Airbyte. For more detailed help, consider reaching out to the Airbyte community on their [discussion forum](https://discuss.airbyte.io/) or opening an issue on their [Github page](https://github.com/airbytehq/airbyte/issues). Sources: - [Forum Post: NGINX reverse proxy behind port forward to local Airbyte webserver](https://discuss.airbyte.io/t/2429) - [Github Issue (closed): Can't get passed Getting Started screen after fresh installation](https://github.com/airbytehq/airbyte/issues/44198) - [Airbyte Public API - Connections](https://reference.airbyte.com/reference/start)

if I already installed airbyte and I want to enable insecure-cookies do I have to run abctl local install again? @kapa.ai

@kapa.ai how can I configure the image pull secret for container orchestrator?

@kapa.ai What is the ajs_anonymous_id that is being sent with airbyte requests?

I still get 403 forbidden, for example on route api/v1/instance_configuration/setup and I have a nginx file that does proxy_pass http://localhost:8000; to my local airbyte instance, is maybe a header missing or what authentication does it need? @kapa.ai

@kapa.ai, how do I know what version of airbyte I am running on?

Airbyte is deployed on kubernetes. How to use the Airbyte API? What should be the url ?

@kapa.ai does airbyte need a DNS name or can also work with an iP? since using --host parameter with an IP does not work, I tried to leave it as localhost and use nginx to proxy pass from my server ip to localhost, but i have 403 errors, so does that come from airbyte?

@kapa.ai, when one click test data source in the UI, what happens behind the scenes in airbyte?

@kapa.ai, is it possible to downgrade from airbyte 0.64.1 to 0.63.11 when using helm charts?

@kapa.ai, during test data source from the UI, I am getting below stacktrace error in UI. Which pod logs can I check for the same error?

io.temporal.failure.ActivityFailure: Activity with activityType='RunWithWorkload' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=8e96f207-a7c7-3d04-bef3-d50835de703b, identity='1@airbyte-064-worker-5585fcbd5d-5js9l', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHE

@kapa.ai I installed airbyte via abctl on aws. I can enter the UI but while setting up my snowflake destination I get this error

Failure executing: PATCH at: <https://10.96.0.1:443/api/v1/namespaces/airbyte-abctl/pods/on-snowflake-check-ccb57db5-d34d-4dd0-8521-9b0be39fb524-0-aornd?fieldManager=fabric8>.

@kapa.ai now i get this error when trying to access the panel TypeError: Cannot read properties of undefined (reading 'mode') at dYe (http://10.10.50.166/assets/core-ewtsdo5l3e.js:352:12904) at http://10.10.50.166/assets/core-ewtsdo5l3e.js:352:13059 at Object.useMemo (http://10.10.50.166/assets/core-ewtsdo5l3e.js:38:25886) at Mt.useMemo (http://10.10.50.166/assets/core-ewtsdo5l3e.js:9:6208) at amt (http://10.10.50.166/assets/core-ewtsdo5l3e.js:352:13023) at oR (http://10.10.50.166/assets/core-ewtsdo5l3e.js:38:19551) at xB (http://10.10.50.166/assets/core-ewtsdo5l3e.js:40:44099) at _B (http://10.10.50.166/assets/core-ewtsdo5l3e.js:40:39820) at qee (http://10.10.50.166/assets/core-ewtsdo5l3e.js:40:39747) at rb (http://10.10.50.166/assets/core-ewtsdo5l3e.js:40:39597)

@kapa.ai where in github is airbyte helm chart defined?

@kapa.ai, if I want to update log level from INFO to DEBUG, should I edit yaml and apply helm upgrade or is kubectl edit better?

@kapa.ai I set up a substream for an incremental parent stream. I just synced 249 new records for the incremental parent stream but 1032 records for the substream were leaded. How is that possible? I was assuming that the sub-stream passes only the newly loaded records from the parent stream to the api.

when i override the values for the helm chart at https://github.com/airbytehq/airbyte-platform/blob/main/charts/airbyte/values.yaml, do i need to have all of the values from the repo are or just ones i want to override

@kapa.ai how do i upgrade using abctl

Hello @kapa.ai, I am seting up various connections MSSQL -> Snowflake. The sync is done using 'Incremental Sync - Append + Deduped'. My table source in MSSQL has CDC activated. My main issue is that the job lasts arround 50min when its only arround 0kb or 0 records, in the log the most repeated string is this:

2024-09-04 23:39:41 source > INFO pool-3-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=45, polls=0
1881
2024-09-04 23:39:41 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.522826179S after its previous call which was also logged.
1882
2024-09-04 23:39:41 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
1883
2024-09-04 23:39:52 source > INFO pool-3-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=46, polls=0
1884
2024-09-04 23:39:52 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.52142362S after its previous call which was also logged.
1885
2024-09-04 23:39:52 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
1886
2024-09-04 23:40:02 source > INFO pool-3-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=47, polls=0
1887

@kapa.ai in the connector builder the tooltip for

parrent stream

-

incremental Parent

states this: "Incremental Parent Causes the parent stream's records to be read incrementally, so only parent records that have been updated since the last sync will be retrieved for use in this stream. This only works if this child stream is Incremental. WARNING: If the parent stream records' cursor fields are NOT updated whenever the relevant child records are updated, then those child records may be missed!" Why is this only working if the child stream is incremental? The api for the child stream only accepts one imput which is the id form the parent stream. Is it even possible to make the child steam incremental if the api is not accepting a date filter for the child stream?

It seems like you're asking about Python 3 in the context of Airbyte. Python 3 is often used in the development and operation of Airbyte connectors. For instance, when setting up a development environment for Airbyte on Windows, Python 3.10 is required and can be installed using the commands provided in the [tutorial video](). Here are the commands for installing Python 3.10 in WSL:

bash sudo add-apt-repository ppa:deadsnakes/ppa sudo apt update apt list | grep python3.10 sudo apt install python3.10 sudo apt install python3.10-distutils curl <https://bootstrap.pypa.io/get-pip.py> -o get-pip.py sudo python3.10 get-pip.py sudo apt install python3.10-venv

In addition, Python 3.10 or later is also a prerequisite for several Airbyte tutorials, such as the [Outdoor Activity Analytics Stack with Airbyte, dbt, Dagster and BigQuery](https://airbyte.com/tutorials/outdoor-activity-analytics-stack-with-airbyte-dbt-dagster-and-bigquery) and the [End-to-end RAG using Airbyte Cloud, S3, BigQuery and Pinecone](https://airbyte.com/tutorials/end-to-end-rag-using-airbyte-cloud-s3-bigquery-and-pinecone). If you haven't installed Python 3.10 or later yet, you can download and install it from [Python's official website](https://www.python.org/downloads/).

@kapa.ai I'm trying to test out airbyte for a POC. I want to move data from AWS DynamoDB tables to AWS Redshift. I installed airbyte locally on my map and am running using abctl. I had no trouble setting up Dynamodb as a source and Resdhift as a destination; however, when I try to create a connection between they dynamodb source and Redshift target, I get this 504 error: "An unknown error occurred. (HTTP 504)" I have a basic yaml config file I'm passing in to try to increase timeouts. Here's the config file:

global:
  deploymentMode: "oss"
  edition: "community"
  env_vars:
    HTTP_IDLE_TIMEOUT: 1800s
webapp:
  ingress:
    annotations:
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: internal
      <http://nginx.ingress.kubernetes.io/proxy-body-size|nginx.ingress.kubernetes.io/proxy-body-size>: 16m
      <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: 1800
      <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: 1800

I'm starting airbyte like this:

abctl local install --values <path_to_config_file>

Any idea how I might troubleshoot/fix this?

@kapa.ai hello! i have k8 environment for Airbyte. And i see that some of destination-postgres-write pods is not evict properly. Main and other containers were terminated, but pods is still alive.