which SSO agents does Airbyte Self-Managed Enterprise support?

what kind of hashing function does Self-Managed Enterprise use for PII masking? does the same input always return the same output?

Is it possible to specify which version of abctl to install?

these logs look odd, any clue why?

2024-10-23 22:02:36 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 457776 bytes..
2024-10-23 22:02:36 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):83 Freed more memory than allocated (457776 of -23991071)
2024-10-23 22:02:36 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 1749070 bytes..
2024-10-23 22:02:36 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):83 Freed more memory than allocated (1749070 of -24448847)
2024-10-23 22:02:36 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (e2893) -- Worker finished flushing. Current queue size: 8
2024-10-23 22:02:59 destination > INFO pool-5-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: -26197917 bytes (-24.984280586242676 MB), %% used: -0.04066451920858555 | Queue `uk_invoice_line_items`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_invoice_items`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_charges`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_customers`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_balance_transactions`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_invoices`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_payment_intents`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_payment_methods`, num records: 114, num bytes: 104.45 KB, allocated bytes: 10 MB | State Manager memory usage: Allocated: -36683677 bytes, Used: -47169437 bytes, percentage Used 1.2858426651177852
2024-10-23 22:02:59 destination > INFO pool-8-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-23 22:03:59 destination > INFO pool-5-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: -26197917 bytes (-24.984280586242676 MB), %% used: -0.04066451920858555 | Queue `uk_invoice_line_items`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_invoice_items`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_charges`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_customers`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_balance_transactions`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_invoices`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_payment_intents`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_payment_methods`, num records: 483, num bytes: 448.84 KB, allocated bytes: 10 MB | State Manager memory usage: Allocated: -36683677 bytes, Used: -47169437 bytes, percentage Used 1.2858426651177852
2024-10-23 22:03:59 destination > INFO pool-8-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-23 22:04:59 destination > INFO pool-5-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 614.4 MB, allocated: -26197917 bytes (-24.984280586242676 MB), %% used: -0.04066451920858555 | Queue `uk_invoice_line_items`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_invoice_items`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_charges`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_customers`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_balance_transactions`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_invoices`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_payment_intents`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | Queue `uk_payment_methods`, num records: 837, num bytes: 775.35 KB, allocated bytes: 10 MB | State Manager memory usage: Allocated: -36683677 bytes, Used: -47169437 bytes, percentage Used 1.2858426651177852
2024-10-23 22:04:59 destination > INFO pool-8-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-23 22:05:41 destination > INFO pool-7-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: ab_source__stripe - uk_invoice_line_items, time trigger: true , size trigger: false current threshold b: 25 MB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (56618) -- Worker picked up work.
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (56618) -- Attempting to read from queue namespace: ab_source__stripe, stream: uk_invoice_line_items.
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):83 Freed more memory than allocated (0 of -26197917)
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (56618) -- Batch contains: 0 records, 0 bytes bytes.
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):83 Freed more memory than allocated (0 of -26197917)
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):83 Freed more memory than allocated (0 of -26197917)
2024-10-23 22:05:41 destination > INFO pool-6-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (56618) -- Worker finished flushing. Current queue size: 0
2024-10-23 22:05:45 destination > INFO pool-7-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: ab_source__stripe - uk_balance_transactions, time trigger: true , size trigger: false current threshold b: 25 MB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (c1f40) -- Worker picked up work.
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (c1f40) -- Attempting to read from queue namespace: ab_source__stripe, stream: uk_balance_transactions.
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):83 Freed more memory than allocated (0 of -26197917)
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (c1f40) -- Batch contains: 0 records, 0 bytes bytes.
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):83 Freed more memory than allocated (0 of -26197917)
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):83 Freed more memory than allocated (0 of -26197917)
2024-10-23 22:05:45 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (c1f40) -- Worker finished flushing. Current queue size: 0

How to pass the spark configs Databricks lakehouse sink

@kapa.ai I added two streams on a hubspot conection and airbyte says it is “queued for next sync” but when the sync comes, nothing happen

I have an S3 sync which is continuously showing the same error

Activity with activityType='JobFailure' failed: 'Activity task timed out'. scheduledEventId=1944, startedEventId=1945, activityId=9ced1588-1404-30ff-9ec6-3cd090e3775d, identity='', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED

. When I click "Sync now", I immediately get the error

Failed to start sync: An unknown error occurred. (HTTP 409)

@kapa.ai how can i use Airbyte API to update the frequency

how can I upload my connector docker images from our google artifact registry to Airbyte Cloud?

I am trying to develop a new stream on an existing connector, but when running it I get the following Exception message: Cannot invoke "io.airbyte.config.ConfiguredAirbyteStream.getDestinationSyncMode()" because "stream" is null. Did I miss something ?

Hey everyone, is there a connector for facebook leads

@kapa.ai with incremental sync, how does airbyte know where to start from? For example is during sync the destination has interruption and got down in the middle of sync, then get bak up again after few minutes. How does airbyte handle this to avoid data lose?

@kapa.ai Why do some stream get stuck on "Queued for next sync" status ? How can I force the streams to sync?

is possible to get JobId into the webhook notification url?

can I transfer data from databricks to snowflake using airbyte?

@kapa.ai Is there a way for incremental load for Google Sheet

@kapa.ai I am setting up custom connector, and facing a pagination issue. The api has pagination config as body json payload follows:

{
  "pageable": {
    "page": 1,
    "pageSize": 500
  }
}

How to configure to inject page number to the payload above?

im using the latest airflow, i need to enable auth to allow the airbyte operator to trigger airbyte jobs?

Is it possible to read SQL Server system views using the mssql source connector?

Is it possible to read Postgres information_schema tables and views in the postgres source connector?

How can I reset a connection airbyte in version

0.61.0

I can find any button anywhere to reset the connection. I need to reset the connection because of this error:

Failure in source: Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency to prevent this from happening in the future. See <https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs> for more details.

worker api removed in 1.1.0 version @kapa.ai

@kapa.ai which http method the webhook notification uses?

base_requester:
    type: HttpRequester
    url_base: <https://www.flow-ms.co.uk/api/v1/6524>
    authenticator:
      type: SessionTokenAuthenticator
      login_requester:
        type: HttpRequester
        path: /
        url_base: <https://www.flow-ms.co.uk/api/api-token-auth>
        http_method: POST
        authenticator:
          type: NoAuth
        request_headers: {}
        request_body_json:
          email: <mailto:bbg.apis@bedsandbars.com|bbg.apis@bedsandbars.com>
          password: "{{ config['password'] }}"
        request_parameters: {}
      session_token_path:
        - token
      request_authentication:
        type: ApiKey
        inject_into:
          type: RequestOption
          field_name: Authorization
          inject_into: header

in above is it possible to use

Authorization: Token {token}

instead of bearer?

why in helm chart we have two places to add DB @kapa.ai

@kapa.ai any reason why the syncs are slower after upgrading from 0.63.5 to 1.1?

@kapa.ai i want to inject a value into an outgoing http request in the connection builder. My request is a post request with a json body like this:

{
  "Source": "Integrator",
  "Data": {
    "ApiDocumentId": "3f134d8f-1f72-4c77-b017-0006a52e6ba2",
    "Request": {
      "Filters": [
        {
          "Field": "RecModified",
          "Operator": "Less than or equal",
          "Value": "2024-09-18T12:55:52.937"
        }
      ],
      "Top": 5
      "Limit": 10
    }
  }
}

I want to inject a value for

Limit

as part of pagination. What key name would i use to nest the field inside the json object?

@kapa.ai how can I switch airbyte docker repository from dockerhub to something else?

I have a source which may not contain a timestamp field in all it's records. Is it possible to provide 2 or more cursor fields in no code connector builder to configure an incremental sync?

@kapa.ai my google ads is failing with error :

The customer account can't be accessed because it is not yet enabled or has been deactivated.