https://linen.dev logo
Join Slack
Powered by
# ask-community-for-troubleshooting
  • j
    In general it seems like Airflow is more general and complex vs focused and simple and developer friendly - which is good in some regards, but bad for simple setups.
  • m
    hey everyone, is there a way to achieve something like a "incremental - overwrite"? i want my sink (postgresql) to contain only new/modified records. (s3 as a source) and not any previously fetched records from the S3 (source).
    m
    n
    • 3
    • 9
  • c
    Hello The Google Ads API is sunsetting v11 on 29 March 2023! I noticed that a recent PR (#21705) bumped the python client version to 19.0.0 which is the equivalent for v12 however the Google Ads connector still uses the v11 APIs Is anyone aware/working on this?
    y
    • 2
    • 1
  • v
    hi airbyte team
  • v
    getting below error after while syncing data from mysql to redshift
  • v
    Copy code
    2023-03-20 04:54:24 WARN i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - Failure during reporting of activity result to the server. ActivityId = 8992106e-ae58-3400-9a8f-845f19368fc6, ActivityType = RunWithJobOutput, WorkflowId=connection_manager_05bd8eee-73b1-4246-b57c-5af6ac247ad5, WorkflowType=ConnectionManagerWorkflow, RunId=2dab3fe5-dfd2-42d4-a44f-747c5924b043
io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskFailed(WorkflowServiceGrpc.java:3866) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$1(ActivityWorker.java:320) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:315) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
  • v
    can someone help in this?
  • s
    Hello Airbyte Team, one of my PRs has been stuck in review for a long time. It has been approved once, but because this is my first PR, it also needs to be reviewed by another set of eyes. Could someone please look over it?
  • l
    Hello Airbyte Team, I want to mention here that one cannot set up the Source Zendesk Chat since the last versions of Airbyte. Potentially since 0.40.3 because of this bug: https://github.com/airbytehq/airbyte/issues/23833 This affects IMO the on-premise and cloud version of Airbyte. I send in PR #24190 which fixes it, but requires a schema change. I was not able to test if it breaks with existing installations since but #23833 broke my Airbyte connection to Zendesk Chat, see issue in GitHub.
  • l
    Since source Zendesk Chat is in GA (=General availability), I think it should be mentioned here.
  • n
    Hello has anyone met any issue with the facebook marketing API ? In my replication i have selected both campaign & ads_insights table but only ads_insights are retrieved
    • 1
    • 1
  • k
    Hi Everyone, Is there a BC connector with Airbyte? Or what is the best approche to retrieve tables from Business Central (Cloud version)?
    n
    • 2
    • 1
  • c
    Hi guys, I am facing a problem on Airbytes after I deployed it on AWS EC2, which I did not encounter when I was running it locally. Does anyone faced this before and know how to solve it?
    Copy code
    Failure Origin: normalization, Message: Something went wrong during normalization
    n
    • 2
    • 11
  • a
    Hi guys, I'm trying to export google analytics v4 data into snowflake through bigquery. My step is to export the data from GA4 to BQ and then create a union of all the events as a view on BQ then with Airbyte connect to bigquery and sync to snowflake. I'm trying to find the best approach to set the replication to append+dedup history using event_date as the cursor field and event_timestamp as the primary_key but I'm not sure if this will get all the data. Please advise on what combination of fields I can use to get all the data. Also, apart from counting the rows in the view on BQ, is their any other way to check if I'm getting all data produced by GA4?
  • p
    Hi all, I am having an issue with the Airbyte OSS instance I have deployed on EC2 running Linux AMI, xlarge. I am using Postgres RDS as a source and S3 as the destination. The sync runs for about 39-42 minutes, is able to transfer 13 million records but then errors out with the following message:
    Copy code
    2023-03-20 12:25:10 [1;31mERROR[m i.a.w.g.DefaultReplicationWorker(replicate):280 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.replicate(DefaultReplicationWorker.java:272) ~[io.airbyte-airbyte-commons-worker-0.42.0.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:190) ~[io.airbyte-airbyte-commons-worker-0.42.0.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:94) ~[io.airbyte-airbyte-commons-worker-0.42.0.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$6(TemporalAttemptExecution.java:202) ~[io.airbyte-airbyte-workers-0.42.0.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
    I’ve already changed the 
    .env
    file to have 10 retries and for the connector to not exit until 50 days later. I’ve attached the logs here. Please let me know how I can go about fixing this issue.
    • 1
    • 2
  • j
    I've been having a very hard time trying to stabliize the Server component of Airbyte on EKS. We were able to get it to work on a single instance using docker compose; but that isn't a sustainable setup as we grow. It's currently throwing some errors about Temporal.
    Copy code
    2023-03-20 13:16:56 ERROR i.a.c.t.ConnectionManagerUtils(getWorkflowState):217 - Exception thrown while checking workflow state for connection id 39afcea3-0a2e-4bfa-80ac-b04db6621a5d
io.temporal.client.WorkflowQueryException: workflowId='connection_manager_39afcea3-0a2e-4bfa-80ac-b04db6621a5d', runId='', workflowType='ConnectionManagerWorkflow'}
    • 1
    • 7
  • g
    hi all, I’m Greg and I’m evaluating Airbyte as a tool for OLTP->OLAP DB replication. I know AWS RDS is not specifically defined as a source, but Postgres is listed. Has anyone had success using AWS RDS (Postgres flavour) as a source?
    n
    • 2
    • 3
  • j
    airbyte is no longer loading on my windows pc. I run 
    docker compose up
    and it goes through some of the boot process and ends with this error: 
    service "bootloader" didn't completed successfully: exit 255
    j
    s
    • 3
    • 9
  • s
    Hello, I am looking for a way to automatically approve non-breaking schema changes in airbyte, using the Salesforce connector
    👀 1
  • g
    Hey guys, how is it going? We are currently hosting Airbyte (version 0.40.22 into our K8S (most specifically EKS) environment we are getting some weird errors (that are happening exclusively into the worker pods), which always happen between sunday/monday. The issue with that is that it seems that when an error happens, all of the other jobs stop working as well, and the logs are not that detailed as well).
    n
    • 2
    • 2
  • l
    Hello everyone. Let’s say I want to call an endpoint - 
    bas_url/invoices?date_start=2023-01-01
    . How do I pass the request parameter of date_start in AIrbyte Low cdk configuration?
  • j
    Hello, I'm running mysql source 0.5.16 and trying to upgrade to 0.5.17 (or anything above), I'm getting the following error message:
    Copy code
    2023-03-20 16:10:10 ERROR i.a.s.p.JobNotifier(notifyJob):133 - Unable to read configuration:
java.lang.NullPointerException: null value in entry: max_seconds_before_source_state_message_emitted=null
    I can't seem to figure out why I'm seeing this. I've looked at the info regarding the 0.6.8 to 0.6.9 upgrade, but since this is still 0.5.xx I don't think it is applicable
  • s
    👋 curious what the community thinks here. My goal is to replicate an entire postgres DB (~500GB) to another postgres, while selectively masking some fields in some tables. Leaving the “why” discussion aside… 1. Is airbyte an acceptable option here? Doesn’t have to be the best option, but would like to leverage it since I already have it setup. 2. Has anyone attempted this with airbyte? If so, what the complexity might be? Thanks!
    n
    • 2
    • 3
  • g
    Hi, I have an issue with a custom dbt connector. We made a custom dbt using a test connector to a test database to the schema public, but now we need it to send the data to the schema of the connector that uses it. But when we made the sync, dbt search the 'airbyte_raw_data' tables in the 'public' schema, that doesn't match with the schema that the production connector is using. We supouse it's because we made it from a connector that send data to public.
  • l
    Hi guys, I get the following error for a Low CDK connector
    Copy code
    jsonschema.exceptions.ValidationError: 'DatetimeBasedCursor' is not one of ['CustomIncrementalSync']
    My manifest.yaml looks like this
    Copy code
    version: "0.1.0"

definitions:
  selector:
    type: RecordSelector
    extractor:
      type: DpathExtractor
      field_path: ["{{ parameters['data_field'] }}"]
  retriever:
    type: SimpleRetriever
    $parameters:
      url_base: "<https://www.zohoapis.eu/books/v3/>"
    record_selector:
      $ref: "#/definitions/selector"
    paginator:
      type: DefaultPaginator
      pagination_strategy:
        type: PageIncrement
        page_size: 20
        start_from_page: 1
      page_size_option:
        field_name: "per_page"
        inject_into: "request_parameter"
      page_token_option:
        type: "RequestOption"
        inject_into: "request_parameter"
        field_name: "page"
    requester:
      $ref: "#/definitions/requester"
  
  requester:
    type: HttpRequester
    url_base: "<https://www.zohoapis.eu/books/v3/>"
    http_method: "GET"
    request_parameters:
      organization_id: "{{ config['organization_id'] }}"
    authenticator:
      type: OAuthAuthenticator
      token_refresh_endpoint: "<https://accounts.zoho.eu/oauth/v2/token>"
      client_id: "{{ config['client_id'] }}"
      client_secret: "{{ config['client_secret'] }}"
      refresh_token: "{{ config['refresh_token'] }}"

  base_stream:
    type: DeclarativeStream
    retriever:
      $ref: "#/definitions/retriever"
      requester:
        $ref: "#/definitions/requester"
  
  incremental_sync:
    type: DatetimeBasedCursor
    cursor_field: "date"
    datetime_format: "%Y-%m-%d"
    start_datetime:
      datetime: "{{ config['date_start'] }}"
      datetime_format: "%Y-%m-%d"
    end_datetime:
      datetime: "{{ today_utc() }}"
      datetime_format: "%Y-%m-%d"
    step: "P1D"
    end_time_option:
      field_name: "date_start"
      inject_into: "request_parameter"
    start_time_option:
      field_name: "date_end"
      inject_into: "request_parameter"
      
  incremental_base_stream:
    incremental_sync:
      $ref: "#/definitions/incremental_sync"
    $ref: "#/definitions/base_stream"
    retriever:
      $ref: "#/definitions/retriever"
      requester:
        $ref: "#/definitions/requester"
  
  invoices:
    $ref: "#/definitions/incremental_base_stream"
    $parameters:
      name: "invoices"
      data_field: "invoices"
      path: "invoices"

streams:
  - "#/definitions/invoices"

check:
  type: CheckStream
  stream_names:
    - "invoices"

spec:
  type: Spec
  documentationUrl: <https://docsurl.com>
  connection_specification:
    $schema: <http://json-schema.org/draft-07/schema#>
    title: Zoho Books Spec
    type: object
    required:
      - access_token
      - organization_id
      - client_id
      - client_secret
      - refresh_token
      - redirect_uri
    properties:
      # 'TODO: This schema defines the configuration required for the source. This usually involves metadata such as database and/or authentication information.':
      access_token:
        type: string
        description: Access token to for Zoho books Oauth
      organization_id:
        type: string
        description: the organization data to pull
      client_id:
        type: string
        description: the organization data to pull
      client_secret:
        type: string
        description: the organization data to pull
      refresh_token:
        type: string
        description: the organization data to pull
      redirect_uri:
        type: string
        description: the organization data to pull
      date_start:
        type: string
        description: Start getting data from that date.
        examples:
          - "%Y-%m-%d"
    Can anyone help me identify the problem?
    • 1
    • 1
  • m
    Hello everyone, I’m trying to do an Airbyte deployment via Helm, and I wanted to use S3 for logging. I’m using the following values for Helm,
    Copy code
    global.logs:
  s3.enabled: true
  s3.bucket: mybucket-airbyte-dev-logs
  s3.bucketRegion: us-west-1
  storage.type: "S3"
  minio.enabled: false
  accessKey:
    existingSecret: airbyte-secret
    existingSecretKey: access_key
  secretKey:
    existingSecret: airbyte-secret
    existingSecretKey: secret_key
minio.enabled: false
    but it doesn’t seem to work. 1 - Minio is still created 2 - Is not using S3. Why do we have two entries on the helm chart for minio? one globally and another as a first-class citizen on the yaml? What is the best practice when it comes to logs? I imagine it is to skip minio and have Airbyte drop the logs straight to S3. Am I right?
    👀 1
    c
    a
    +5
    • 8
    • 29
  • m
    Hey guys, We’re currently examining Airbyte OSS to be embed in a service which will be used by multiple customers to process data from their own data sources into our SaaS platform. 1. Is this use case is valid to Airbyte OSS? 2. How should we differentiate between tenants ? (Resources-wise, secure-wise)? 3. In the documentation it’s written that running thousands of connections might yield issues. We’re expecting to handle such a workload in the future, and even more. Is this just a matter of scaling resources, or we might hit a hard limit in the scenario I described? Thanks!
    n
    • 2
    • 3
  • s
    Hi there, I'm currently trying to do an incremental data replication from Klaviyo into an Azure PostgreSQL database. I have completed the section of the data downoad but it seems that it is hanging at the first model run: 1 of 6 START incremental model _airbyte_landing.klav_events_stg. The data is only about 5GB and looking at the pg_stat_activity, I can see that the dbt model runs for a short while and then disappears. I have attached the log. Is there anyone that has experienced this?
    f93f68b5_70cd_4101_9490_b27571a228ca_logs_628_txt
    n
    • 2
    • 1
  • f
    Hello, I'm trying to test Airbyte locally (M1 Pro) via 
    ./run-ab-platform.sh
    and the 
    airbyte-worker
    keeps throwing warnings as shown below. The Web UI shows 
    Cannot reach server. The server may still be starting up.
    Copy code
    airbyte-worker                      | 2023-03-21 02:30:59 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):269 - Ignoring exception while trying to request Temporal namespace:
airbyte-worker                      | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline CallOptions will be exceeded in 9.999276542s.
    On `airbyte-cron`:
    Copy code
    airbyte-cron                        | 2023-03-21 02:36:18 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):269 - Ignoring exception while trying to request Temporal namespace:
airbyte-cron                        | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline CallOptions will be exceeded in 9.992372959s.
airbyte-cron                        | 	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.52.1.jar:1.52.1]
airbyte-cron                        | 	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.52.1.jar:1.52.1]
airbyte-cron                        | 	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.52.1.jar:1.52.1]
airbyte-cron                        | 	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getSystemInfo(WorkflowServiceGrpc.java:4139) ~[temporal-serviceclient-1.17.0.jar:?]
airbyte-cron                        | 	at io.temporal.serviceclient.SystemInfoInterceptor.getServerCapabilitiesOrThrow(SystemInfoInterceptor.java:95) ~[temporal-serviceclient-1.17.0.jar:?]
    Any suggestions? Thanks!
  • s
    I got the same issue above but with trying to start on ec2 instance with this guide: https://docs.airbyte.com/deploying-airbyte/on-aws-ec2/ I'm not sure how to check error logs, should I be going inside the docker instances or running without the -d flag?
    • 1
    • 2
1...165166167...245Latest