Hey I got the low code yaml file working nicely except for o Airbyte #help-connector-development

Hey I got the low code yaml file working nicely, e...

Jon Erik Kemi Warghed

04/07/2023, 4:18 PM

Hey I got the low code yaml file working nicely, except for one problem, I want to add a transformation and I am not sure where I am going wrong. I added like this with the itention of each record also getting the streamslice.job_id it was collected from, but using the UI to test it, no records are getting this field added. What am I missing?

Copy code

record_selector:
        type: RecordSelector
        extractor:
          type: DpathExtractor
          field_path:
            - data
      transformations:
        - type: AddFields
          fields:
            - path: job_id
              value: '{{ stream_slice.job_id }}'
        - type: RemoveFields
          field_pointers:
              - ["attributes", "first-name"]
              - ["attributes", "last-name" ]
      paginator:
        type: DefaultPaginator
        page_token_option:
          type: RequestPath
        pagination_strategy:
          type: CursorPagination

cursor_value: '{{ response[''links''][''next''] }}'

Alexandre Girard (Airbyte)

04/07/2023, 4:21 PM

Hi Jon, Do you have a stream slicer that defines a job_id field?

Jon Erik Kemi Warghed

04/07/2023, 4:23 PM

This part works for me so yes I think it should be configured requester: type: HttpRequester url_base: https://api.teamtailor.com/ path: /v1/jobs/{{ stream_slice.job_id }}/candidates

Jon Erik Kemi Warghed

04/07/2023, 4:23 PM

This is the definition you are asking for right?

👍 1

Jon Erik Kemi Warghed

04/07/2023, 4:23 PM

parent_stream_configs: - type: ParentStreamConfig parent_key: id partition_field: job_id

Jon Erik Kemi Warghed

04/07/2023, 4:25 PM

The removal doesnt work either, so my guess I have misconfigured the transformations somehow.

Alexandre Girard (Airbyte)

04/07/2023, 4:34 PM

hard to tell without seeing the full yaml file... is the

transformations

field at the stream level or at the retriever level? from the snippet, it looks like it's at the retriever level. Can you try moving it to the stream as per this schema

Jon Erik Kemi Warghed

04/07/2023, 4:42 PM

That is correct and my mistake moving it up to stream probably makes it work however I get: "Could not perform read with with error: Every message grouping should have at least one request and response"

Jon Erik Kemi Warghed

04/07/2023, 4:43 PM

Which I interpret it as if the response was empty then it becomes a bit angry 🙂

Jon Erik Kemi Warghed

04/07/2023, 4:45 PM

Added this to the stream level

Copy code

transformations:
      - type: AddFields
        fields:
          - path: [ "field1" ]
            value: "static_value"

Jon Erik Kemi Warghed

04/07/2023, 4:47 PM

I will run with the data and see if it is useful, otherwise I will convert it to a python connector I guess.

Alexandre Girard (Airbyte)

04/07/2023, 4:49 PM

are you using the connector builder? If so, what version are you using?

Jon Erik Kemi Warghed

04/07/2023, 4:50 PM

Yup trying it out, 0.41.0

Jon Erik Kemi Warghed

04/09/2023, 8:08 AM

Hey follow up question, I got the transformations to work when running spec, check, discover and read and everything is working fine. Then using the schema tool to infer schemas, then I reran pointing at those, everything looked great. However when I push the image to my airbyte server (upgraded to .43) I get the following error message:

Copy code

2023-04-09 07:58:30 INFO i.a.w.g.DefaultReplicationWorker(getReplicationOutput):544 - failures: [ { 
 "failureOrigin" : "replication", 
 "internalMessage" : "No properties node in stream schema", 
 "externalMessage" : "Something went wrong during replication", 
 "metadata" : { 
 "attemptNumber" : 1, 
 "jobId" : 15 
 }, 
 "stacktrace" : "java.lang.RuntimeException: No properties node in stream schema\n\tat io.airbyte.workers.general.DefaultReplicationWorker.populateStreamToAllFields(DefaultReplicationWorker.java:693)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromSrcAndWriteToDstRunnable$7(DefaultReplicationWorker.java:384)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\n", 
 "timestamp" : 1681027049119 
} ]

Jon Erik Kemi Warghed

04/09/2023, 8:09 AM

Asked chat gpt to validate the schema for me and it said it looked fine as well 🙂 Can post it if needed but a bit confused here.

Jon Erik Kemi Warghed

04/11/2023, 7:10 AM

Update for anyone looking, rewrote the connector in python and now it is working fine so all good.

2 Views

Open in Slack

Previous Next