Hello Me again slightly smiling face I am trying to find a w Airbyte #help-connector-development

Hello! Me again :slightly_smiling_face: I am tryin...

Rachel RIZK

11/07/2022, 4:38 PM

Hello! Me again 🙂 I am trying to find a way to

increment - append

data instead of

full refresh - overwrite/append

with the low-code CDK. Here's how the data is structured:

Copy code

testdata = { 
  'rows': [
    {
      'data_row1': [{ 'kpi': 2, 'date': "2022-11-07" }],
      'metadata_row1': {'id': 1}
    },
    {
      'data_row2': [{ 'kpi': 8, 'date': "2022-11-07" }],
      'metadata_row2': {'id': 2}
    },
  ]
}

• To get all rows, I need to set

record_selector

rows

field • But to get the date field (enabling increment with slicing) I need to set the

cursor_field

to something like that:

["data_row1", "0", "date"]

data_row1[0]['date']

. ◦ With 1st solution, I have an error because it won't accept a list ◦ With 2nd solution, the script runs but the cursor is not working and displays older data than what's in

sample_state.json

I'm wondering, is there actually a way to activate increment on this kind of data? It's the last step before I can contribute for a new connector 🙏

✅ 1

Rachel RIZK

11/07/2022, 4:42 PM

Also, I tried by setting

record_selector: ["rows", "0", "data_row1"]

cursor_field: "date"

and it worked well, but it's not correct as I'm only getting the first row for each call. It's when I am nesting the cursor_field that it's simply not working anymore 😕

Brian Lai (Airbyte)

11/09/2022, 10:50 PM

hey rachel! so unfortunately the framework doesn’t currently support using a nested cursor field and can only select at what is at the top level. We do have a ticket tracking this, but we haven’t had a chance to figure out when we’ll tackle it.

Brian Lai (Airbyte)

11/09/2022, 10:53 PM

one idea could be to use an add field transformer to “move” the field to the top level where it can then be used as a cursor. that could be a possible solution. I did want to clarify the example you sent, so for each row item, the object will have different field names like

data_row1

data_row2

💡 1

Rachel RIZK

11/10/2022, 6:44 PM

oh! thank you so much for this clever workaround, it actually worked with an added field that is then called as a cursor 🎉

Rachel RIZK

11/10/2022, 6:46 PM

the object has the same name actually, more like

data_row

for each row item

Rachel RIZK

11/10/2022, 6:50 PM

I used something like this for the added field:

Copy code

path: ['date']
value: ' {{ stream_slice['end_time'] }}'

then I was able to call

'date'

in cursor field, which was finally stored in state 🥳

Rachel RIZK

11/10/2022, 6:51 PM

I’ll be posting the PR for new connector soon loading

Brian Lai (Airbyte)

11/11/2022, 3:22 AM

Glad to hear it! When you mentioned different field names in the example, I wasnt sure if that would work since the transformer wouldn't be able to handle the changing field, but happy to hear it worked for your connector 🙂

3 Views

Open in Slack

Previous Next