• Is this your first time deploying Airbyte?: No • OS Version / Instance: Debian • Memory / Disk: 32GB memory / 50GB disk • Deployment: docker-compose • Airbyte Version: 0.40.26 • Source name/version: elastic-search custom connector • Destination name/version: bigquery • Step: schema discovery in webapp • Description: I developped a custom source connector for Elastic Search using the Python CDK for version 0.39.42 because the standard one was not fit for my use case (needed to do incremental updates). I created a static schema file located in the file

catalog/configured_catalog.json

that looks like this :

{
  "streams": [
    {
      "stream": {
        "name": "my_stream",
        "json_schema": {},
        "supported_sync_modes": [
          "full_refresh",
          "incremental"
        ],
        "source_defined_cursor": "True",
        "default_cursor_field": [
          "date"
        ]
      },
      "sync_mode": "incremental",
      "destination_sync_mode": "overwrite"
    }
]

I deployed it on Airbyte version 0.39.42 and it was working fine. Now, we updated our Airbyte version to 0.40.26 and the connector is not working fine anymore. The problem is located at the schema discovery step. When I go to “replication” to see my schema :

image▾

1340×512 42.9 KB▾

I get this error message :

image▾

2448×1096 268 KB▾

There is no error log on server side. The only error log I get is on the browser side which says (as on the screenshot) :

TypeError: Cannot convert undefined or null to object
    at Function.keys (<anonymous>)
    at or (CatalogSection.tsx:101:41)
    at sa (react-dom.production.min.js:157:137)
    at qa (react-dom.production.min.js:180:154)
    at Ba (react-dom.production.min.js:178:169)
    at ja (react-dom.production.min.js:177:178)
    at Gs (react-dom.production.min.js:274:126)
    at Au (react-dom.production.min.js:250:347)
    at Ou (react-dom.production.min.js:250:278)
    at Cu (react-dom.production.min.js:250:138)
ls @ react-dom.production.min.js:216
n.payload @ react-dom.production.min.js:217
ho @ react-dom.production.min.js:130
Wa @ react-dom.production.min.js:184
Gs @ react-dom.production.min.js:269
Au @ react-dom.production.min.js:250
Ou @ react-dom.production.min.js:250
Cu @ react-dom.production.min.js:250
_u @ react-dom.production.min.js:243
(anonymous) @ react-dom.production.min.js:123
t.unstable_runWithPriority @ scheduler.production.min.js:18
Vi @ react-dom.production.min.js:122
Ki @ react-dom.production.min.js:123
M @ scheduler.production.min.js:16
b.port1.onmessage @ scheduler.production.min.js:12

The local tests (

check

,

spec

,

discover

,

read

) work fine. Is there a thing I need to modify/update in the connector Python code to get it to work with version 0.40.26 ? I am sure it is an issue with the version since it is working fine on 0.39.42. Thanks,

I have the same problem!! It is imposible to debug because there is not errors

Nahid did you check the Network tab the API response to see what is failing? Also there are any errors in the server logs?

@Marcos Marx (Airbyte) Yes there is no error log on Network tab, discover_schema request seems to go fine

Strange, there isn’t any request errors. Others connections work?

Other custom connectors made with the Python CDK do not work either when trying to discover the schema via the webapp.

@Nahid Oulmi @Mario Beteta https://github.com/airbytehq/airbyte/pull/20869 solves the issue, mostly of Airbyte team is OOO today. The workaround is to rollback to version 0.40.25 until get the PR merged

Thanks @Marcos Marx (Airbyte) 🙂 I fixed the issue on version 0.40.26 by filling the static schema files for every stream in my source instead of leaving the schema files empty. That was a lot of work since the schemas are quite big, and also not necessary since the output I wanted was the raw tables (unstructured json files), but it fixed the bug.

I'm running into this same issue on v.0.40.30. I'm using a custom connector built through the custom connector helper scripts. My schema files are not empty.

Hello Carson, do you mind creating a new message in the channel? Looks the problem related was fixed.