Hey all, we have a custom MS Entra ID connector th...
# ask-community-for-troubleshooting
c
Hey all, we have a custom MS Entra ID connector that we forked and seems to not accept creating a new source via the API (I'm using Postman for the request in this case). The reason seems to be that it does not accept
microsoft-entra-id
as the
sourceType
, I got that value from the source code of the connector. The request body is as follows:
Copy code
{
  "name": "microsoft_entra_id_test",
  "workspaceId": "{{WORKSPACE_ID}}",
  "configuration": {
    "sourceType": "microsoft-entra-id",
    "start_date": "2017-01-25T00:00:00Z",
    "enable_experimental_streams": false,
    "credentials": {
      "credentials_title": "OAuth Credentials",
      "client_id": "{{CLIENT_ID}}",
      "client_secret": "{{CLIENT_SECRET}}",
      "refresh_token": "{{REFRESH_TOKEN}}"
    }
  }
}
Response:
Copy code
{
  "status": 400,
  "type": "[<https://reference.airbyte.com/reference/errors](https://reference.airbyte.com/reference/errors)>",
  "title": "value-not-found",
  "detail": "The submitted value could not be found.",
  "documentationUrl": null,
  "data": {
    "value": "microsoft-entra-id"
  }
}
This throws an error, saying it cannot find the
sourceType
of that value. This can be right since the connector we are using is a custom one so I requested the source details to see what kind of
sourceType
it needs to work, it said it was set as an empty string:
Copy code
{
  "sourceId": "8cf5d308-7056-45e5-8948-11d217607019",
  "name": "Microsoft Entra Id",
  "sourceType": "",
  "definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5",
  "workspaceId": "7d4c5653-bd89-454a-b520-173e77c68fc8",
  "configuration": {
    "user_id": "**********",
    "client_id": "**********",
    "tenant_id": "**********",
    "client_secret": "**********",
    "__injected_declarative_manifest": {},
    "__injected_components_py_checksums": {}
  },
  "createdAt": 1761225116
}
That's fine, however, setting it to an empty string yields the same response:
Copy code
{
  "name": "microsoft_entra_id_test",
  "workspaceId": "{{WORKSPACE_ID}}",
  "configuration": {
    "sourceType": "",
    "start_date": "2017-01-25T00:00:00Z",
    "enable_experimental_streams": false,
    "credentials": {
      "credentials_title": "OAuth Credentials",
      "client_id": "{{CLIENT_ID}}",
      "client_secret": "{{CLIENT_SECRET}}",
      "refresh_token": "{{REFRESH_TOKEN}}"
    }
  }
}
Response:
Copy code
{
  "status": 400,
  "type": "[<https://reference.airbyte.com/reference/errors](https://reference.airbyte.com/reference/errors)>",
  "title": "value-not-found",
  "detail": "The submitted value could not be found.",
  "documentationUrl": null,
  "data": {
    "value": ""
  }
}
Is there any idea what I need to add to
sourceType
? I thought it was
microsoft-entra-id
but that does not work. Many thanks!
k
Based on the information in the knowledge sources, the error you are encountering (
value-not-found
for
sourceType: "microsoft-entra-id"
) is a common issue when using the Airbyte Public API to create sources for connectors that are not mapped in the backend's name-to-definition mapping. This mapping is what allows the API to resolve a
sourceType
string (like
"microsoft-entra-id"
) to the correct connector definition. If your connector is custom or not included in this mapping, the API will not recognize the
sourceType
and will return the error you are seeing. This issue has been reported for other connectors as well, such as
google-drive
and
brevo
, where the connector is available in the UI but not recognized by the API because the mapping is missing or out of sync with the available connectors. In those cases, the workaround is to use the
definitionId
instead of
sourceType
in your API request. The
definitionId
is the UUID of the connector definition, which you can retrieve from the Airbyte UI or via the API if you list available connector definitions. When you use
definitionId
, the API does not need to resolve the string name and can directly use the connector you specify. Here is how you should structure your request:
Copy code
json {   "name": "microsoft_entra_id_test",   "workspaceId": "{{WORKSPACE_ID}}",   "definitionId": "{{DEFINITION_ID}}",   "configuration": {     "start_date": "2017-01-25T00:00:00Z",     "enable_experimental_streams": false,     "credentials": {       "credentials_title": "OAuth Credentials",       "client_id": "{{CLIENT_ID}}",       "client_secret": "{{CLIENT_SECRET}}",       "refresh_token": "{{REFRESH_TOKEN}}"     }   } }
Replace
{{DEFINITION_ID}}
with the actual definition ID of your custom Microsoft Entra ID connector (which you can find in the source details you posted:
"definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5"
). Key points: - The
sourceType
field is only recognized if the connector is mapped in the backend. For custom connectors, this is often not the case. - Use the
definitionId
field instead of
sourceType
to create a source for a custom connector via the API. - This approach is supported by the Airbyte Public API documentation, which states: "One of configuration.sourceType or definitionId must be provided" and provides an example using
definitionId
 [Airbyte Public API - createSource](https://reference.airbyte.com/reference/start). If you use
definitionId
, your request should succeed as long as the rest of your configuration is correct. If you continue to have issues, ensure your custom connector is properly registered and available in your Airbyte instance, and that you are using the correct
definitionId
for your connector. --- References: - [Airbyte Public API - createSource](https://reference.airbyte.com/reference/start) - [Issue with value-not-found for unmapped connectors](https://github.com/airbytehq/airbyte/issues/49731) - [Similar issue for google-drive connector](https://github.com/airbytehq/airbyte/issues/35653)
c
Thanks @kapa.ai, posting the
definitionId
is not accepted:
Copy code
{
  "name": "microsoft_entra_id_test",
  "workspaceId": "{{WORKSPACE_ID}}",
  "configuration": {
    "definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5",
    "start_date": "2017-01-25T00:00:00Z",
    "enable_experimental_streams": false,
    "credentials": {
      "credentials_title": "OAuth Credentials",
      "client_id": "{{CLIENT_ID}}",
      "client_secret": "{{CLIENT_SECRET}}",
      "refresh_token": "{{REFRESH_TOKEN}}"
    }
  }
}
Response:
Copy code
{
  "status": 422,
  "type": "<https://reference.airbyte.com/reference/errors#unprocessable-entity>",
  "title": "unprocessable-entity",
  "detail": "The body of the request was not understood",
  "documentationUrl": null,
  "data": null
}
k
A message from kapa.ai
The error occurs because the
definitionId
should not be placed inside the
configuration
object. According to the Airbyte Public API documentation,
definitionId
is a top-level field in the request body, not part of
configuration
 [Airbyte Public API - createSource](https://reference.airbyte.com/reference/start). Here is the correct structure for your request:
Copy code
json {   "name": "microsoft_entra_id_test",   "workspaceId": "{{WORKSPACE_ID}}",   "definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5",   "configuration": {     "start_date": "2017-01-25T00:00:00Z",     "enable_experimental_streams": false,     "credentials": {       "credentials_title": "OAuth Credentials",       "client_id": "{{CLIENT_ID}}",       "client_secret": "{{CLIENT_SECRET}}",       "refresh_token": "{{REFRESH_TOKEN}}"     }   } }
Key points: -
definitionId
must be at the top level of the JSON, not inside
configuration
. - Only the connector-specific settings go inside
configuration
. If you structure your request this way, it should be accepted by the API, provided all other values are correct and your custom connector is properly registered [Airbyte Public API - createSource](https://reference.airbyte.com/reference/start). If you still encounter issues, double-check the required fields for your connector’s configuration, as the schema must match exactly what the connector expects.
c
@kapa.ai, I see but now there's another error:
Copy code
{
  "name": "microsoft_entra_id_test",
  "workspaceId": "{{WORKSPACE_ID}}",
  "definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5",
  "configuration": {
    "start_date": "2017-01-25T00:00:00Z",
    "enable_experimental_streams": false,
    "credentials": {
      "credentials_title": "OAuth Credentials",
      "client_id": "{{CLIENT_ID}}",
      "client_secret": "{{CLIENT_SECRET}}",
      "refresh_token": "{{REFRESH_TOKEN}}"
    }
  }
}
This is the response:
Copy code
{
  "status": 500,
  "type": "<https://reference.airbyte.com/reference/errors>",
  "title": "unexpected-problem",
  "detail": "An unexpected problem has occurred. If this is an error that needs to be addressed, please submit a pull request or github issue.",
  "documentationUrl": null,
  "data": {
    "message": "io.airbyte.data.exceptions.ConfigNotFoundException: config type: STANDARD_WORKSPACE id: 496177ae-6b44-4e41-82b2-8d88be82e55d"
  }
}
k
A message from kapa.ai
The error message you are seeing:
Copy code
"io.airbyte.data.exceptions.ConfigNotFoundException: config type: STANDARD_WORKSPACE id: 496177ae-6b44-4e41-82b2-8d88be82e55d"
indicates that Airbyte cannot find the workspace with the ID you provided. This is not an issue with your connector or the request structure, but rather with the workspace itself. This exact error has been reported in other contexts, such as when updating connectors or creating destinations, and is typically caused by one of the following: - The workspace ID does not exist in your Airbyte instance. - There is a misconfiguration or corruption in the Airbyte database regarding workspaces. - The Airbyte instance is not fully initialized or has issues after an upgrade or migration. What you can do: 1. Double-check the workspace ID: Make sure that
{{WORKSPACE_ID}}
is correct and that the workspace actually exists in your Airbyte instance. You can list all workspaces using the API (
GET /workspaces
) to verify this [Airbyte Public API - Workspaces](https://reference.airbyte.com/reference/start). 2. Try creating a new workspace: If the workspace is missing or corrupted, try creating a new one and use its ID in your request. 3. Check for known issues: This error has been seen after upgrades or fresh installs, especially with Airbyte 1.6.x, where the system looks for a workspace with ID
00000000-0000-0000-0000-000000000000
or another invalid ID. In some cases, downgrading to a previous version (e.g., 1.5.1) resolved the issue, or the problem was fixed after a fresh install ([see related issues](https://github.com/airbytehq/airbyte/issues/59196), [another example](https://github.com/airbytehq/airbyte/issues/55716)). 4. If using Terraform or automation: Ensure that the workspace is created and available before you attempt to create sources or destinations in it. If after verifying the workspace ID and ensuring the workspace exists you still encounter this error, it may be a deeper issue with your Airbyte deployment, and you may need to check the server logs or consider re-initializing your Airbyte instance. Summary: The error is not related to your connector or request body, but to the workspace ID not being found in your Airbyte instance. Double-check the workspace's existence and validity before proceeding. --- References integrated from: - [Airbyte Public API - Workspaces](https://reference.airbyte.com/reference/start) - [Issue: ConfigNotFoundException for STANDARD_WORKSPACE](https://github.com/airbytehq/airbyte/issues/59196) - [Issue: 500 error when creating destination via Terraform](https://github.com/airbytehq/airbyte/issues/55716)
c
@kapa.ai Thanks I think that is helping, after correcting the workspace I got the following error:
Copy code
{
  "name": "microsoft_entra_id_test",
  "workspaceId": "{{WORKSPACE_ID}}",
  "definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5",
  "configuration": {
    "start_date": "2017-01-25T00:00:00Z",
    "enable_experimental_streams": false,
    "credentials": {
      "credentials_title": "OAuth Credentials",
      "client_id": "{{CLIENT_ID}}",
      "client_secret": "{{CLIENT_SECRET}}",
      "tenant_id": "1234",
      "user_id": "1234",
      "refresh_token": "{{REFRESH_TOKEN}}"
    }
  }
}
Response:
Copy code
{
  "status": 422,
  "type": "<https://reference.airbyte.com/reference/errors#unprocessable-entity>",
  "title": "unprocessable-entity",
  "detail": "The body of the request was not understood",
  "documentationUrl": null,
  "data": {
    "message": "json schema validation failed when comparing the data to the json schema. \nErrors: $: required property 'client_id' not found, $: required property 'client_secret' not found, $: required property 'tenant_id' not found, $: required property 'user_id' not found "
  }
}
k
A message from kapa.ai
New message text here
c
This has fixed it. 👍
@kapa.ai, is there a way to get the
definitionId
from the API? I was using it to create a source here but that one that I used I got it from requesting an existing source. Getting the source:
Copy code
{
  "sourceId": "8cf5d308-7056-45e5-8948-11d217607019",
  "name": "Microsoft Entra Id",
  "sourceType": "",
  "definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5",
  "workspaceId": "7d4c5653-bd89-454a-b520-173e77c68fc8",
  "configuration": {
    "user_id": "**********",
    "client_id": "**********",
    "tenant_id": "**********",
    "client_secret": "**********",
    "__injected_declarative_manifest": {},
    "__injected_components_py_checksums": {}
  },
  "createdAt": 1761225116
}
Used the following to create a source via the API:
Copy code
{
  "name": "microsoft_entra_id_test",
  "workspaceId": "{{WORKSPACE_ID}}",
  "definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5",
  "configuration": {
    "start_date": "2017-01-25T00:00:00Z",
    "enable_experimental_streams": false,
    "client_id": "{{CLIENT_ID}}",
    "client_secret": "{{CLIENT_SECRET}}",
    "tenant_id": "1234",
    "user_id": "1234"
  }
}
Then the response is `200 OK`:
Copy code
{
  "sourceId": "56cceb22-c5b5-40e5-8533-e082857cc2d4",
  "name": "microsoft_entra_id_test",
  "sourceType": "",
  "definitionId": "9e1d5c73-b7a2-4642-b224-f78379fdf4d5",
  "workspaceId": "7d4c5653-bd89-454a-b520-173e77c68fc8",
  "configuration": {
    "user_id": "**********",
    "client_id": "**********",
    "tenant_id": "**********",
    "start_date": "2017-01-25T00:00:00Z",
    "client_secret": "**********",
    "enable_experimental_streams": false
  },
  "createdAt": 1761272541
}
k
A message from kapa.ai
Yes, you can retrieve the
definitionId
for source connectors using the Airbyte API. According to the Airbyte documentation, you can list all available source definitions (which include their `definitionId`s) via the following endpoint:
Copy code
GET /workspaces/{workspaceId}/definitions/sources/
This endpoint will return a list of all source definitions available in your workspace, each with its
definitionId
and other metadata. This is the recommended way to programmatically obtain the
definitionId
for any connector, including custom ones, rather than extracting it from an existing source object [Create custom connectors from YAML or Docker images using the API and Terraform](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform). If you are working with YAML-based custom connectors, you can use:
Copy code
GET /workspaces/{workspaceId}/definitions/declarative_sources/
This will list all YAML source definitions and their IDs. These endpoints allow you to discover all connector definitions available in your Airbyte instance and obtain the correct
definitionId
for use in source creation requests.