hey,
Is there a way to fetch streams after setup s...
# replication-ask-aiw
Wajdi M
07/31/2023, 11:27 AMhey,
Is there a way to fetch streams after setup source without setup destination or sourceConnection?
k
kapa.ai
07/31/2023, 11:27 AMBased on the information provided in the extracts, it seems that Airbyte requires both a source and a destination to be set up for a connection to be established and for data to be fetched and written. The connection is a configuration for syncing data between a source and a destination, and it includes parameters like the sync schedule, destination namespace and stream names, a catalog selection, sync mode, and optional transformations [source](https://docs.airbyte.com/understanding-airbyte/connections). However, if you're looking to fetch data from a source without setting up a destination, you might want to consider building a custom connector for your specific use case. Airbyte allows for the development of custom connectors, and you can learn more about this process in the [connector development documentation](https://docs.airbyte.com/connector-development/). Please note that this information is based on the provided extracts and for a more accurate answer, you might want to refer to the complete Airbyte documentation or reach out to the Airbyte community.
m
Marcos Marx (Airbyte)
07/31/2023, 1:04 PMYou can use the discover_schema operation https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/sources/discover_schema
✅ 1
w
Wajdi M
07/31/2023, 1:19 PMBig Thank @Marcos Marx (Airbyte) I was thinking connectionId is required but it's not lol
By the way this api */v1/scheduler/sources/discover_schem*a has issue I follow the doc. but always raise this error
Copy code
"message": "Internal Server Error: Cannot invoke \"java.util.UUID.toString()\" because the return value of \"io.airbyte.config.SourceConnection.getSourceId()\" is null",👀 1
m
Marcos Marx (Airbyte)
07/31/2023, 1:28 PMThe endpoint you’re using is used by the scheduler which needs the Source ID. This endpoint uses the current catalog and validate with the new one. The one I shared to you is used before creating the source.
w
Wajdi M
07/31/2023, 1:31 PMYeah, I know that but I have passed a valid sourceId but the same issue raised !
👀 1
Wajdi M
07/31/2023, 2:34 PMHere's the body for the endpoint with issue
Copy code
{
"sourceId": "624f5942-eb95-403e-a8c0-e46a238eb3c1",
"sourceDefinitionId": "36c891d9-4bd9-43ac-bad2-10e12756272c",
"connectionConfiguration": {
"start_date": "2017-01-25T00:00:00Z",
"credentials": {
"client_id": "******",
"client_secret": "*****",
"refresh_token": "******",
"credentials_title": "OAuth Credentials"
}
},
"workspaceId": "b5d89da8-b137-4583-9a4f-89042ee991a2"
}s
Sergio
01/16/2024, 8:46 PMHi @Wajdi M, how did you make the API call? I get a 405 error. Do you use "https://cloud.airbyte.com/api/v1/sources/discover_schema"?
w
Wajdi M
01/17/2024, 2:12 PMHi @Sergio, yes I'm now using /v1/sources/discover_schema and works for me
2 Views