How would I replace a source's manifest.yaml file ...
# replication-ask-ai
c
How would I replace a source's manifest.yaml file with a custom one in deployment of Airbyte OSS @kapa.ai?
k
The knowledge sources don’t describe a direct, documented way to “swap out” the manifest.yaml of an existing catalog source definition in an Airbyte OSS deployment. They do describe two related patterns: 1. Create a new custom connector from a YAML manifest via API/Terraform (OSS ≥ 1.6) Airbyte 1.6 introduced public API endpoints to create/update YAML source definitions from a Connector Builder manifest. [[Airbyte 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] For YAML sources in a workspace you can: -
POST /workspaces/{workspaceId}/definitions/declarative_sources/
– create a YAML source definition from your manifest -
PUT /workspaces/{workspaceId}/definitions/declarative_sources/{definitionId}
– update that YAML source definition with a new manifest -
DELETE /workspaces/{workspaceId}/definitions/declarative_sources/{definitionId}
– delete that YAML source definition These endpoints are explicitly for custom YAML definitions; they cannot be used to modify Airbyte’s public connector definitions. [[Airbyte 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] So, in OSS, the supported pattern is: - Treat your modified
manifest.yaml
as a new custom YAML connector. - Use the declarative_sources API (or Terraform provider) to create/update that custom definition in your workspace. - Point your sources/connections at this custom definition instead of the catalog one. 2. Clone a catalog connector into a custom YAML connector and modify it A forum answer for the Recurly connector describes the practical workflow (in the Builder / YAML world): [[Recurly forum](https://discuss.airbyte.io/t/10023)] 1. Create a new
source-<name>
in your workspace from scratch. 2. Copy the contents of the catalog connector’s
manifest.yaml
from GitHub into your new source. 3. Make your changes and test the streams you changed. 4. Use “Publish → Contribute to Airbyte” if you want to upstream the fix. This again relies on creating a new custom connector rather than replacing the manifest of the existing catalog definition. --- ### What this means for your question Based on the docs: - There is no documented mechanism in these sources to directly replace the
manifest.yaml
of an existing built‑in source definition in an OSS deployment. - The documented and supported approach is: - Create a custom YAML source definition in your OSS workspace using your custom
manifest.yaml
(via API/Terraform or Builder export). [[Airbyte 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform); [Recurly forum](https://discuss.airbyte.io/t/10023)] - Migrate your existing source(s)/connection(s) to use this new custom definition. If you specifically need to overwrite the catalog connector’s manifest in-place, the knowledge sources do not describe or support that path, so I can’t provide a reliable, documented procedure for it.
The knowledge sources don’t describe a direct, OSS‑specific procedure to “replace a source’s
manifest.yaml
in deployment,” so I can’t give a step‑by‑step OSS deployment answer with full confidence. What is documented: - Airbyte 1.6+ exposes API endpoints to create, update, and delete YAML (declarative) source definitions using a Connector Builder manifest. These are under
/workspaces/{workspaceId}/definitions/declarative_sources/...
for YAML sources. [[Release 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] -
POST /workspaces/{workspaceId}/definitions/declarative_sources/
– create YAML source definitions -
PUT /workspaces/{workspaceId}/definitions/declarative_sources/{definitionId}
– update a specific YAML source definition (i.e., replace its manifest) -
DELETE /workspaces/{workspaceId}/definitions/declarative_sources/{definitionId}
– delete a YAML source definition These endpoints are available via the public API and Terraform provider. [[Release 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] - The same release note explicitly says you **cannot use these endpoints to modify Airbyte’s public connector definitions**; they are for custom connectors only. [[Release 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] - A forum answer for modifying a catalog connector (Recurly) suggests the pattern: create a new custom source in your workspace, copy the catalog connector’s
manifest.yaml
from GitHub, modify it, test, and then publish/contribute. [[Recurly forum](https://discuss.airbyte.io/t/10023)] Putting this together, the supported pattern (for OSS as well, since these are platform APIs) to “replace” a manifest is: 1. Create a custom YAML source definition in your OSS workspace using your custom
manifest.yaml
via: -
POST /workspaces/{workspaceId}/definitions/declarative_sources/
with your manifest content. [[Release 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] 2. Point your sources/connections to this custom definition instead of the built‑in catalog connector. 3. If you later need to change the manifest for that custom definition, update it with: -
PUT /workspaces/{workspaceId}/definitions/declarative_sources/{definitionId}
and pass the new manifest YAML. [[Release 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] 4. If you were trying to override a built‑in connector’s manifest in place: the docs explicitly say that’s not supported via these APIs; you must work through a custom connector definition instead. [[Release 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] If you share whether you’re trying to modify a built‑in connector vs. an existing custom YAML connector, I can map this more precisely to your case using the same documented APIs.
c
If I can't modify a catalog source - how would I add a second option to the base_url of the Iterable source @kapa.ai?
k
A message from kapa.ai
You’re right that you can’t modify the built‑in Iterable catalog connector in place. The knowledge sources describe two realistic options to get a second
base_url
(e.g., EU vs US) for Iterable: --- ### 1. Build a custom / forked connector (recommended) From the Iterable thread, the suggested path is to iterate on the existing source connector and publish it as your own custom connector image, then use that instead of the catalog one. [[Iterable forum](https://discuss.airbyte.io/t/5094)] High‑level steps (as described there): 1. Clone the
source-iterable
connector from the Airbyte repo. 2. Modify the code or manifest so that: -
url_base
(or equivalent) is configurable via a config field (e.g.
region
or
base_url
), and - you switch between
<https://api.iterable.com>
and
<https://api.eu.iterable.com>
based on that config. 3. Build and publish your own Docker image for this connector. 4. In your OSS deployment, register this as a custom source definition (custom Docker connector) and use it instead of the catalog Iterable source. [[Custom connectors](https://docs.airbyte.com/integrations/custom-connectors)] This gives you a proper, maintainable way to support multiple regions. --- ### 2. Create a new low‑code/YAML connector based on Iterable’s manifest The same thread notes that Iterable is now a low‑code connector, but it uses custom extractors, so it’s not cleanly editable in the Builder UI. [[Iterable forum](https://discuss.airbyte.io/t/5094)] However, the general pattern for low‑code connectors is: 1. Take the existing
manifest.yaml
from GitHub. 2. Create a new YAML/low‑code connector in your workspace (via Builder or as a declarative YAML source). 3. Paste the manifest, then: - Add a new config field (e.g.
region
). - Update the `Requester`’s
url_base
to depend on that config (e.g. choose EU vs US). How exactly to express that logic in YAML is not shown in the provided sources, so I can’t give a concrete snippet. 4. Use this new YAML connector instead of the catalog Iterable source. The docs confirm that YAML connectors use a
Requester
with a
url_base
field, which is where you’d plug in your configurable base URL. [[Requester docs](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/requester)] --- ### 3. What you cannot do - You cannot directly edit the catalog Iterable connector’s manifest in Airbyte Cloud or OSS via the supported APIs; the 1.6 APIs and Terraform only allow creating/updating custom YAML or Docker definitions, not modifying public catalog connectors. [[Airbyte 1.6 notes](https://docs.airbyte.com/release_notes/v-1.6#create-custom-connectors-from-yaml-or-docker-images-using-the-api-and-terraform)] --- So, to add a second
base_url
option for Iterable, you need to: - either fork the connector and run it as a custom Docker source, or - clone its low‑code manifest into a new YAML connector and extend it there, then point your connections to that custom definition.