<@U04S8PLJWMT> is there any way to enable airbyte ...
# replication-ask-ai
a
@kapa.ai is there any way to enable airbyte mapping on airbyte open source version ?
k
Based on the knowledge sources, Mappings are not available in Airbyte Open Source (Community edition). The feature availability is clearly stated: > Not available — Core | Not available — Standard | Available — Plus | Available — Pro | Available — Enterprise Flex | Available — Self-Managed Enterprise [Mappings docs] Mappings (hashing, encryption, field renaming, and row filtering) were introduced in the API in version 1.3 and added to the UI in version 1.5, but exclusively for Self-Managed Enterprise and paid Cloud plans. [v1.3 release notes] [v1.5 release notes] Your options 1. Upgrade to Self-Managed Enterprise — This unlocks mappings along with other enterprise features. You can talk to Airbyte sales to get a license. 2. Use dbt for post-sync transformations — If you're on Airbyte Cloud, you can use native dbt integration for custom transformations. However, this is also not available for Open Source. [Core Concepts] 3. Handle transformations externally — In the open-source version, you would need to apply any field hashing, renaming, or filtering logic in your downstream data pipeline or warehouse after the sync completes. There is no configuration flag or workaround documented in the knowledge sources to enable mappings on the Community edition. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai can I create custom connector that do mapping like encryption on the fly while sync ?
k
Yes, there are a couple of approaches you can use to apply transformations like encryption within a custom connector during sync: 1. Connector Builder with Custom Components If you're building a source connector using the Connector Builder, you can use Custom Components to implement a
RecordTransformation
class in Python that encrypts or transforms fields on the fly as records are processed. [Custom Components] Here's the pattern:
Copy code
from dataclasses import dataclass
from airbyte_cdk.sources.declarative.transformations import RecordTransformation

@dataclass
class EncryptField(RecordTransformation):
    def transform(self, record, config=None, stream_slice=None, stream_state=None):
        # Apply your encryption logic here
        record["sensitive_field"] = your_encrypt_function(record["sensitive_field"])
        return record
This allows you to override built-in components with your own Python logic. [Custom Components] > Note: Custom Components are available in Core and Self-Managed Enterprise plans only. 2. Connector Builder Transformations (Hashing via Jinja) If you only need hashing (not full encryption), the Connector Builder supports a built-in
hash
filter via Jinja templating in the "Add field" transformation. For example:
Copy code
{{ record['sensitive_field'] | hash('md5') }}
[Record Processing] Limitations to be aware of • The Connector Builder approach works for source connectors — you're transforming data as it's read from the source. • This is not the same as Airbyte's native Mappings feature, which operates at the connection/stream level. • For full RSA encryption specifically, the knowledge sources do not explicitly document a built-in Jinja filter for it — you would need to implement it via Custom Components using Python. In summary, yes it is possible, but it requires building a custom connector and implementing the transformation logic yourself in Python via Custom Components. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.