@kapa.ai I am building a source connector using the connector builder. Using apple API I get a file TSV file but it had metadata on the first 3 lines, the CsvDecoder then got the wrong header. how can i skip the first 3 lines in my parsing ?

New message text here

@kapa.ai You gave me the s3 connector documentation so it is wrong do you have official technical doc on CsvDecoder to heck if it can skip ?

You're right to push back on that. Looking at the official documentation for the CsvDecoder component used in the Connector Builder, here is what it actually lists: > CsvDecoder — Select 'CSV' for response data that is formatted as CSV. Can specify an encoding (default: 'utf-8') and a delimiter (default: ','). > Properties: > •

encoding

(string) > •

delimiter

(string) > •

set_values_to_none

(array) [YAML Reference] That's the complete list of properties documented for the

CsvDecoder

component. There is no

skip_rows_before_header

or equivalent option available in the Connector Builder's CsvDecoder. So unfortunately, based on the official documentation, the CsvDecoder does not support skipping rows before the header. If you need to handle metadata lines at the top of your TSV file, the CsvDecoder alone won't be able to do it natively. You may need to consider a different approach, such as a custom Python connector or using a file-based source connector (like S3 or GCS) which do support

skip_rows_before_header

. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai can i easily create this connector and use it in airbyte cloud ?

Based on the conversation context, you're asking whether you can easily create a custom connector (that handles TSV files with metadata rows) and use it in Airbyte Cloud. Here's what the knowledge sources say: Using Connector Builder on Airbyte Cloud Yes, you can use the Connector Builder directly in Airbyte Cloud. The process is: 1. Access Connector Builder in the Airbyte web app by selecting "Builder" in the left-hand sidebar 2. Build and iterate on your connector 3. Publish it to your workspace 4. Configure a Source based on the released connector 5. Use the Source in a connection to sync data [Connector Builder overview] However, your specific use case is the challenge As established in our previous conversation, the Connector Builder's

CsvDecoder

does not support skipping rows before the header. Since your Apple API returns a TSV with 3 metadata lines, the Connector Builder alone won't handle this natively. Your options: • Python CDK connector: You can build a custom Python connector with full control over response parsing (including skipping rows), but deploying custom Docker-based connectors to Airbyte Cloud is more restricted — the knowledge sources note that "Custom images are something you could more easily ship to an OSS version, Cloud I believe is more strict." [Forum: Adding new low-code connector] • Airbyte OSS (self-hosted): Custom Docker-based connectors are more straightforwardly supported. [Custom Connectors] In summary: building a simple connector via the Builder and deploying to Cloud is easy, but your specific TSV-with-metadata use case goes beyond what the Builder's CsvDecoder supports, making it more complex — especially on Airbyte Cloud. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai based on our conversation can you write me a ticket explaining why i have this issue, why I am blocked and what I need (a parameter in

CsvDecoder

to skip the first lines)

Here's a ticket you can use based on our conversation: ------------------------- Feature Request: Add

skip_rows_before_header

(or equivalent) to

CsvDecoder

in Connector Builder Summary The

CsvDecoder

component in the Airbyte Connector Builder does not support skipping rows before the header row, blocking users from correctly parsing CSV/TSV files that contain metadata lines at the top. ------------------------- Problem Description When building a source connector using the Connector Builder against an API that returns a TSV file, the response includes metadata on the first 3 lines before the actual header row. The

CsvDecoder

reads the first line as the header, resulting in incorrect schema detection and failed parsing. ------------------------- Root Cause The official

CsvDecoder

component only exposes three properties [YAML Reference]: •

encoding

(string) — default:

utf-8

•

delimiter

(string) — default:

,

•

set_values_to_none

(array) There is no parameter to skip rows before the header. This is in contrast to file-based source connectors (S3, GCS, Azure Blob Storage, etc.), which do support a

skip_rows_before_header

option [CSV format settings]. ------------------------- Impact / Blocked Users building API-based connectors via the Connector Builder that receive CSV/TSV responses with metadata headers are completely blocked from correctly parsing those files. There is no workaround available within the Connector Builder. ------------------------- Request Add a

skip_rows_before_header

parameter to the

CsvDecoder

component, consistent with the behavior already available in file-based connectors, where it is defined as: > "The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field." Optionally, also consider adding

skip_rows_after_header

for parity. ------------------------- Proposed YAML Example

decoder:
  type: CsvDecoder
  delimiter: "\t"
  encoding: "utf-8"
  skip_rows_before_header: 3

------------------------- Priority Justification Without this feature, users must resort to building a full custom Python connector, which is significantly more complex and has deployment restrictions on Airbyte Cloud. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.