Hi everyone, We are using StarRocks PIPES for a re...
# questions-and-troubleshooting
s
Hi everyone, We are using StarRocks PIPES for a real-time Databricks sync use case. Recently, we’ve started encountering row/batch ingestion failures. From initial investigation, we suspect the issue might be related to how Parquet files are written to S3. In some cases, the partition column (used in the S3 path) is also getting written inside the Parquet file schema itself. This appears to be causing a duplicate column conflict during ingestion. We’re seeing errors like the following: Error:
Access storage error. Error message: Duplicate column name 'DH_CITY_NAME' in files table schema
The schema in the failed task shows
DH_CITY_NAME
appearing twice. Has anyone working with StarRocks PIPES encountered similar duplicate column issues, especially when syncing Parquet files from S3 with partition columns? Would love to connect and share more details if you’ve seen something similar. Thanks in advance!