Hi everyone, we got an interesting problem including a TimescaleDB on top of a Postgres. Our largest table (>200GB) is partitioned on a specific event timestamp column, but not on it's "created_at" column. Now if we create a connector in Airbyte using "created_at" the replication is just terrible, as TimescaleDB could not properly optimize the query and will query all hypertable buckets (for us > 1000). My idea would be to create a more specific descendant of the Postgres Source, where the cursor field AND the hypertable dimension column can be specified. Does this make sense?

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2891 to ensure timely resolution!

Hey @Alexander Pospiech, I’m not too familiar with TimescaleDB so I’m unsure how helpful i can be here. We have an issue regarding the creation of a TimescaleDB connector but no one has taken it on yet. So your idea is to modify the Postgres connector to include an additional dimension column field? I don’t see a way to make it work with the current Postgres connector.

The issue is similar but different 🙂 We need a proper timescaledb source connector, not a destination. Yes, the current Postgres connector is unusable with TimescaleDB. I will create a more explanatory issue for that, as we already thought about a solution.

Sounds good, feel free to make an issue on our Github explaining what your proposed solution is.