Hello, I'm developing a connector and I have a tri...
# connector-development
k
Hello, I'm developing a connector and I have a tricky operation to do : • I have a stream that returns a list of objects, each one has an id and some statistic fields • The statistic fields are wrong, but I can get the right values by calling another endpoint • This other endpoint allows to batch requests for multiple object ids I'm not sure how I could implement that in a simple way, do you have any recommendation about it ? Maybe a connector that faces similar issues ? Or some doc that could help me ? Thanks 😁
m
Airbytes philosophy of ELT kind of makes the assumption that you use Airbyte to replicate data raw, and handle transformations in a downstream layer. Personally, I would either make the first stream ignore the statistics fields that are wrong, and add the other endpoint as another stream (minor transformstion in airbyte layer). OR I would just include the wrong fields as is, and tell your downstream analysts not to use them (no transformation in airbyte layer but theres a risk people use the wrong data somewhere else)