Hi, I have a question to the dependency between 2 ...
# connector-development
Hi, I have a question to the dependency between 2 API Streams in a source. I am using the Faros TypeScript CDK. I want to use email activities (schema link) and campaigns (schema link) as an example. An email activity is a child under a campaign (like users to teams). In the source stream definition, campaign_id is required to get the children activities. The final data schema contains everything under email activities PLUS campaign.type. My question is, the source connector definitely fetch campaigns, in order to get email activities. My question is, how can I reference another stream when the destination connector processes a stream? Or we should always treat the each writeStream independent?
@Christopher Wu could you help here? 😄
Hi, Chris, thanks for the faros CDK, it’s very neat. I learn how to use reflection in typescript.
I saw okta is implemented. What would you do if you want to extract users and groups from okta, and you want to output a type of user, which includes the user + a list of group names the user joins?
Hi Yiyang, glad to see you’re using our CDK! I need a bit of clarification on your question. Are you talking about implementing a source stream that depends on another source stream’s records to output its own records to a destination, or are you talking about implementing a destination that needs to combine records from multiple streams to output the final record?
I am implementing a destination that needs to combine records from multiple streams. I saw the dependencies queue you created, that’s a witty solution for the first case you described.
Gotcha, so the destination converter (we’ll call it C) that is responsible for outputting the final record should specify the stream(s) it depends on like so: https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/faros-destination/src/converters/github/issues.ts#L22-L26 Then, for each record that C receives, it can fetch the corresponding record(s) from the streams it depends on, using the provided StreamContext like so: https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/faros-destination/src/converters/github/issues.ts#L62-L65 You can also fetch all the records from a stream that C depends on if you need to do more complex filtering logic besides a normal id lookup: https://github.com/faros-ai/airbyte-connectors/blob/bb850a6952b5b72c8bf49358266fb5[…]9796/destinations/faros-destination/src/converters/converter.ts
wow, thank you! that’s really helpful.
I saw every converter contains the id. Is it useful to de-dupe? Sorry, it’s another topic. Hope it’s really a quick question.
Yes if most or all of your converters will read ids from the same source record field you can just have your converters extend a base converter https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/faros-destination/src/converters/jira/common.ts#L68-L72 and then override that for any special cases
that’s a good idea, we are thinking about that. We only want to support dedupe. The configured catalog for source includes
, it’s a list of keys. We want to join that configured keys to generate the id.