Hi, I have a question to the dependency between 2 ...
# connector-developmentw
Will Sargent
02/09/2022, 8:23 PMHi, I have a question to the dependency between 2 API Streams in a source. I am using the Faros TypeScript CDK. I want to use email activities (schema link) and campaigns (schema link) as an example. An email activity is a child under a campaign (like users to teams). In the source stream definition, campaign_id is required to get the children activities. The final data schema contains everything under email activities PLUS campaign.type. My question is, the source connector definitely fetch campaigns, in order to get email activities. My question is, how can I reference another stream when the destination connector processes a stream? Or we should always treat the each writeStream independent?
m
Marissa Pagador
02/09/2022, 10:47 PM@Christopher Wu could you help here? 😄
Marissa Pagador
02/09/2022, 10:49 PMHi, Chris, thanks for the faros CDK, it’s very neat. I learn how to use reflection in typescript.
Marissa Pagador
02/09/2022, 10:50 PMI saw okta is implemented. What would you do if you want to extract users and groups from okta, and you want to output a type of user, which includes the user + a list of group names the user joins?
Marissa Pagador
02/10/2022, 12:03 AMHi Yiyang, glad to see you’re using our CDK! I need a bit of clarification on your question. Are you talking about implementing a source stream that depends on another source stream’s records to output its own records to a destination, or are you talking about implementing a destination that needs to combine records from multiple streams to output the final record?
Marissa Pagador
02/10/2022, 12:05 AMI am implementing a destination that needs to combine records from multiple streams. I saw the dependencies queue you created, that’s a witty solution for the first case you described.
Marissa Pagador
02/10/2022, 12:12 AMGotcha, so the destination converter (we’ll call it C) that is responsible for outputting the final record should specify the stream(s) it depends on like so: https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/faros-destination/src/converters/github/issues.ts#L22-L26
Then, for each record that C receives, it can fetch the corresponding record(s) from the streams it depends on, using the provided StreamContext like so: https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/faros-destination/src/converters/github/issues.ts#L62-L65
You can also fetch all the records from a stream that C depends on if you need to do more complex filtering logic besides a normal id lookup: https://github.com/faros-ai/airbyte-connectors/blob/bb850a6952b5b72c8bf49358266fb5[…]9796/destinations/faros-destination/src/converters/converter.ts
j
Jeffery Powell
02/10/2022, 9:38 PMwow, thank you! that’s really helpful.
Jeffery Powell
02/10/2022, 9:39 PMI saw every converter contains the id. Is it useful to de-dupe? Sorry, it’s another topic. Hope it’s really a quick question.
z
Zak Keener
02/10/2022, 11:11 PMYes if most or all of your converters will read ids from the same source record field you can just have your converters extend a base converter https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/faros-destination/src/converters/jira/common.ts#L68-L72 and then override that for any special cases
a
Antonio Grass
02/11/2022, 6:41 PMthat’s a good idea, we are thinking about that. We only want to support dedupe. The configured catalog for source includes
primary_key