https://linen.dev logo
Join Slack
Powered by
Hi, We are a non profit with constituent data spli...
# general
b
Hi, We are a non profit with constituent data split across many different databases. Donor data, even registration data (with many departments handling this in their own databases), etc. Our goal is to unify our donors (households, and individuals) into one data warehouse, and also do a reverse ETL, so that changes from one system can reflect in others, once constituents are connected. Is this a use case for rudderstack? How can rudderstack help us here (if at all)?
h
That's a great use case for rudderstack
b
where would be the appropriate place to do deduplication? a transformation?
also the schema we need doesn't match the "warehouse" schema, is that an issue? does it matter?
h
what warehouse are you using
it doesnt have to match since you can map the data fields
I would de-dupe in the warehouse, get the data streamed in and join/merge/dedupe into views
b
We're using postgres
What does it mean to "dedupe in the warehouse". It's a postgres db
h
hmmm postgres doesnt have the best options (wouldnt call it a warehouse imo)
you can create mat views
reverse etl from this deduped view of clients/customer/whatever you call them
b
Deduping in our case requires a lot of complicated logic...like we compare contact info, names that might appear in the wrong field (nickname vs formal name), etc
h
do you use dbt?
b
Nope I'm just a lowly software engineer, not a data engineer :)
h
time to learn then 🙂
Open in Slack
PreviousNext
Powered by