Hello Everyone, I have a query regarding pipeline design. I must migrate data from the old database to the new one, keeping both live and sync. Both have different schema designs.
Both systems have ID and UUID as primary keys for a table. While migration, I need to keep the mapping somewhere, for historical data as well as new data.
I am creating Two Jobs,
For each event -
1. It takes an ID from the source database, adds a UUID on runtime, and sinks in a target mapping table. ( ID - UUID mapping)
2. For the second Job, read the Source table, Join it with the Mapping table to extract UUID and SINK in the target table.
I do not want my UUID ( generated for each event) to get lost while updating/deleting, so I created JOB 1.
Is it possible that I can run Job2 only after Job1 for each event?