I mistakenly used a transformer and lost some data. How do I backfill this?

Hey There! 👋 Your message has been received by the RudderStack team. Our standard customer support hours are 9-6 PM EST, but we will forward this request to your Technical Account Manager, and they will get back to you shortly. Please use the thread for any additional comments.

We would need more context on this. What did you do and what data was lost?

@quiet-wolf-72320 We still need this data. Can you provide us with an export of the last 30 days of data ingested? I can work with that.

Hey @able-controller-17526 👋🏽 To quickly hop in here — Like Akhilesh said event replay is enterprise only feature. That being said let me see internally if we have any of that data — RudderStack in principle doesn’t store data once it has been processed.

thanks @delightful-coat-13923, as far as I understand, given we have set up a 30-day data retention this should be available somehow, right?

Apologies for that! Yes even though you have that selected — The tier you are on we drop tables as soon as the events are processed. Sometimes there might be delays where this might help but otherwise for our Starter customers we unfortunately don’t store any data after those have been processed 🙏🏽 And checking internally that is the case here as well — Unfortunately We do not have any data stored for y’alls workspace.

@delightful-coat-13923 I'm confused here. Does this mean that the only way for us to save this data is to have the 'Gateway dumps' in our plan (that is, upgrading to Enterprise)? I was assuming we were safe due to that 30-day rolling window. This is bad and unclear. The same docs even mention in Step 3 that 'choosing this options allows Rudderstack to store and delete your event data on a rolling 30-day basis'. What's even worse is the section you linked to is only available at the bottom of the docs and not expressed clearly within the Settings UI. We just lost a week of data assuming this was clearly being saved.

@able-controller-17526 We understand and we are working on fixing this in the UI to make it more clear.🙏🏽 That being said — Event replay is a feature we support for our enterprise customers only and we do mention that in our docs. In our Data Management docs, we have added a few notes saying that all this depends on the plan you are on but I can take that back to our team and be more clear here.

I don't want Event Replay, I just want an export of the last 30 days. I can do the joins my self TBH

@able-controller-17526 We only store data for enterprise customers for event replay feature, In general RudderStack does not store any data after it’s been processed. And Yes, You can connect your own bucket to store Sample events/responses and Proc Errors but gateway dumps are available on Enterprise customers only.

Thanks for the clarification

Thank you! We will definitely work on making this more clear 🙏🏽

Going forward, given on the starter plan we got no clear option to save this data (gateway dumps), would you say the best option is to connect a second warehouse and connect everything there aswell without any transformations?

Yes! You could definitely do that. And you could connect your own S3 for Proc errors i.e any failed events that happened on the user transformation side. One thing to note — Not like that would happen but if you then had a User transformation on that warehouse then those events would have been dropped as well. But i am sure that wouldn’t be the case, i just wanted to explicitly call that out.

yep, I understand, this second warehouse for 'gateway dumps' wouldn't have any transformations attached to it, in order to not drop any event I'll still setup the S3 bucket for proc errors and failed events

That sounds good! I’ll share our docs for Amplitude as a source here: https://www.rudderstack.com/docs/sources/extract/amplitude/