How many events-per-second is enough for 'reasonable scale' for a demo these days? I'm running 10K over a Kafka topic to simulate a Prometheus feed of fake metrics to drive a Grafana table, which results in an events table of a few GB/day. The demo is to show how you'd iterate real-time data pipelines using git and CICD, so we're not proving that we can process data at scale because we do that elsewhere, but that you can do the hard thing of iterating streaming API endpoints and Tables without breaking production at a 'reasonable scale'. What is reasonable?

My ex-colleague has a startup for generating fake data to Kafka (or webhooks). I think he’ll know :) https://x.com/michaeldrogalis?s=21&t=qjOqD4DeQ6vjh9EW1KZD0g

I wouldn't be bothered by scale as much for a functional demo, I'd be interested in seeing my use-cases being covered and working (or the workarounds I can do to make them work). So if large messages might be a problem, I put some of those High throughput I show that (1000 events a second? Just throwing a number) - this also depends on what compute you are provisioning for the demo (so I'd like to see close to linear growth of capacity when adding a worker for example) I would publish some load benchmark and how you produced it, if someone wants to challenge your performance numbers, but such a test can run in the days/weeks not something you can really show in a demo

seems like an odd thing to post to a thread

wrongly posted removing