https://linen.dev logo
Join Slack
Powered by
Shots fired. <Warpstream> (ex-Datadog folks who bu...
# general
d
Shots fired. Warpstream (ex-Datadog folks who built Husky) are claiming kafka-wire protocol compatible message brokers over S3 without all the fuss of managing Kafka, with a P99 latency of ~1s, huge throughput, and cost reductions of 5-10x through eliminating network and infrastructure cross traffic.
šŸ˜„ 2
šŸæ 2
m
given how many times I've seen a file on s3 used as shitty streaming service I'm curious about this one.
g
high-latency / high-throughput is not a tradeoff that I've seen often in the space. Maybe they discovered an under-served set of use-cases.
c
ā˜ļø one of the authors of this blog post is a cofounder
m
I don't think it's high-latency/high-throughput that's the attraction. It's "ok" latency and cheap.
g
Yeah, I'm wondering when is 1s considered "ok". Of course, I hear more about the "Kafka is slow" cases, but 1s is far above anything I've seen.
m
I've heard far more "anything below 5 minutes is good enough" use cases
šŸ‘ 1
g
Probably biased, but I've ran into many more "200ms is unacceptable, happy to pay as much as needed for 50ms p999"
šŸ˜„ 1
interesting
yeah, I've seen them, but not on Confluent Cloud (or even not on Kafka)
m
for sure. those use cases are also far more valuable. but there's a large swath of folks that it doesn't make economic sense to pay for 50ms p9
g
We'll see how it goes. Definitely interesting to see someone exploring this side of the space
m
I also have a heavy bias from my time in the cloud, "shit streaming service" was the joke that wasn't really a joke. There's a TON of people using S3 as a queue
g
FWIW, if you are ok with 1s latency and don't have high throughput use-cases, Confluent Cloud Basic is quite a steal.
a
INFINITELY scalable? šŸ¤Ø
g
@Mitch can you point at anyone who wrote about using S3 as a queue? I thought I've seen everything, but not this one. I wonder what's the mechanics.
m
auto-scales! CROSS PLATFORM! RUST! all required keywords
d
I think there's a lot of people who are on Snowflake / Databricks and can't get <1min easily or cheaply enough (hell, under 20mins), so anything faster is going to be considered. I also run into a lot of people who say 'Confluent cloud seems to be best-in-class, but it's way more expensive than Aiven/MSK/Event Hubs/etc. for my use case' so they look around for other options because they can make the trade off of wanting something that will allow them to update their dashboards or review their logs within a few seconds of generation, but not cost them $25k+/year
c
Users that need super low latency are also usually users that complain a lot about things šŸ˜‰ The "silent majority" are fine with a lot less stringent requirements
m
g
@Mitch they are using SNS and SQS, yes?
m
Sometimes.
d
Consider the really simple use case of CDC from a postgres database, your P95 is going to be <10s without lots of tuning, so paying a bunch extra for the message bus to be <1s latency doesn't help you at all
šŸ‘ 1
g
I know about SNS and SQS. "S3 as a queue" implied that you have queue semantics with S3... They get the semantics from SQS and SNS
m
there's also a bunch that just write out files and something does a glob(*.json) every N seconds
or a lambda s3 trigger.
g
understood
d
I suppose the other valid question here is: are the customers going to be sticky and profitable where usecases are characterised by 'cheap, simple, and fast-enough'. My intuition is there's a lot of small operators who would like this, but that doesn't necessarily translate into piles of $25k+/y contracts
šŸ‘ 1
c
Isnā€™t s3 ā€œeventually consistentā€, and as such is there any implication to delivery guarantees?
a
Iā€™m still stuck on ā€œinfinitely scalable.ā€ What???
I must not be the target audience for this because my reaction to that is, thatā€™s completely ridiculous, and if you even suggest that, it means you probably donā€™t understand what it means.
c
https://docs.warpstream.com/warpstream/reference/kafka-protocol-and-feature-support Somehow they plan to support Txn's. Interesting. However, I see no mention of compacted topics. (You can tell, I do care a bit about Streams šŸ˜‰ )
g
@Alex Clemmer I think there's an implied fine-print about the exact dimension of scale. For example, I think of storage in S3 as "infinitely scalable" because I've literally never ran out (including some tests that involved petabytes).
a
I guess, but when I think of scalability of streaming systems I think of things other than ā€œwhether I ran out of storageā€
g
same same
a
Like, yes, very important. But not the only important thing.
g
their architecture is pretty unique, so I'm having trouble reasoning about likely bottlenecks
āž• 1
but... it is usually the control plane šŸ™‚
m
I still donā€™t really ā€œgetā€ the broker-ish thing
g
same
a
When you say that itā€™s unique, you mean the idea that theyā€™re basically building a stream-oriented data warehouse on top of s3 directly?
g
and its all serverless?
a
Yeah so itā€™s ā€œsnowflake but for streamsā€
g
Snowflake isn't Serverless
you pick specific machines for the specific DBs
a
Do you actually provision VMs for it?
g
Yes. They have an abstraction, but you choose a size of warehouse
šŸ¤” 1
d
yeah snowflake is definitely choose-your-own-infrastructure sizing, very typical of BYOC platforms.
a
Ok, I am not a ā€œserverless guyā€ or whatever but I do think of serverless as having some capacity controls normally.
But itā€™s billed incrementally.
g
So with Snowflake, you know that you can run out of compute at some point.
a
Got it.
g
And since Snowflake has storage/compute separation, this network can be a bottleneck (depending on exact data model)
d
I would say that serverless usually has limits imposed, but the user isn't typically on the hook to worry about hitting a scaling limit outside of those limits - it's the vendors resonsibility to understand their offering and be sure safeguards are in place. We had one recently where a user was procedurally creating new tables during test cycles, and not deleting them on completion - this caused us to realise we hadn't limited the number of tables an individual user could create and fix it.
šŸ’Æ 2
g
Exactly. # of tables is a very common "scale limit" that no one thinks about.
Or even "how many tables can I create and delete per second"
d
love those rate limiting problems
g
This was an explicit design goal of KRaft
Both much higher number of topics/partitions, but even more important - ability to add/remove them fast (we had customers with dynamic logic that needed hundreds a second... not something we initially anticipated!)
a
Anyway what I was trying to say was Iā€™m not sure what the bottlnecks are but because itā€™s built on S3 Iā€™m also not sure what the semantic guarantees are.
g
yeah
This is just the launch, so they will probably add information over time
a
With Snowflake there is more tolerance for correctness and freshness lapses because itā€™s not on txn hotpath. So is warpstream targeting analytics workloads or are they trying to actually power stuff where those things matter?
d
honestly it reminds me a lot of S3Guard - additional capability over S3. Eventually S3Guard was actually rolled into the underlying S3 protocol and the Hadoop stack retired it. I'm not saying that would be the end-game here, but the situation where an additional functionality is useful to a lot of people and implemented over a serverless offering feels similar.
g
They can't do transactional workloads at 1s latency
this is insane
I understand that CDC and analytics can be "5 minutes is fine". But not for OLTP.
a
That would be very surprising to me as well.
d
oh for sure I wouldn't use for this OLTP in any situations that spring to mind, but these latencies are fine for OLAP* For varying grades of OLAP
a
At least not if itā€™s ā€œinfinitely scalableā€
g
I'm currently fighting to get latencies from 11ms to 1ms P99. Because this is what OLTP needs.
So yeah, they have to be CDC+OLAP of some kind
Maybe more to compete with MaterializeDB than Confluent / Aiven.
a
oh you mean 1s latency for OLTP is not acceptable.
g
yeah
OLTP workloads often have a human staring at a browser window...
a
Eh, maybe! People will tolerate that for stuff that is extremely critical if it means you are doing something like replicating heavilt.
g
1s is considered too high
maybe
a
or ensuring consistency over huge geographies.
But I do not think they are doing that here.
g
I don't think you can trade off responsiveness to the end user
a
Mostly yes, but I think there are times people will choose that.
g
JetBlue comes to mind
Their app is so slow
a
I have seen people accept > 1s latency on user deprovisioning for truly global services.
g
yeah, I can see this
maybe specific parts of the experience
a
Yeah.
Anyway excited to see where it goes.
g
Often people end up with those massive global systems because they need low latencies. "Your website is unusable from India" is not uncommon...
a
I am a lot more optimistic about this than, say, RedPanda (with apologies)
g
Really?
interesting! we'll see
I'm still betting on Confluent šŸ™‚
āž• 2
ā¤ļø 1
a
Well you worked at confluent so you tell me
I think confluent will figure all that stuff out. Itā€™s way harder to compete with fundamentally better architectures. TBD if this is fundamentally better.
g
We non-VCs have two ways of betting: ā€¢ Our money: Shares in public companies ā€¢ Our time: Work for them or in their ecosystem I did both with Confluent. And I can do neither with RedPanda or Wrap šŸ™‚
a
Also, again, you are the expert, but the RP technical blog leaves me with many more questions than answers. I do not have this experience with Confluent.
g
interesting
I usually dig RP's tech blogs
But yeah, communicating well is its own skill, and Confluent invested heavily in this
I used to have all the endless blog reviews, but it does make for better outcomes.
a
I know it is more subtle than this but ā€œwe implemented this in raft and so we can be fasterā€ is not an argument I am prepared to buy without some indication they understand the subtleties of this claim.
g
They shared that they have C* architecture, right? This is 90% of the secret sauce
I think they did share all the good technical bits, but I agree it may be messy and hard to grok
Someone should write the RedPanda definitive guide šŸ˜‰
šŸ˜„ 1
a
So like: Raft is a very ā€œlinearā€ algorithm and is very sensitive to even tiny transient network blips, whereas paxos has very fine-grained units of consensus like ballots. So in Raft you get to build a second, secret, unspecified protocol on top of Raft to get it to be performant and not constantly livelocking.
No mention of this at all. Just raft is better.
g
hahaha, I see what you mean
c
I'm currently fighting to get latencies from 11ms to 1ms P99. Because this is what OLTP needs.
11 ms is quite fast already, for example, non-musical humans can't normally detect under 40ms. I'm assuming this is relevant for when you have many many requests behind-the-scenes to service one user request?
Also again, I'm not sure how they plan to claim they're a global service with transactional semantics given how S3 is built...
g
It is relevant when the founder is obsessed with benchmarks šŸ˜‰ I can't think of a real business reason here. But I want my product to be fast.
Clearly I'm on team RP and not Wrap šŸ™‚
c
šŸ˜„ ours is between 8-20ms, which is still as fast as it gets in terms of workflow engines, not sure if we could get any faster and still have 3-way cross-AZ replication
RedPanda?
a
I am going to guess that a huge % of RP gains (if they are real) are simply writing it again with perf in mind. That is a legitimate strategy. It is very hard to tell from their benchmarks where the gains really happen.
g
I meant in terms of mindset and culture
RP culture is clearly obsessed with latency
Wrap culture is "our thing is not latency"
c
Most of the RedPanda benchmarketing is compared to Kafka with fsync called on every single message, which is somewhat of an unfair comparison...
g
agree
I am not a fan of the way they benchmarked
a
So why team RP then lol
g
As a company, they think latency matter
c
Jack Vanlightly had a fantastic in-depth blog about Kafka vs RedPanda. Turns out that if you have 50 producers, Kafka does much better. RedPanda does better with 4 producers (one per partition) all writing GB/s, which is a very rare usecase
g
They built a new architecture that should in theory be faster
a
Compare to Neonā€™s blog: https://neon.tech/blog/paxos
šŸ‘€ 1
g
yeah, it is interesting
c
g
but makes sense
a
Itā€™s just very good. Detailed explanations of why it works, what role it plays, how they arrived at the decision, etc.
šŸ’Æ 1
g
There are fast variants for Raft
and everything is fast if you do multi-quorum
a
Totally.
Iā€™m just saying itā€™s subtle.
g
it is
a
Also they are not easier to implement than paxos or zab.
g
you are 100% right to worry about the lack of details
a
That RP seems to think it is, suggests they may not have done this before. šŸ™‚
g
I'm not sure about ZAB
a
well. personally I think there is a thermodynamic law for distributed state, which I will informalyl refer to as the ā€œconservation of painā€ law. You can shift it around but you canā€™t create or destroy it.
šŸ˜‚ 3
If you want raft to be performant, reliable, and not live-locking spuriously, it is definitely as hard to do as various types of paxos.
g
I believe in this in general, for any given problem requirements šŸ™‚
a
ha ha
I have to run but it is very interesting youā€™re team RP, I will keep that in mind. I still think Confluent can figure this out.
Architectural superiority is a lot harder IME
g
Some protocols are easier to trace and troubleshoot. And IMO, this property leads to more reliable systems faster.
ā€¼ļø 1
c
Is it settled though that RP performs better in practice? I really think it depends on which benchmarketer you believe.
g
This is one of the reasons I'll take Kraft over the old controller any day
a
<halfway out the door> yes that is right and itā€™s a very underrated point
operational predictability is worth like 50 architectural IQ points
g
@Colt McNealy They are faster for very large number of partitions. One can make the claim that you don't actually need this. So, as usual - it depends.
šŸ‘€ 1
It is all solvable, so I have no doubt Kafka (or at least Confluent Cloud) will catch up soon.
c
From Confluent's side there's theoretically nothing stopping them from writing their own Kafka-compatible thing in Assembly and swapping it out for Kafka in their cloud service.
g
Right.
c
However, after reading benchmarks from both companies I'm not sold that the difference (at least for my use-case) is TOO massive between RP/Apache Kafka. That said, I haven't yet played with RedPanda since it's not "free for production use"
g
for your use-case, I would definitely stick to Kafka
you rely on the latest improvements in Streams...
plus, you had a cool way to avoid large number of partitions
I would be shocked if your product actually runs successfully on RP
šŸ‘€ 1
c
Did we discuss that on the podcast? (Using one state store on each Streams Task for all types of objects, instead of doing the lazy thing and creating a new 
KeyValueStore<String, Foo>
and 
KeyValueStore<String, Bar
and 
...<String, Baz>
etc?
And I know we've discussed licenses in the past, but the RP license doesn't permit free production use (plus there's no Strimzi for RP), so that's a non-starter for me.
The Licensor hereby grants you the right to copy, modify, create derivative works, redistribute, and make non-production use of the Licensed Work. The Licensor may make an Additional Use Grant, above, permitting limited production use.
šŸ’Æ 1
m
What Gwen said is important. Last I looked RP didn't cover all the kafka verbs
r
@Alex Clemmer could you expand re warpstream vs redpanda? we use redpanda in production, but our scale is miniscule, itā€™s just looping transactional outbox events via cdc back to queue, so literally single-digit kilobytes per sec (that includes e2e kminion monitoring lol). but I am expecting it to grow
oh, yeah. you meant the raft part. iā€™ve also wondered how there was no comparison whatsoever and my mental image of raft is that itā€™s super chatty. but RP works well when you need latency and you need it to be lightweight for non-large scale
@Colt McNealy wait, what does non-production use mean? you can play with it, but not use in your project?
c
@Rauan Mayemir correct, that's what I believe it means...however, their website seems different than their github: https://redpanda.com/deployment-options?section=self-hosted I'm not 100% sure I know what they're saying with their license now that you ask about it.
a
@Rauan Mayemir I have no specific complaint, the entire technical blog and docs just kind of rubs me the wrong way.
r
@Colt McNealy yeah thatā€™s where my shock comes from. docs, website, slack say very different things
a
I have worked on both a production stream processing system that processed like, x00M events a day, and a leadership election system. And the whole thing just kind of makes me nervous.
The flip side is that I am NOT the target audience. I donā€™t think youā€™d ever in your right mind target people like me.
šŸ˜ 1
r
@Gwen Shapira Iā€™ve actually thought about warpstream use case where it would make sense, probably for telemetry or analytics data. but ā€˜1sā€™ write latency still seems outrageous to me. why not just use a regular kafka and sink stream into s3
g
if there's a lot of telemetry/analytics, cost of a Kafka cluster can still be high.
r
but with very short retention?
f
Thanks for giving me an insightful morning read! šŸ˜„
r
looked up the mentioned article. itā€™s a big deal, iā€™m surprised itā€™s not being discussed or disputed https://jack-vanlightly.com/blog/2023/5/15/kafka-vs-redpanda-performance-do-the-claims-add-up
ā€¼ļø 1
c
The Vanlightly piece mentions that Kafka does much better when Record Keys (sending to specific partitions based on key) are involved. And that is Streams. Also, supposedly RP doesnā€™t do as great after running for several hours. But the JVM is tried tested and trueā€¦ Iā€™ve got no skin in the game but it does feel like RP is guilty of benchmarketing.
m
What a thread.... this is awesome! Thank you all for all these insights and resources, I've learned much, as I'm mostly a user of these technologies, what a rush (and a pain!) to be a developer on these?! I wouldn't sleep at night thinking of all the lost data and bugs... I can see using Warpstream for very specific workflows, I wouldn't run any "regular" queue off S3 for various reasons, but for example we are using SQS in our organization, and we use it in some cases to pass large payloads (big for a message, not GiB => more like MiB), however - as many of you know SQS isn't very good at that, so we pass a link to S3 in the message, and our consumer lib knows to fetch the file (as if it is in the message as far as the consumer is concerned) - this can be a nicer solution maybe, but it seems very pre-mature, more of an alpha or so (tried the demo, it created a LOT of files). Anyway - keep this up! and thank you !
g
Fwiw, Iā€™m using SNS a bunch. Itā€™s built-in integrations and notifications are great. Iā€™m surprised no one is offering anything similar on Kafka.
c
@Gwen Shapira there's a KIP (which Matthias had many things to say about šŸ˜‚) to add Queue semantics to Kafka, is that what you're thinking of?
m
That Kip looked really familiar @Gwen Shapira
g
Itā€™s on the way to what I was thinking of, but a service that you can add ā€œrulesā€ with things like ā€œemail me if the message has field X with value over Yā€ is what Iā€™m really afterā€¦
Generally speaking, queue semantics are long long overdue (you can tell because there are at least 2 good projects that create these on top of Kafka)
@Mitch looks familiar?
m
That KIp would make my life easier
šŸ’Æ 1
m
Fwiw, Iā€™m using SNS a bunch. Itā€™s built-in integrations and notifications are great. Iā€™m surprised no one is offering anything similar on Kafka.
@Gwen Shapira you mean a service for notifications from Kafka or Over Kafka (or both)? Ahh, read a bit below - sure, but that's really a very basic stream processor, with some control plane to define the rules, and track work, right? Doesn't really require queue semantics, The only feature I'd put in before anything is a silent window - e.g. please don't bombard me with notifications (or at least aggregate them a bit) so I'm not flooded
r
Someone did think of the use case of delivering telemetry data via warpstream. But I still donā€™t get why use kafka protocol for this if itā€™s not fast, itā€™s just a bottleneck at this point. https://x.com/gortron/status/1693020979897782726?s=46&amp;t=jUOotRVS0rxQvp3AVXQlcw
m
@Rauan Mayemir I don't see how this serves anyone with a ms/s latency requirement - the alert/pagerduty/automated action (any thing the system does based on data stream) based on processing in the end system will already be late (maybe a few minutes isn't dramatic?). It sounds very nice for data moving, and file / artifact storage, but too slow for anything that needs speed (client UI/UX?)
Open in Slack
PreviousNext
Powered by