A cloud native embedded storage engine built on object storage.

SlateDB

https://crates.io/crates/slatedb/0.8.1

Though rust is somewhat fine for this even with foreign errors, exceptions are hell for this, exceptions popping from anywhere in the stack…

<@&1303423411908903005> nudge:

https://github.com/slatedb/slatedb/pull/849

<@283949658655555584> https://x.com/criccomini/status/1963684996809621735

(oh i see, it's a thread with the highlights! I will check it out in a sec)

Yes, sorry I was away from the keyboard. It’s a summary.

I took a crack at target GH issues for 0.9.0:

https://github.com/slatedb/slatedb/milestone/10

LMK if there are any others you think should be added.

there's no double-ended iterator in slatedb eh? was wondering if I could get the last key that matches, in a scan.

https://github.com/slatedb/slatedb/issues/438

the PR implementing it doesn't seem too bad, but looks abandoned hehe

Ya it’s dormant. Want to pick it up? I think it shouldn’t be too bad

maybe, it's just a big distraction from my current work and I can probably work around it.

I'll pull it into 0.9 anyway .. I suspect we can find someone to take it

the PR implementing it doesn't seem too

https://github.com/slatedb/slatedb/pull/530

Somewhat-not-very-technical, It'd be nice to maintain a list of officially supported object storage providers. I feel it'd probably be a valuable piece of info

From my end I have

Ok:
- Aws s3 
- Azure Blob
- Cloudflare R2

Somewhat ok (but borderline)
- Hetzner object storage

Ko:

- Linode
- Backblaze b2
- Alibaba OSS (Ok through opendal, but that's a workaround)

Ya, this is a good idea. Probably can go in the readme

Linode returns 404s on objects after put, even though they claim strong consistency

Awesome. Lemme update the docs. For Hetzner: what makes it borderline? Just their crappy network?

It throws a ton of errors and sometimes objects are fast, sometimes they drip at kilobytes a second

<@283949658655555584> reported that wasabi had consistency issues but I was not able to reproduce myself

might've been more related to my latency issues

I don’t think so, because this epoch message is very suspicious

Getting code examples for each of them in the getting started page would help with adoption

Cannot really do that easily, region names are not standard, some needs account ids in the endpoint...

Garage s3 is not ok too https://github.com/Barre/ZeroFS/issues/121#issuecomment-3259393364

(Forgot to mention it but minio is fine, Ceph + radosgw is fine too)

Somewhat-not-very-technical, It'd be

https://rclone.org/commands/rclone_serve_s3/

I'm not totally clear what this is. It lets you serve stuff through the S3 protocol? So like.. SFTP, https, HDFS, etc?

So if it works well it can get interesting

Yes but it’s less friction to me, I have rclone installed almost everywhere and it’s available in pretty much every package manager

https://github.com/Barre/ZeroFS/issues/143

Hmm, this one is a bit weird as the retry policy seems to be ignored

Haven't dug in, but we limit retries to five minutes.

We don't retry those I think! Gotta fix that

And the thing I describe with the PUTs returning errors but succeeding is a big pain

So that you can test failure modes against their service

Yea, this should get fixed when we add the object metadata tag and have the writers double check before propagating the failure

To me hetzner look a bit like a logistics company trying to do software

And they probably don’t know how to actually fix all this mess

What I am not sure to get is why “manifest exist not retrying” (the warn) is not fatal then

Because code above it catches the error, updates the in memory copy, and writes to the next id

(The compactor has updated it so the writer is not fenced even though the file exists)

https://github.com/Barre/ZeroFS/issues/

<@&1303423411908903005> I need 👀 on this one. DST is failing every night and it's annoying.

https://github.com/slatedb/slatedb/pull/862

Finally, got it again (after your commit that adds debug) <@839536195603922974> 

https://github.com/Barre/ZeroFS/actions/runs/17526643026/job/49777996250#step:6:68

It only ever happened on MacOS, so there's probably something suspicious going on with the storage

I wonder if it's not related to https://github.com/minio/minio/issues/21526

I reverted to an older version and it fixed it, scary...

https://github.com/minio/minio/issues/21526#issuecomment-3263651630

https://github.com/minio/minio/pull/21567

I think GCP has a similar thing. I saw some note once about conditional check on multipart in GCS. I actually have a GH issue open to check it.

If you look at the PR I linked they ack-ed that it’s a bug

Finally, got it again (after your commit

Hello folks,
We are trying to benchmark the scan_with_options endpoint for our use-case to use slatedb as an intermediate cache.

However, the read throughput is getting capped at around 320MB/sec with CPU being the bottleneck.

Some configuration pointers would really be helpful to improve the per reader throughput?

Few questions:

1. What's youre target throughput?
2. Can you share your configs?
3. When you say "reader throughput" are you using a `DbReader`? Or are you calling `scan_with_options` on the writer client?
4. Are you using compression?

(In general, I would say 320MB/sec is what I've experienced as near-peak load as well)

The main way to speed up reads is with caching...

cc: <@460026905685524501> could provide more details on the configurations.

Hi <@839536195603922974> !
1. Generally, we are evaluating against some other options and trying to see how close we can get to raw S3 throughput. For a single scan, the read throughput is actually even less (30 - 50 MB/s), so the first step is to optimize the single scan case.
2. Configs: defaults for everything except the block size, which is 16KB and flush interval, which is 3 seconds. I did try with and without an object cache on disk but did not notice any changes.
3. We have been using both methods (`scan_with_options` on both DbReader and the writer client)
4. We are not using compression.

Additionally, scan_options -> [durability: inmemory, readahead: 4mb, cache_blocks:false]

> The main way to speed up reads is with caching...
I did have a few questions on this as well. As I understand there is an inmemory cache and an (optional) disk cache for object store? I did see a lot of s3 calls even after setting an object cache, is there a setting we might be missing in scan_options / object store config?

> I did have a few questions on this as well. As I understand there is an inmemory cache and an (optional) disk cache for object store? I did see a lot of s3 calls even after setting an object cache, is there a setting we might be missing in scan_options / object store config?
The on-disk cache starts empty on the first run. As you scan, it persists chunks of the S3 objects locally on disk. These chunks are then re-used for future scans (even after restarting the process, and can also be shared across processes).

So if your test starts with an empty cache for the first time and does a scan, the caching won't do anything for you. What exactly is your test doing?

The test starts by first writing a set amount of data through slate (1Gb/10Gb with items either 1kb/10kb). Then we all that data back through scan. We are trying different scan patterns, such as:
1. Single scan on writer.
2. Multiple concurrent scans on writer.
3. Single scan with `DbReader`.
4. Multiple concurrent scans on `DbReader`
5. Multiple instances of `DbReader` (on different pods) doing concurrent scans.

You might consider a much higher readahead. IIRC, we default to 4MiB object chunks, so you would only be front-running one chunk. A 4MiB chunk would mean you are only fetching one chunk ahead.

I do see that we're hard-coding the scan iterator to just 1 fetch task, though:

```
let sst_iter_options = SstIteratorOptions {
    max_fetch_tasks: 1,
    blocks_to_fetch: read_ahead_blocks,
    cache_blocks: options.cache_blocks,
    eager_spawn: true,
};
```

You might try bumping this up locally to see if it improves throughput (if so, we can expose this through a ScanOption)

Gotcha. I did initially have a 128 mB readahead but it was leading to the process crashing due to `OutOfMemory` with multiple readers. Overall, we also want to support a decent number of concurrent readers.
Ack on `max_fetch_tasks`, will try it out, thank you!

Gotcha. What version are you running? You might benefit from this PR for your use cases (1) and (2):

https://github.com/slatedb/slatedb/pull/747

It is part of 0.8.

We are using the master branch from about 3 weeks ago. Will update.

K.. that PR is right on the border then 😛

Take a look at it; it requires some configs to use (preload and whatnot)

Hello folks,

<@839536195603922974> I'm updating your old p99 slides for a SlateDB talk next week (happy to share my fun new stylized slides/diagrams when ready) and was wondering if you had done anything new beyond the nightlies for performance numbers I can share. Otherwise I can just quote the old number from p99 conf