Hi Greptime team 👋 I’m trying to use GreptimeDB in production on AWS, but without Kubernetes (no operator). I’d really like to go this route, but I’m not 100% sure what the best non-K8s setup is (especially on OSS). Open questions: 1/ With S3 as object storage, what must be persisted locally on datanodes for correctness/recovery (WAL, etc.)? In practice on AWS, would you recommend EBS or instance storage for that? 2/ On OSS, when a datanode is permanently removed, does Meta automatically re-assign region leadership / redistribute regions, or do we need to run admin commands as part of scale-in? 3/ Maintenance mode: docs mention using it for temporary node unavailability (upgrades/rollouts). Should it also be used as part of scale-in/out procedures when not using Kubernetes? 4/ If you have a “known good” reference architecture for AWS without Kubernetes, that would help a lot. I'm considering to run a cluster on ECS/Fargate (+ S3 + RDS), but I’m unsure about the operational procedure for safe rollouts and scale-in/out. Thanks Davide

Hi Davide, 1. Yes — the WAL must be persisted locally. On AWS, we recommend using EBS, since data stored on instance storage will be lost after a node restart (unless data loss of unflushed WAL entries is acceptable). 2. For OSS deployments, you’ll need to use the region migration command to reassign regions when a datanode is permanently removed. 3. Yes, you should also use maintenance mode as part of your scale-in/out procedures when not running under Kubernetes.

Ok, thanks super helpful! I've got a few follow-up questions 1/ What’s the real-world scenario where a lost WAL becomes an issue? Don't the other nodes still have it? 2/ If I ignore the EBS advice and run datanodes on ephemeral disk, what’s the real bad outcome you’re warning me about: do I risk losing recent acknowledged writes, or is it “just” slower recovery / more manual steps after a restart? 3/ Wouldn’t OSS still do an automatic failover when a node is lost/unreachable, removing the need for manual migration? 4/ I don't get the reason here. What specifically happens if I don’t enable it during scale-out / scale-in? What are the risks?

1. Yes, but it depends on your business — whether a small amount of data loss is acceptable. Frontend nodes are stateless, so they don’t keep WAL data themselves. 2. You’d risk losing recently acknowledged writes. once a datanode restarts, any WAL data stored locally is gone. 3. Yes, if region failover is enabled, the system will automatically handle it — you don’t need to migrate regions manually. 👉 Region failover documentation 4. During scale-out or scale-in, the cluster state can become unstable. This may cause the MetaSrv to make wrong decisions — for example, failing over a healthy region.

Thank you again. I understand that frontend nodes are stateless and don't keep WAL data. But what about the other datanodes? In case one node fails, wouldn't WAL data still be on the other nodes?

Ok, I see. No, only the region leader writes to the WAL, so there’s only one copy of it. Other nodes don’t keep a copy of the WAL.

If the regional leader goes down for, say, two minutes, MetaSrv will automatically assign a different node as leader for those regions so writes can continue. When the original leader comes back, what happens then? The new leader has been writing and has its own WAL now

When the original leader comes back, what happens then? The new leader has been writing and has its own WAL now

The original leader will become a follower. After failover, if it returns, it will be closed.

I see, thanks. When you say the original leader “returns” and will be “closed”: 1/ What happens to the WAL data that was on the original leader’s disk for that region? Is it discarded/ignored once failover has happened, or is any of it ever replayed/used when the node comes back? 2/ More generally, in HA with region failover enabled, do you guarantee that acknowledged writes are not lost? Or is it possible to lose acknowledged writes in some failure cases (even with WAL on EBS)? I’m asking because if HA mainly guarantees availability (reads / continued service) but not durability of acknowledged writes, I might prefer a simpler single-node setup on durable storage.

Is it discarded/ignored once failover has happened?

Yes, it will be ignored once failover has occurred, even if the regional leader comes back online later.

guarantee that acknowledged writes are not lost

We have two types of WAL: 1. Local WAL (disk-based) 2. Remote WAL (Kafka-based) Only when using Remote WAL can we guarantee that acknowledged writes will not be lost. This is because the Local WAL maintains only a single copy on one datanode backed by EBS, and multi-attach is not supported. As a result, it does not provide redundancy. In contrast, the Remote WAL leverages a shared Kafka topic with multiple replicas across datanodes, ensuring redundancy and durability. Therefore, acknowledged writes are guaranteed not to be lost.

If Local WAL can’t guarantee acknowledged-write durability anyway, what’s the concrete benefit of putting it on EBS vs ephemeral? Also, is Remote WAL through Kafka the only supported path today, or is there any roadmap for Local WAL redundancy?

> If Local WAL can’t guarantee acknowledged-write durability anyway, what’s the concrete benefit of putting it on EBS vs ephemeral? If Region Failover is not enabled and recovery relies on Kubernetes restarts, then using EBS is still meaningful. When Kubernetes restarts the pod on the same node, all previously acknowledged writes on the datanode remain intact because the WAL persists on EBS. However, if Region Failover is enabled (i.e., the failed region is migrated to another datanode), then the WAL must be accessible across multiple datanodes. In that scenario, simply placing the WAL on node-local EBS is insufficient, since it cannot be shared across nodes.

1/ If region failover is disabled, does that mean there is no automatic leader switch for those regions (so writes to those regions stop) until the original datanode is back (or we manually migrate)? 2/ If region failover is enabled, how long does it usually take to kick in, and is that timing configurable (timeouts/heartbeats)? In the meantime, are writes to those regions rejected, or do they block/retry until a new leader is ready?

until the original datanode is back (or we manually migrate)

Yes

and is that timing configurable (timeouts/heartbeats)?

It depends on the heartbeat configuration. After 3 missed heartbeats plus a 1-second buffer, the region is considered failed. Typically, it takes about 10–30 seconds to complete a region failover. If the connection is hard-dropped, Detection may take longer, with an additional delay of around 10 seconds.

In the meantime, are writes to those regions rejected

Yes. Writes will be rejected/failed during that time. The frontend does not automatically retry user writes, so your write agent needs to implement retry logic.

Let’s see whether I understand this correctly: 1/ If

region_failover

is enabled, we have a partial write outage of ~10–30s (maybe more) until failover completes. Client requests are rejected (no automatic retry by frontend). More importantly, even if the original node comes back 2–5 minutes later, its local WAL is ignored. So with Local WAL, we can still have acknowledged write loss in this mode (EBS vs instance storage doesn’t change the “ignored” part), and the only way to guarantee “no acknowledged write loss” is Remote WAL (Kafka). Is that correct? 2/ If we don’t use

region_failover

, we avoid that specific loss mode, but we have a partial write outage for as long as it takes to bring the original node back (or until we manually migrate regions). If the node comes back, it can recover from its local WAL, which is why EBS matters here. Is that correct? 3/ Does this also mean we’ll experience partial write outage during node upgrades (instance size / GreptimeDB version), since writes to the regions hosted on that node would be rejected while it restarts? If so, in Kubernetes deployments, what’s the usual procedure (maintenance mode / suppress failover vs allow failover), and roughly how long does that write outage typically last in your experience?

1/

Yes

2/

Yes

node would be rejected while it restarts?

Yes

what’s the usual procedure (maintenance mode / suppress failover vs allow failover), and roughly how long does that write outage typically last in your experience?

For now, we recommend using maintenance mode. The write outage duration depends on the DataNode restart time and typically ranges from about 30 seconds to a few minutes.

1/ This Local WAL behavior seems different from what I’d expect from a common Raft consensus logic. Are we misunderstanding it? 2/ Are there any plans to make failover “safe” with Local WAL without requiring Kafka (Remote WAL)? E.g., some form of WAL replication / quorum / configurable ack policy (similar spirit to ClickHouse insert quorum). 3/ Does any of the above change depending on the ingestion interface? (SQL over Postgres/MySQL vs batch ingestion): are SQL writes also rejected the same way during failover/restart, and do they follow the same “acknowledged write” durability rules? 4/ Separate but related: for SQL clients, in your experience, is the MySQL or Postgres interface generally faster (ingestion + queries), or are they comparable?

1/ Local WAL vs common Raft expectations Yes — Local WAL behaves differently from a typical Raft-style consensus implementation. In our design, Local WAL is not replicated to multiple followers before acknowledging the client. Writes are persisted only to the current leader’s local disk/WAL, rather than being submitted/committed through a follower quorum the way Raft normally does. 2/ Plans to make failover “safe” with Local WAL (without Kafka/Remote WAL) At the moment, no — we don’t currently have a plan to make Local WAL failover “safe” in the sense of guaranteed durability across failover without Remote WAL (Kafka). A “safe” Local WAL failover would require some form of WAL replication / quorum commit / configurable ack policy (similar to insert quorum concepts). That is not supported today. 3/ Does this change by ingestion interface (SQL vs batch ingestion)? The durability rules depend on the ingestion path: • SQL writes (Postgres/MySQL protocol): these writes still rely on WAL. If you enable failover while using Local WAL, there is a window where data that was acknowledged to the client may still be lost if it hadn’t been fully flushed in a way that survives leader loss (because the WAL is not replicated). During failover/restart periods, SQL writes can be rejected/unavailable until the system finishes leader recovery / becomes writable again, and they follow the same acknowledged-write durability limitations described above under Local WAL. • Batch/Bulk ingestion: this is different — it flushes data directly to object storage. In practice, the “acknowledged write” durability here is tied to object storage durability semantics (i.e., once the object store write is confirmed, it’s durable in that storage layer). 4/ MySQL vs Postgres interface performance (SQL clients) In most real workloads they are comparable

One thing I’m still confused about: 1/ I write to the region leader and get an ACK 2/ I read from a follower for the same region right after Am I guaranteed to read the latest updated data, or can follower reads be stale? If they can be stale, what’s the expected behavior / how do I ensure a read hits the leader when needed?

Yes, reads from followers can be stale.

Then, how can I make sure to read from leader when needed?

We provide a

READ_PREFERENCE

option to control whether reads are directed to the leader or to followers. https://docs.greptime.com/enterprise/read-replicas/query-read-replicas/

Is setting

READ_PREFERENCE

= ‘leader’ a “preference” or a “strict rule"?

The strict rule