Hi, we plan to use Pulsar for our massive HTML page generation. We generate HTML pages with page path as a message key and markup as an event value. Our volume is around 1000k pages with a 30kB size (~30GB per all-in-one generation). Once the pages are ready, they are exposed to a single topic and read by web servers. These web servers can start on demand - there can be 0 web servers for some time.
Our goals:
1. read all HTML pages (latest versions) by web servers at any point
2. keep the latest version of each page even when no web servers are connected (no subscription)
3. remove old messages (old pages)
What are your recommendations around message retention settings and topic compaction?