Thanks for the detailed proposal, Jusang. I can't give you a maintainer decision on whether the change belongs in the HTTP provider or the fetcher engine, or whether a backport to

release/0.9.x

would be accepted. None of that is covered in what I have, so the best place for those answers is the discussion on your issue: https://github.com/permitio/opal/issues/990 What I can share is how fetching is structured today, since it bears on the placement question: • Fetch providers are the pluggable components OPAL uses to fetch data from sources on demand, and each is a Python class that derives from

BaseFetchProvider

. The fetching logic goes in

_fetch_()

, with optional post-processing in

_process_()

.(1) • Providers are loaded from the modules listed in

OPAL_FETCH_PROVIDER_MODULES

.(1) • The fetching machinery is split between the

FetchingEngine

, the

fetch_worker

, and

BaseFetchProvider

, which are separate modules in

opal-common

.(1) • On the data flow, OPAL clients fetch data directly from the source using instructions sent in the update event. The server sends those instructions rather than the data itself.(2) • The client then saves the fetched data to the policy store at the entry's

dst_path

, as seen in client logs like "Saving fetched data to policy-store".(3) Nothing in these sources says where per-source response state (such as a stored ETag) would live between fetches, or whether the engine or the provider is the right owner. That's the main design question for your issue. If you write against a custom provider, note that from OPAL 0.10,

opal-common

requires pydantic v2. This affects any new config or event models you add to a provider.(1) I can't confirm what this means for a 0.9.x backport, since the sources only note that 0.9.x and earlier ship pydantic v1.(1)

To learn more, see the sources I considered: [[1] How to extend OPAL with custom Fetch Providers](https://docs.opal.ac/tutorials/write_your_own_fetch_provider) [[2] FAQ | OPAL](https://docs.opal.ac/FAQ) [[3] How to detect all OPAL_DATA_CONFIG_SOURCES have been fully loaded at startup](https://github.com/permitio/opal/discussions/887)

Hi team, a follow-up on #990 (conditional requests for the HTTP data fetcher). Thanks again @david for the design notes. I’ve opened a PR that follows them: https://github.com/permitio/opal/pull/1001 • Opt-in flag on

HttpFetcherConfig

, GET only.

NotModified

/

ValidatedData

result types live in

opal_common.fetcher

. • The provider stays stateless. Validators travel on the fetch event, and

DataUpdater

keeps them keyed by url +

dst_path

+ a config hash. They are cleared in

get_base_policy_data()

. • The 304 is handled before

raise_for_status()

, with tests for both aiohttp and httpx. • External OPA: before trusting a 304, the updater checks that the store still has data at

dst_path

, and falls back to a full fetch if it doesn’t. There is also an optional max-age. • New

not_modified

field on

DataEntryReport

. The transaction is marked successful. I’ve put a few open questions in the PR description (the report fields on a 304, and how new fields should pass the wire_compat check). I noticed #998 was opened on the same issue before I got to it. From reading it, it seems to differ from the notes in a few places: • The validators are kept in module-level state in the provider. • The cache key is url +

dst_path

, without the config. • A 304 is reported as

fetched=True, saved=False

. • Nothing covers an external OPA restart. For context, #998 also looks like it may be automated: GitHub search shows the same account opening 200+ PRs across dozens of repos in about a day (190 on Nov 5 alone). You may want to take a look at it from that angle. I may well be misreading it, though. I’m happy to go with whichever direction you prefer, or to fold anything useful from #998 into mine.