1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by majisourav99

<https://github.com/linkedin/venice/commit/2bb458c1e49798a7e12ca2a8f0536e5d896e05f2|2bb458c1>

- [controller] Check for ERROR/KILLED status in request topic (#2256) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by sushantmane

<https://github.com/linkedin/venice/commit/91ba0cd3d7cf74029f1bced4ec802332cf9dc32f|91ba0cd3>

- [server] Add latency breakdown to Helix state transition completion logs (#2255) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by xunyin8

<https://github.com/linkedin/venice/commit/d88dcb6fc2e7a306c816165317a7c9c6e2c5f496|d88dcb6f>

- [protocol][common][controller] Write new version swap messages for CDC consumers (#2245) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by m-nagarajan

<https://github.com/linkedin/venice/commit/55e649cb5707620d23ab674d84740c25407825c0|55e649cb>

- [router] Add more aggregation to pending_request_count and add more metrics for host selection (#2258) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by ymuppala

<https://github.com/linkedin/venice/commit/761099dd124f840efeab96af969ff76fee89cd5a|761099dd>

- [fc] adding protobuf support (#2241) linkedin/venice

#2260 Revert PR 2258 to not increase the frequency of per route metrics Pull request opened by m-nagarajan ## Problem Statement PR 2258 introduced new metrics whose frequency could result in performance issues as its in hot path ## Solution 1. Revert the PR 2. Add Avg() to existing

pending_request_count

metric ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by m-nagarajan

<https://github.com/linkedin/venice/commit/b82a42c29b8c7dc6e26632fe6fff53e4f481341e|b82a42c2>

- Revert PR 2258 to not increase the frequency of per route metrics (#2260) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by sofiaz11

<https://github.com/linkedin/venice/commit/a85e482cca30775c6195dd39e9502e95e20a3234|a85e482c>

- [controller] Add config to skip hybrid store RT topic compaction policy update (#2257) linkedin/venice

#2261 [controller] Respect source region param for STREAM push requests Pull request opened by sushantmane ## [controller] Respect source region param for STREAM push requests Add support for overriding PubSub bootstrap servers in STREAM push jobs based on the source region specified in the request. This enables stream pushes to target the correct regional PubSub clusters instead of the request-handling controllers's local PubSub cluster. ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by KaiSernLim

<https://github.com/linkedin/venice/commit/c103ea3a261db0f98194df010834ba2a9846c8e8|c103ea3a>

- [server] Global RT DIV: Fix Duplicate Message Due to Offset Record Sync (#2253) linkedin/venice

#2262 [dvc][cdc] MVP for seekable DVRT CDC Pull request opened by kvargha ## Problem Statement 1. DVRT CDC does not support seek. 2. CompletableFuture from start in DVRT CDC will hang forever if no records are ingested. ## Solution 1. Added seek support for

seekToCheckpoint

and

seekToTimestamp

in DVRT CDC. It is only externally available in the version specific CDC client. Additionally, we don't allow seeking to an already subscibed partition, because DaVinci doesn't support it. If they want to seek, they will need to unsubscribe then seek. 2. Added a timeout during start to not block applications that don't have any records in the version topic. ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

#2263 [controller][avro] Add doc field for v25 to make v24 and v25 NOT identical Pull request opened by minhmo1620 ## Problem Statement AdminOperation v24 and v25 schemas are currently identical, which prevents us from registering this schema to system store. ## Solution Add doc field for AddVersion (in v25) (the field got changed between these two versions) to make two schemas unique. Add test to serialize the schema with v24 (old schema) and deserialize the message with v25 (with doc field) to make sure the v25 message in admin topic will be deserialized with new schema. ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by haoxu07

<https://github.com/linkedin/venice/commit/3e2cca5619d9be19be84d6a0003b49e8ab1568b5|3e2cca56>

- [controller] Adding logic to prevent re-creating store after store deletion within a configurable time window (#2243) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by kvargha

<https://github.com/linkedin/venice/commit/5c6fbaa50008580bab8dc009be68e5ad5ef774d9|5c6fbaa5>

- [dvc][cdc] MVP for seekable DVRT CDC (#2262) linkedin/venice

#2264 [controller] extend rt versioning to system stores Pull request opened by arjun4084346 ## Problem Statement This pull request introduces several changes related to real-time topic handling and system store utilities. The main focus is on adjusting how real-time topic names are composed and how applicability checks for real-time versioning are performed, as well as adding debug output to facilitate troubleshooting. Real-time Topic Name Composition and Applicability: • Modified

Utils.composeRealTimeTopic(String storeName)

to use the overloaded method with a version number (defaulting to 1), and added a debug print statement to output the composed topic name. • Changed

Utils.isRTVersioningApplicable(String storeName)

to always return

true

, bypassing previous checks for system and participant stores, likely for testing purposes. ## Solution ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

#2265 [cdc] Sunset the old implementation of BootstrappingVeniceChangelogConsumer and mark the DVRT implementation for GA Pull request opened by kvargha ## Problem Statement 1. The old implementation of

BootstrappingVeniceChangelogConsumer

is no longer in use. 2. There isn't a way to use the DVRT implementation of

VeniceChangelogConsumer

. 3. There are APIs in

VeniceChangelogConsumer

that won't be supported in the DVRT implementation. ## Solution 1. Deleted the old implementation of

BootstrappingVeniceChangelogConsumer

, and made the DVRT implementation the default returned from the factory. Additionally, removed the Experimental flag from

BootstrappingVeniceChangelogConsumer

since it is running in production right now and is ready for wide adoption. Lastly, updated the JavaDoc for

BootstrappingVeniceChangelogConsumer

to reflect this. 2. Renamed

isExperimentalClientEnabled

to

isNewStatelessClientEnabled

, so that the

VeniceChangelogConsumerClientFactory

can return the DVRT implementation for

VeniceChangelogConsumer

. 3. Added the Deprecated annotation for

seekToEndOfPush

,

pause

, and

resume

in

VeniceChangelogConsumer

. ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by minhmo1620

<https://github.com/linkedin/venice/commit/1700a36037b94f74ecbd5d6cd1a6aaaf335ab3f7|1700a360>

- [controller][avro] Add doc field for v25 to make v24 and v25 NOT identical (#2263) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by jingy-li

<https://github.com/linkedin/venice/commit/3ff8733d98f3300e46c6b1a73636137e5c3f72e9|3ff8733d>

- [server][dvc] skip counting "no peer found" errors to accurately reflect transfer status (#2259) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by arjun4084346

<https://github.com/linkedin/venice/commit/b8c6cb7e5c0205e7e617d3e86b834c0fc8ecc78c|b8c6cb7e>

- [controller] [admin-tool] replace skipAdminMessageByOffset with skipAdminMessageByPosition (#2229) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by sushantmane

<https://github.com/linkedin/venice/commit/5e25f4c57be3d41d4946b513591a396e23730433|5e25f4c5>

- [controller] Log client service principal for controller requests (#2254) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by sushantmane

<https://github.com/linkedin/venice/commit/5420915573d102a63ad4b9677b8cc3e49702351c|54209155>

- [controller] Respect source region param for STREAM push requests (#2261) linkedin/venice

#2267 [draft] DO NOT REVIEW TRYING TO RESOLVE MERGE CONFLICTS Admingrpc2 Pull request opened by arjun4084346 ## Problem Statement ## Solution ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

#2268 [router] use latency based least loaded routing Pull request opened by m-nagarajan ## Problem Statement Least loaded routing based on pending request count is not ignoring hot hosts when there is low QPS ## Solution 1. continue using pending request count to mark a host healthy/unhealthy 2. From the list of healthy hosts based on pending request count, use latency based least loaded routing to select the host to send traffic to. Added new config

router.latency.based.routing.enabled

=> Default is false ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). Enabled this config by default for GH actions run: https://github.com/m-nagarajan/venice/actions/runs/19128006488 https://github.com/m-nagarajan/venice/actions/runs/19128006470 ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

#2269 [WIP][server] Add a heartbeat lag based replica auto resubscribe feature Pull request opened by sixpluszero ## [server] Add a heartbeat lag based replica auto resubscribe feature This is an optional feature that aims to improve consumption resiliency. When we observe current version resource heartbeat delay grows beyond certain threshold, it could potentially be that underlying consumer is not functioning well. While we are also working on figuring out the root cause of that, this feature aims to mitigate it in the runtime by asking the SIT to try to perform a resubscribe action that brings the replica to another consumer in the same consumer pool. ## Solution ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? Work in progress • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

#2270 [doc] a brief doc to Venice DIV Pull request opened by lluwm ## Problem Statement Add a doc to Venice DIV mechanism. ## Solution Add a doc to Venice DIV mechanism. ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by kvargha

<https://github.com/linkedin/venice/commit/ce374687f1b363235cd6e1855c68ba99d09329eb|ce374687>

- [cdc] Sunset the old implementation of BootstrappingVeniceChangelogConsumer and mark the DVRT implementation for GA (#2265) linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by lluwm

<https://github.com/linkedin/venice/commit/4ce0ddd2c1d99b6b5b5f09ce5eef8d504bf49f07|4ce0ddd2>

- [doc] a brief doc to Venice DIV (#2270) linkedin/venice

#2271 [protocol] Introduce a new trackingIncrementalPushStatus in PartitionState Pull request opened by sixpluszero ## [protocol] Introduce a new trackingIncrementalPushStatus in PartitionState Add this new map field to track replica's recent incremental push status. This tracking is intended for blob transfer, when receiver receives a RocksDB blob, it should scan this field and report the status to the tracking system (push status system store), so that controller who is polling the ongoing incremental push status can receive the correct signal. ## Solution ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? N/A Protocol only • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice

1 new commit pushed to

<https://github.com/linkedin/venice/tree/main|main>

by arjun4084346

<https://github.com/linkedin/venice/commit/061ed4e34dd2bcc5f3ea6ca366056d4b7c1a9620|061ed4e3>

- [controller] [protocol] use pubsub position in admin metadata update path used by START_FABRIC_BUILDOUT admin tool command (#2199) linkedin/venice

#2272 [dvc] Sunset automatic on disk subscription for da vinci Pull request opened by kvargha ## Problem Statement There is a feature/bug where DaVinci automatically subscribes to stores that are on disk, even if the user doesn't create a client to subscribe to it. The config

DA_VINCI_SUBSCRIBE_ON_DISK_PARTITIONS_AUTOMATICALLY

was added to safely address #650. ## Solution We have set

DA_VINCI_SUBSCRIBE_ON_DISK_PARTITIONS_AUTOMATICALLY

to false for every da vinci user for 3 weeks and not one user has complained. So this is safe to sunset completely. ### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,

synchronized

,

RWLock

) are used where needed. • No blocking calls inside critical sections that could lead to deadlocks or performance degradation. • Verified thread-safe collections are used (e.g.,

ConcurrentHashMap

,

CopyOnWriteArrayList

). • Validated proper exception handling in multi-threaded code to avoid silent thread termination. ## How was this PR tested? • New unit tests added. • New integration tests added. • Modified or extended existing tests. • Verified backward compatibility (if applicable). ## Does this PR introduce any user-facing or breaking changes? • No. You can skip the rest of this section. • Yes. Clearly explain the behavior change and its impact. linkedin/venice