GitHub
05/05/2025, 11:02 PMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/05/2025, 11:49 PM<https://github.com/linkedin/venice/tree/main|main>
by gaojieliu
<https://github.com/linkedin/venice/commit/8d5e5739b42bbab39514e77790c1e8f3667a0e51|8d5e5739>
- [server][da-vinci] Bumped up rocksdb-9.10.0 (#1764)
linkedin/veniceGitHub
05/06/2025, 1:04 AMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/06/2025, 5:00 AMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/06/2025, 9:25 PM## AI Generated Summary of Changes
This pull request introduces significant changes to the PubSub integration within the Da Vinci client, focusing on simplifying configuration management and improving the internal handling of PubSub components. The changes include replacing thewith a more comprehensive PubSub component initialization process, refactoring constructors across multiple classes, and updating thePubSubPositionDeserializer
to streamline PubSub consumer creation.VeniceChangelogConsumerClientFactory
### PubSub Configuration and Initialization Enhancements:
• Introduced a new methodininitializePubSubInternals
to centralize the initialization of PubSub-related components (e.g.,ChangelogClientConfig
,PubSubPositionDeserializer
) based on consumer properties. This replaces manual configuration and ensures consistency. [1] [2]PubSubConsumerAdapterFactory
• Removed themethod and its associated getter fromsetPubSubPositionDeserializer
, as the deserializer is now initialized internally. [1] [2]ChangelogClientConfig
### Constructor Refactoring:
• Updated constructors for,InternalLocalBootstrappingVeniceChangelogConsumer
, andLocalBootstrappingVeniceChangelogConsumer
to remove theVeniceAfterImageConsumerImpl
parameter. These classes now rely on the internally managed PubSub components inPubSubPositionDeserializer
. [1] [2] [3]ChangelogClientConfig
### Streamlined PubSub Consumer Creation:
• Refactoredto replace theVeniceChangelogConsumerClientFactory
method withgetConsumer
, which uses the new PubSub initialization logic ingetPubSubConsumer
. This ensures that all PubSub components are derived from a single source of truth. [1] [2]ChangelogClientConfig
### Code Cleanup:
• Removed unused imports and redundant fields related to the oldapproach across multiple files. [1] [2]PubSubPositionDeserializer
These changes enhance maintainability and reduce the risk of configuration inconsistencies by consolidating PubSub-related logic into a single initialization method.### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,
synchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/06/2025, 10:14 PM<https://github.com/linkedin/venice/tree/main|main>
by gaojieliu
<https://github.com/linkedin/venice/commit/40e70431d1de277d23bea91b5f707e1900ced2c4|40e70431>
- Revert "[server][da-vinci] Bumped up rocksdb-9.10.0 (#1764)" (#1768)
linkedin/veniceGitHub
05/07/2025, 1:04 AM<https://github.com/linkedin/venice/tree/main|main>
by misyel
<https://github.com/linkedin/venice/commit/8c48f8b7e9ea79749dbe98fa88943e5d315f0cec|8c48f8b7>
- [controller] Fix store migration after a target region push with deferred swap (#1760)
linkedin/veniceGitHub
05/07/2025, 5:53 PMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/07/2025, 8:08 PM<https://github.com/linkedin/venice/tree/main|main>
by misyel
<https://github.com/linkedin/venice/commit/3ad768c59f0ab8999910eafd851b75a70ea71c41|3ad768c5>
- [da-vinci] Add delayed ingestion in dvc for target region push with deferred swap (#1510)
linkedin/veniceGitHub
05/07/2025, 8:08 PMDEFERRED_VERSION_SWAP_SERVICE_WITH_DVC_CHECK_ENABLED
is set to false in the parent controller
## How was this PR tested?
Added integration test
## Does this PR introduce any user-facing changes?
• No. You can skip the rest of this section.
• Yes. Make sure to explain your proposed changes and call out the behavior change.
linkedin/veniceGitHub
05/07/2025, 9:31 PMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/07/2025, 11:17 PM<https://github.com/linkedin/venice/tree/main|main>
by m-nagarajan
<https://github.com/linkedin/venice/commit/8ab13739398db470f73c5c40e252485b2a91bfb9|8ab13739>
- [router] Add MetricEntityStateGeneric for flexible dimension handling for OTel (#1667)
linkedin/veniceGitHub
05/07/2025, 11:17 PMMetricEntityState
are for specific cases with specific number of dimensions which should be enums that are perf and GC optimized by reusing Attributes
rather than recreate it everytime. But for cases like controllers where the metric emission is infrequent and is not processing tens of thousands or more of QPS, it would benefit to have a generic non cached version of a subclass rather than creating multiple new subclasses for all combination of dimensions.
## Solution
1. Introduce MetricEntityStateGeneric
which takes in baseDimensionsMap
during init and all remaining dimensions as a Map
during record()
call where it validates the data for all required dimensions. This can be reused for most of the metrics in controllers. This will also help in code maintainability by reducing some of the subclasses needed.
2. Unlike other subclasses which are typesafe and most of the checks are compile time, this class will have some of the checks during runtime, thus can fail during runtime. The failures are logged with a redundant log filter and are capture in an internal metric: venice.internal.metric_record_failure
.
3. This can't be used for 0 dynamic dimensions case and it will throw an error during initialization. MetricEntityStateBase
should be used instead.
### Code changes
• Added new code behind a config. If so list the config names and their default values in the PR description.
• Introduced new log lines.
• Confirmed if logs need to be rate limited to avoid excessive logging.
### Concurrency-Specific Checks
Both reviewer and PR author to verify
• Code has no race conditions or thread safety issues.
• Proper synchronization mechanisms (e.g., synchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/08/2025, 12:00 AM<https://github.com/linkedin/venice/tree/main|main>
by xunyin8
<https://github.com/linkedin/venice/commit/169d1031530b3fc0892481b4345f3081e55af12d|169d1031>
- [server] Improve ReadQuotaEnforcementHandler init() behavior and visibility (#1757)
linkedin/veniceGitHub
05/08/2025, 12:49 AMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/08/2025, 5:37 AM1s latency, which is unexpected.
This PR disables this feature to avoid the above behavior. Whether retry upon these error response or not will be decided by the user instead of the transporting layer.### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,
synchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/08/2025, 7:27 AMThis pull request refactors theclass to improve memory efficiency and error handling by switching to a lazy initialization approach forPubSubPositionTypeRegistry
instances. It also updates the test cases to align with the new lazy-loading behavior.PubSubPositionFactory
### Refactoring for Lazy Initialization:
• Replaced the eager initialization ofwith a lazy-loading approach usingtypeIdToFactoryMap
andVeniceConcurrentHashMap
. This ensures thatcomputeIfAbsent
instances are only created when accessed. (PubSubPositionFactory
, [1] [2]internal/venice-common/src/main/java/com/linkedin/venice/pubsub/PubSubPositionTypeRegistry.java
• Removed themethod, as factory instances are now created on demand instead of being preloaded. (instantiateFactories
, internal/venice-common/src/main/java/com/linkedin/venice/pubsub/PubSubPositionTypeRegistry.javaL265-L286)internal/venice-common/src/main/java/com/linkedin/venice/pubsub/PubSubPositionTypeRegistry.java
### Constructor and Dependency Updates:
• Updated the constructor ofto stop preloading factory instances intoPubSubPositionTypeRegistry
, aligning with the lazy-loading design. (typeIdToFactoryMap
, internal/venice-common/src/main/java/com/linkedin/venice/pubsub/PubSubPositionTypeRegistry.javaL123)internal/venice-common/src/main/java/com/linkedin/venice/pubsub/PubSubPositionTypeRegistry.java
### Test Case Adjustments:
• Modified the test case### Code changes • Added new code behind a config. If so list the config names and their default values in the PR description. • Introduced new log lines. • Confirmed if logs need to be rate limited to avoid excessive logging. ### Concurrency-Specific Checks Both reviewer and PR author to verify • Code has no race conditions or thread safety issues. • Proper synchronization mechanisms (e.g.,to validate the lazy initialization behavior by triggering factory creation during runtime instead of during registry construction. (testRegistryRejectsUnknownClassName
, internal/venice-common/src/test/java/com/linkedin/venice/pubsub/PubSubPositionTypeRegistryTest.javaL88-R92)internal/venice-common/src/test/java/com/linkedin/venice/pubsub/PubSubPositionTypeRegistryTest.java
synchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/08/2025, 7:47 AMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
User will no longer have to specify offset lag or time lag as it has been confusing for them and it is not accurate without proper write rate consideration.
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/08/2025, 3:53 PM<https://github.com/linkedin/venice/tree/main|main>
by m-nagarajan
<https://github.com/linkedin/venice/commit/0d7d335e93164b0ad6a98d769ddcf0610b089ad1|0d7d335e>
- [tc][fc][dvc] Add client availability/latency metrics in Otel (#1689)
linkedin/veniceGitHub
05/08/2025, 5:26 PMGitHub
05/08/2025, 6:21 PM<https://github.com/linkedin/venice/tree/main|main>
by gaojieliu
<https://github.com/linkedin/venice/commit/fd3706552eb8467cf42c4ba7770bbde5b158c842|fd370655>
- [fast-client][router] Disable auto retry in httpclient5 to avoid delay upon 429/503 response (#1772)
linkedin/veniceGitHub
05/08/2025, 6:31 PM<https://github.com/linkedin/venice/tree/main|main>
by sushantmane
<https://github.com/linkedin/venice/commit/d272ae59915402765c58e6e2a6ce5e98be7bf1d7|d272ae59>
- [server][controller][dvc][cc] Lazily instantiate PubSubPositionFactory on demand (#1773)
linkedin/veniceGitHub
05/08/2025, 7:14 PMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/08/2025, 7:45 PMKafka
from KafkaDataIntegrityValidator
.
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
linkedin/veniceGitHub
05/08/2025, 8:25 PM<https://github.com/linkedin/venice/tree/main|main>
by KaiSernLim
<https://github.com/linkedin/venice/commit/7b7df46726d0fb347d91a239845409d4bc70c6d2|7b7df467>
- [server] Rename DIV Class (#1776)
linkedin/veniceGitHub
05/08/2025, 8:46 PM<https://github.com/linkedin/venice/tree/main|main>
by kvargha
<https://github.com/linkedin/venice/commit/60b6466dba9f84997b938086f876245a49c1e13e|60b6466d>
- [dvc][server][doc] Add exponential backoff for zk update retries + doc cleanup (#1734)
linkedin/veniceGitHub
05/09/2025, 12:26 AMcom.github.luben:zstd-jni:1.5.6-8
, which includes a fix to ensure all native code execution is protected by a shared lock for improved thread safety.
## Solution
1. Dependency Upgrade
• Upgraded Zstd JNI to version 1.5.6-8
1. Exception Handling in Tests
• Updated exception assertions in TestZstdLibrary.java:
• Previous: e.getMessage().equals("Src size is incorrect")
• Current: e.getMessage().equals("nb of samples too low")
• All affected unit tests have been updated accordingly and are now passing.
1. Test Fix for Sample Count Validation
• A ZstdException was triggered in one test due to providing only 10 samples, violating the minimum requirement (sampleSizes.length <= 10)). This check enforces that at least 11 samples are required to proceed. So, when your test only had 10 samples, it triggered the exception: "nb of samples too low"
### Code changes
• Added new code behind a config. If so list the config names and their default values in the PR description.
• Introduced new log lines.
• Confirmed if logs need to be rate limited to avoid excessive logging.
### Concurrency-Specific Checks
Both reviewer and PR author to verify
• Code has no race conditions or thread safety issues.
• Proper synchronization mechanisms (e.g., synchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/09/2025, 12:28 AMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/09/2025, 12:42 AMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/veniceGitHub
05/09/2025, 3:13 AMsynchronized
, RWLock
) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap
, CopyOnWriteArrayList
).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.
## How was this PR tested?
• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).
## Does this PR introduce any user-facing or breaking changes?
• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.
linkedin/venice