Upon registering (pushing) metadata for offline se...
# pinot-dev
a
Upon registering (pushing) metadata for offline segment the latter is unconditionally assigned to servers defined by
tenants.server
, without considering
tierConfigs
. Only on rebalance (triggered by SegmentRelocator job) it will be "moved" to a proper tenant. This behavior is not really convenient when table is refilled regularly as segments will initially end up in default "warm" tier, while preferably they shall be properly distributed. My question is: if this has been done intentionally? Or is it just waiting for someone to come and fix it (with an additional flag to preserve current behavior)?
m
What does your tier config look like? Are you regularly pushing old data that directly needs to go to tier?
a
Right, table is periodically reconciled, and I would prefer that data "older than X days" went directly to the required tenant. I'm working on a patch for our local fork, but I wonder if this change would be desired in the upstream as well?
m
So seems like you are pushing aged data into Pinot (somewhat atypical). This does seems like a good optimization in such case. You can certainly file an issue and also PR and we can discuss there.
🍷 1
n
Hi @Alex P. did you file a PR for this? This is an interesting contribution which will benefit us so looking forward to it
x
And when pushing segments to target tier directly, you’d want to ensure that the
segment.tier
field in SegmentZKMetadata is set properly. e.g. in this rebalanceTable method (which is used by SegmentRelocator task as well), it sets target tier before relocating segments across tiers. feel free to tag me to review the PR :)
a
Hi All, I've made changes in my local fork, but haven't yet submitted a PR, I'm planning to do this within these two weeks.
Indeed, due to the recent changes regarding the multi-dir support, it's not enough to redefine
assignSegment
method for
SegmentAssignment
.
My fork is ~8mo older than upstream where this problem doesn't exist, so I need to spend a bit more time than just porting my change.
Once I submit my PR I'll tag you @Xiaobing, as I was going anyway 😄 Thanks!
👍 1
Hey, this is the PR: https://github.com/apache/pinot/pull/10746. I never had any sizable experience with Java, so please feel free to leave your comments/remarks. Thanks! cc: @Xiaobing for review 🙂
👀 1
Hi All, could anyone retrigger tests for the PR above? I'm not sure if the observed failures are relevant to the changes. Also I see some transient failures in other PRs. I wonder if such failures are "normal", or they confidently indicate problems with the code?