How to achieve something similar to `local directory sequenc apache-pinot #troubleshooting

Channels

test-channel-bot

feat-presto-connector

troubleshooting

discuss-validation

minion-improvements

fix-numerical-predicate

pinot-perf-tuning

feat-partial-upsert

pinot_website_improvement_suggestions

segment-write-api

pinot-k8s-operator

release-certifier

udf-type-matching

transform-functions

latency-during-segment-commit

feat-text-search

pinot-realtime-table-rebalance

inconsistent-segment

feat-rt-seg-complete

flink-pinot-connector

minion-star-tree

pinot-rack-awareness

fix_llc_segment_upload

metrics-plugin-impl

complex-type-support

feat-geo-spatial-index

feat-compound-types

v2_engine_beta_feedback

inconsistent-perf

metadata-push-api

feat-pravega-connector

enable-generic-offsets

multiple_streams

feat-better-schema-evolution

s3-multiple-buckets

presto-pinot-connector

multi-region-setup

segment-cold-storage

odsc-europe-2022-workshop

bangalore-meetups

new-office-space

time-based-segment-pruner

group-by-refactor

pql-sql-regression

custom-aggregators

getting-started

#troubleshooting

Title

y

Yash Agarwal

08/26/2020, 10:47 AM

How to achieve something similar to

local.directory.sequence.id=true

in

SparkSegmentGenerationJobRunner

?

I have added a PR for the same. Let me know if there are any concerns.

https://github.com/apache/incubator-pinot/pull/5927/files

x

Xiang Fu

08/26/2020, 2:40 PM

I will take a look, this if running multiple days?

y

Yash Agarwal

08/26/2020, 2:40 PM

Yes.

x

Xiang Fu

08/26/2020, 2:40 PM

And each day has multiple segments

Got it

y

Yash Agarwal

08/26/2020, 2:40 PM

Yes