Idlan Amran
05/13/2025, 9:24 AMXiang Fu
Idlan Amran
05/13/2025, 10:18 AMepoch_days then add a transform function for the column ToEpochDaysBucket(now(),1) . then add the value to to primaryKeyColumns on our schema. at least this is the most doable for our project atmXiang Fu
Xiang Fu
Xiang Fu
Idlan Amran
05/13/2025, 10:21 AMXiang Fu
Idlan Amran
05/13/2025, 10:22 AMXiang Fu
Idlan Amran
05/13/2025, 10:23 AM"taskTypeConfigsMap": {
"UpsertCompactionTask": {
"schedule": "0 8 * ? * *",
"bufferTimePeriod": "1d",
"invalidRecordsThresholdPercent": "30",
"invalidRecordsThresholdCount": "300000",
"tableMaxNumTasks": "100",
"validDocIdsType": "SNAPSHOT"
}
}
im kinda confused on this config, especially on invalidRecordsThresholdPercent , invalidRecordsThresholdPercent and invalidRecordsThresholdCount, do you have time to explain abit if you dont mind ?Xiang Fu
Xiang Fu
Xiang Fu
"UpsertCompactionTask": {
"schedule": "0 * * ? * *",
"bufferTimePeriod": "0d",
"invalidRecordsThresholdPercent": "30",
"tableMaxNumTasks": "100",
"validDocIdsType": "SNAPSHOT"
}Xiang Fu
Idlan Amran
05/13/2025, 10:25 AMIdlan Amran
05/13/2025, 10:26 AMXiang Fu
Xiang Fu
Idlan Amran
05/13/2025, 10:27 AMIdlan Amran
05/13/2025, 10:28 AMIdlan Amran
05/16/2025, 2:37 PMUpsertCompactionTask will clear invalid records per segment or it will combine multiple segments into one too ?Idlan Amran
05/16/2025, 3:12 PMXiang Fu
Idlan Amran
05/19/2025, 12:46 AMIdlan Amran
05/20/2025, 9:43 AMpinot-server:
image: apachepinot/pinot:1.3.0-21-amazoncorretto
platform: linux/amd64
container_name: pinot-server
command: >
StartServer
-zkAddress zookeeper:2181
-serverHost pinot-server
volumes:
- ./pinot_data/data:/data
- ./pinot_data/server-data:/tmp/data
- ./config:/config
- ./input:/input
restart: unless-stopped
ports:
- "8098:8098"
- "8097:8097"
healthcheck:
test: [ "CMD-SHELL", "curl -f http://localhost:8097/health/readiness || exit 1" ]
interval: 30s
timeout: 120s
retries: 15
start_period: 120s
depends_on:
- pinot-controller
environment:
JAVA_OPTS: "-Xms1G -Xmx1G -XX:+UseG1GC"
deploy:
resources:
limits:
cpus: '1.0'
memory: '2G'
abit context on my docker compose configXiang Fu
Xiang Fu
Idlan Amran
05/20/2025, 1:26 PMXiang Fu
Idlan Amran
05/20/2025, 1:31 PMIdlan Amran
05/20/2025, 1:33 PMIdlan Amran
05/20/2025, 1:50 PM