charlie
04/10/2025, 10:57 PMaggregateMetrics set to true )? Or am I misunderstanding?
In my pre-controller-upgrade.yaml file I create a stream and table for a realtime table. Then I execute the following StreamOp :
- type: streamOp
description: publish rows to my-table
op: PRODUCE
streamConfigFileName: streams/my-table.json
numRows: 16
inputDataFileName: data/my-table/_generated_segment_0.csv
recordReaderConfigFileName: data/my-table/recordReaderConfig.json
tableConfigFileName: tables/my-table.json
data/my-table/_generated_segment_0.csv contains 16 rows of data that will be published to my kafka stream. When pinot consumes these 16 rows, only 14 will end up in my table because of ingestion aggregation.
But the test suite fails because the number of rows that are written to the table is less than the numRows I set in my StreamOp definition.
private void waitForDocsLoaded(String tableName, long targetDocs, long timeoutMs) {
LOGGER.info("Wait Doc to load ...");
AtomicLong loadedDocs = new AtomicLong(-1);
TestUtils.waitForCondition(
() -> {
long existingTotalDocs = fetchExistingTotalDocs(tableName);
loadedDocs.set(existingTotalDocs);
return existingTotalDocs == targetDocs;
}, 100L, timeoutMs,
"Failed to load " + targetDocs + " documents. Found " + loadedDocs.get() + " instead", true,
Duration.ofSeconds(1));
}
Exception in thread "main" java.lang.AssertionError: Failed to meet condition in 60000ms after 0 errors, error message: Failed to load 16 documents. Found -1 instead
at org.testng.Assert.fail(Assert.java:99)
at org.apache.pinot.util.TestUtils.waitForCondition(TestUtils.java:152)
at org.apache.pinot.compat.StreamOp.waitForDocsLoaded(StreamOp.java:309)
at org.apache.pinot.compat.StreamOp.produceData(StreamOp.java:259)
at org.apache.pinot.compat.StreamOp.runOp(StreamOp.java:156)
at org.apache.pinot.compat.BaseOp.run(BaseOp.java:81)
at org.apache.pinot.compat.CompatibilityOpsRunner.runOps(CompatibilityOpsRunner.java:65)
at org.apache.pinot.compat.CompatibilityOpsRunner.main(CompatibilityOpsRunner.java:86)
Failed before controller upgrade
Because the numRows argument is used to both read rows from the input csv and to confirm all rows were loaded, it doesn't seem like it is possible to test a table with ingestion aggregation?Xiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
charlie
04/11/2025, 9:14 PMdescription: Operations to be run before controller upgrade
operations:
- type: streamOp
description: create Kafka topic my-topic
op: CREATE
streamConfigFileName: streams/my-table.json
- type: tableOp
description: Create realtime table my-table
op: CREATE
schemaFileName: schemas/my-table.json
tableConfigFileName: tables/my-table.json
recordReaderConfigFileName: data/my-table/recordReaderConfig.json
- type: streamOp
description: publish rows to my-topic
op: PRODUCE
streamConfigFileName: streams/my-table.json
numRows: 16
inputDataFileName: data/my-table/_generated_segment_0.csv
recordReaderConfigFileName: data/my-table/recordReaderConfig.json
tableConfigFileName: tables/my-table.json
- type: queryOp
description: Run query on my-table using SQL
queryFileName: queries/my-table/_generated_test_queries.queries
expectedResultsFileName: query-results/my-table/_generated_test_query_results.resultscharlie
04/11/2025, 9:15 PMcompCheck.sh script that the table create op has succeeded before moving on to the stream op.charlie
04/11/2025, 9:17 PMcharlie
04/11/2025, 9:19 PMprivate void waitForDocsLoaded(String tableName, long targetDocs, long timeoutMs) {
LOGGER.info("Wait Doc to load ...");
AtomicLong loadedDocs = new AtomicLong(-1);
long finalTargetDocs = targetDocs - 2;
TestUtils.waitForCondition(
() -> {
long existingTotalDocs = fetchExistingTotalDocs(tableName);
LOGGER.info("Existing total docs found: " + existingTotalDocs);
loadedDocs.set(existingTotalDocs);
return existingTotalDocs == finalTargetDocs;
}, 100L, timeoutMs,
"Failed to load " + targetDocs + " documents. Found " + lastExistingTotalDocs[0] + " instead", true,
Duration.ofSeconds(1));
}
}charlie
04/11/2025, 9:23 PMcharlie
04/11/2025, 9:25 PMXiang Fu