Heyo! Is it expected that the pinot compatibility ...
# troubleshooting
c
Heyo! Is it expected that the pinot compatibility test suite doesn't support testing tables with ingestion aggregation (
aggregateMetrics
set to
true
)? Or am I misunderstanding? In my
pre-controller-upgrade.yaml
file I create a stream and table for a realtime table. Then I execute the following
StreamOp
:
Copy code
- type: streamOp
    description: publish rows to my-table
    op: PRODUCE
    streamConfigFileName: streams/my-table.json
    numRows: 16
    inputDataFileName: data/my-table/_generated_segment_0.csv
    recordReaderConfigFileName: data/my-table/recordReaderConfig.json
    tableConfigFileName: tables/my-table.json
data/my-table/_generated_segment_0.csv
contains 16 rows of data that will be published to my kafka stream. When pinot consumes these 16 rows, only 14 will end up in my table because of ingestion aggregation. But the test suite fails because the number of rows that are written to the table is less than the
numRows
I set in my StreamOp definition.
Copy code
private void waitForDocsLoaded(String tableName, long targetDocs, long timeoutMs) {
    LOGGER.info("Wait Doc to load ...");
    AtomicLong loadedDocs = new AtomicLong(-1);
    TestUtils.waitForCondition(
        () -> {
          long existingTotalDocs = fetchExistingTotalDocs(tableName);
          loadedDocs.set(existingTotalDocs);
          return existingTotalDocs == targetDocs;
        }, 100L, timeoutMs,
        "Failed to load " + targetDocs + " documents. Found " + loadedDocs.get() + " instead", true,
        Duration.ofSeconds(1));
  }
Copy code
Exception in thread "main" java.lang.AssertionError: Failed to meet condition in 60000ms after 0 errors, error message: Failed to load 16 documents. Found -1 instead
        at org.testng.Assert.fail(Assert.java:99)
        at org.apache.pinot.util.TestUtils.waitForCondition(TestUtils.java:152)
        at org.apache.pinot.compat.StreamOp.waitForDocsLoaded(StreamOp.java:309)
        at org.apache.pinot.compat.StreamOp.produceData(StreamOp.java:259)
        at org.apache.pinot.compat.StreamOp.runOp(StreamOp.java:156)
        at org.apache.pinot.compat.BaseOp.run(BaseOp.java:81)
        at org.apache.pinot.compat.CompatibilityOpsRunner.runOps(CompatibilityOpsRunner.java:65)
        at org.apache.pinot.compat.CompatibilityOpsRunner.main(CompatibilityOpsRunner.java:86)
Failed before controller upgrade
Because the
numRows
argument is used to both read rows from the input csv and to confirm all rows were loaded, it doesn't seem like it is possible to test a table with ingestion aggregation?
x
are you first create the table then start data ingestion? one possibility is that if you ingest data earlier then pinot may miss data
also have you tried to query pinot for which row got missed and also try to debug from your IDE for the missing rows
another place to check is the kafka consumer
see if the kafka topic has expected 16 rows
c
Yah, I'm creating the table first. Here is the full test phase yaml file:
Copy code
description: Operations to be run before controller upgrade
operations:
  - type: streamOp
    description: create Kafka topic my-topic
    op: CREATE
    streamConfigFileName: streams/my-table.json
  - type: tableOp
    description: Create realtime table my-table
    op: CREATE
    schemaFileName: schemas/my-table.json
    tableConfigFileName: tables/my-table.json
    recordReaderConfigFileName: data/my-table/recordReaderConfig.json
  - type: streamOp
    description: publish rows to my-topic
    op: PRODUCE
    streamConfigFileName: streams/my-table.json
    numRows: 16
    inputDataFileName: data/my-table/_generated_segment_0.csv
    recordReaderConfigFileName: data/my-table/recordReaderConfig.json
    tableConfigFileName: tables/my-table.json
  - type: queryOp
    description: Run query on my-table using SQL
    queryFileName: queries/my-table/_generated_test_queries.queries
    expectedResultsFileName: query-results/my-table/_generated_test_query_results.results
I see in the output from the
compCheck.sh
script that the table create op has succeeded before moving on to the stream op.
It is expected that the 16 input rows read from the csv and published to kafka would turn in to 14 rows in the table because of ingestion aggregation I think.
Manually setting the number of expected rows in the table to 14 (targetDocs is equal to the numRows param defined in my test yaml file) in the waitForDocsLoaded method of the StreamOp class seems to solve the issue. My StreamOp succeeds and I can see all 14 records in the table.
Copy code
private void waitForDocsLoaded(String tableName, long targetDocs, long timeoutMs) {
    LOGGER.info("Wait Doc to load ...");
    AtomicLong loadedDocs = new AtomicLong(-1);
    long finalTargetDocs = targetDocs - 2;
    TestUtils.waitForCondition(
        () -> {
          long existingTotalDocs = fetchExistingTotalDocs(tableName);
          LOGGER.info("Existing total docs found: " + existingTotalDocs);
          loadedDocs.set(existingTotalDocs);
          return existingTotalDocs == finalTargetDocs;
        }, 100L, timeoutMs,
        "Failed to load " + targetDocs + " documents. Found " + lastExistingTotalDocs[0] + " instead", true,   
        Duration.ofSeconds(1));
  }
}
2 records from the input csv aren't written to the table, but rather aggregated on ingestion. This behavior is expected, but I don't think the test suite is designed to handle this use case?
Wonder if maybe the args for the stream operation should include an inputNumRows and outputNumRows to be able to handle ingestion aggregation instead of just numRows. Thoughts?
x
Oic, then it makes sense. Either use output rows or just use a query for expected results