Hi Everyone, JDBC Sink Batch Insert Not Triggerin...
# troubleshooting
d
Hi Everyone, JDBC Sink Batch Insert Not Triggering - Need Help(FLINK SQL) Problem: JDBC sink with 603,234 rows, parallelism=30, buffer-flush-max-rows=5000 is not doing batch inserts. All rows seem to be inserted individually.
Copy code
CREATE TABLE daily_err_agg_stats (
    ...
) WITH (
    'connector' = 'jdbc',
    'url' = 'jdbc:postgresql://...',
    'table-name' = 'daily_err_agg_batch_v3_testing',
    'sink.buffer-flush.max-rows' = '5000',
    'sink.buffer-flush.interval' = '10s',
    'sink.max-retries' = '3'
    -- NO PRIMARY KEY (should be INSERT mode, not UPSERT)
);
Expected: • 603,234 ÷ 30 = ~20,107 rows per subtask • 20,107 ÷ 5,000 = 4 batches per subtask + remainder • Total: ~120 batch INSERT operations • Tried different buffer sizes (1000, 5000) and intervals (2s, 10s) Actual: Appears to be doing individual row inserts instead of batches Environment: • Flink 1.17 • PostgreSQL sink • Batch execution mode (RuntimeExecutionMode.BATCH) • No primary key constraint (should avoid UPSERT mode) Questions: 1. Does buffer-flush-max-rows work per subtask or globally? 2. Are there any other settings needed to force batch mode? 3. How can I verify if batching is actually happening? Any insights appreciated! 🙏 STATS from PG admin:
Copy code
INSERT INTO daily_err_agg_batch_v3_testing(projectId, orgId, consumptionId, senderNumber, gupshupTemplateId, templateName, templateLanguage, accountType, failureCode, operatorErrorCode, failureReason, channelType, destinationCountry, sourceSystem, conversationCategoryType, pricingCategory, windowStart, windowEnd, failedCount, isDayLevel, modifiedTimestamp, derivedPricingCategory, pricingType, pricingModel, isbillable) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25)