Dheemanth Gowda
07/07/2025, 2:52 PMCREATE TABLE daily_err_agg_stats (
...
) WITH (
'connector' = 'jdbc',
'url' = 'jdbc:postgresql://...',
'table-name' = 'daily_err_agg_batch_v3_testing',
'sink.buffer-flush.max-rows' = '5000',
'sink.buffer-flush.interval' = '10s',
'sink.max-retries' = '3'
-- NO PRIMARY KEY (should be INSERT mode, not UPSERT)
);
Expected:
• 603,234 ÷ 30 = ~20,107 rows per subtask
• 20,107 ÷ 5,000 = 4 batches per subtask + remainder
• Total: ~120 batch INSERT operations
• Tried different buffer sizes (1000, 5000) and intervals (2s, 10s)
Actual:
Appears to be doing individual row inserts instead of batches
Environment:
• Flink 1.17
• PostgreSQL sink
• Batch execution mode (RuntimeExecutionMode.BATCH)
• No primary key constraint (should avoid UPSERT mode)
Questions:
1. Does buffer-flush-max-rows work per subtask or globally?
2. Are there any other settings needed to force batch mode?
3. How can I verify if batching is actually happening?
Any insights appreciated! 🙏
STATS from PG admin:
INSERT INTO daily_err_agg_batch_v3_testing(projectId, orgId, consumptionId, senderNumber, gupshupTemplateId, templateName, templateLanguage, accountType, failureCode, operatorErrorCode, failureReason, channelType, destinationCountry, sourceSystem, conversationCategoryType, pricingCategory, windowStart, windowEnd, failedCount, isDayLevel, modifiedTimestamp, derivedPricingCategory, pricingType, pricingModel, isbillable) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25)