... why would i be getting data delivered to the f...
# troubleshooting
g
... why would i be getting data delivered to the flink writer (output to Paimon) but it gets stuck there and does not go further.
Copy code
INSERT INTO c_paimon.outbound.children
SELECT 
    JSON_VALUE(data, '$._id') as _id,
    JSON_VALUE(data, '$.name') as name,
    JSON_VALUE(data, '$.surname') as surname,
    JSON_VALUE(data, '$.gender') as gender,
    JSON_VALUE(data, '$.dob') as dob,
    JSON_VALUE(data, '$.nationalid') as nationalid,
    JSON_VALUE(data, '$.family_id') as family_id,
    JSON_VALUE(data, '$.father_nationalid') as father_nationalid,
    JSON_VALUE(data, '$.mother_nationalid') as mother_nationalid,
    ROW(
        JSON_VALUE(data, '$.address.street_1'),
        JSON_VALUE(data, '$.address.street_2'),
        JSON_VALUE(data, '$.address.neighbourhood'),
        JSON_VALUE(data, '$.address.town'),
        JSON_VALUE(data, '$.address.county'),
        JSON_VALUE(data, '$.address.province'),
        JSON_VALUE(data, '$.address.country'),
        JSON_VALUE(data, '$.address.country_code'),
        JSON_VALUE(data, '$.address.postal_code'),
        JSON_VALUE(data, '$.address.parcel_id')
    ) as address,
    created_at
FROM postgres_catalog.inbound.children;
a
Are you referring to the Records Sent metric in the ui? I don't know about the sinks used for Paimon. But I've worked with sinks that do not report a Records Sent even though they did indeed write data. Have you checked whether this Record Sent being 0 could be a false negative (i.e stuff is really being written but UI not reflecting this)? Another possible scenario could be that the Global Committer is waiting for checkpoint to happen before flushing the data to Paimon
g
there is nothing being written to the file. checkpointing is set to 30 seconds. I have another job thats a CTAS thats writing happily.