Hi everyone, We’ve hit a strange bug. We have a Realtime table with upserts and a relatively large primary key, where one of the columns is a string of up to 2048 characters. When an

UpsertCompactionTask

runs, that column gets truncated to 512 characters, which breaks the application since we’re losing data. Am I missing some configuration here, or is this a bug/intentional behavior?

String types are truncated to 512 by default. You need to specify the max length in schema. This is to avoid accidental garbage values to sneak in (it happens more often than one can think).

@Mayank I already updated the schema. I set the the size to 65. The problematic column is bidders. This is the table schema:

{
  "schemaName": "newPublisherAuctionsStats",
  "enableColumnBasedNullHandling": false,
  "dimensionFieldSpecs": [
    {
      "name": "publisherId",
      "dataType": "STRING",
      "notNull": false
    },
    {
      "name": "publisherName",
      "dataType": "STRING",
      "notNull": false
    },
    {
      "name": "bidders",
      "dataType": "STRING",
      "maxLength": 65535,
      "notNull": false
    },
    {
      "name": "code",
      "dataType": "STRING",
      "notNull": false
    },
    {
      "name": "adUnitId",
      "dataType": "STRING",
      "notNull": false
    },
    {
      "name": "auctionId",
      "dataType": "STRING",
      "notNull": false
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "auctionsCount",
      "dataType": "LONG",
      "notNull": false
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "createdAt",
      "dataType": "TIMESTAMP",
      "notNull": false,
      "format": "TIMESTAMP",
      "granularity": "1:HOURS"
    }
  ],
  "primaryKeyColumns": [
    "publisherId",
    "bidders",
    "code",
    "createdAt",
    "auctionId"
  ]
}

So you are saying that the data is intact even after segments are committed. It is only after the compaction task that the column gets truncated? That seems like a bug, yes. cc: @deepthi ch

@Mayank I’m not sure when a segment is committed, so if you want me to confirm the above, I’ll need a bit more guidance on how to do that. In the meantime, here’s what I’ve observed: 1. I put some events into my Kafka brokers. 2. Pinot consumed them. 3. I could see the correct data in my Pinot table. 4. A few minutes later, the compaction task ran (it runs once every hour). 5. The data in my table was broken afterward. (bidder column was truncuated)

Can you share the external view

@MayankHere is my external view:

{
  "id": "newPublisherAuctionsStats_REALTIME",
  "simpleFields": {
    "BATCH_MESSAGE_MODE": "false",
    "BUCKET_SIZE": "0",
    "IDEAL_STATE_MODE": "CUSTOMIZED",
    "INSTANCE_GROUP_TAG": "newPublisherAuctionsStats_REALTIME",
    "MAX_PARTITIONS_PER_INSTANCE": "1",
    "NUM_PARTITIONS": "15",
    "REBALANCE_MODE": "CUSTOMIZED",
    "REPLICAS": "1",
    "STATE_MODEL_DEF_REF": "SegmentOnlineOfflineStateModel",
    "STATE_MODEL_FACTORY_NAME": "DEFAULT",
    "isTablePaused": "false"
  },
  "mapFields": {
    "newPublisherAuctionsStats__0__0__20250901T1338Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__10__20250909T1007Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__11__20250909T1017Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__12__20250910T1017Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__13__20250910T1243Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__14__20250910T1449Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "CONSUMING"
    },
    "newPublisherAuctionsStats__0__1__20250902T1338Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__2__20250903T1338Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__3__20250904T1338Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__4__20250905T1338Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__5__20250906T1338Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__6__20250907T1339Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__7__20250908T1339Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__8__20250909T1002Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "newPublisherAuctionsStats__0__9__20250909T1006Z": {
      "Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    }
  },
  "listFields": {}
}

$segmentName

@Mayank Thanks. I confirm I have broken rows in Online segment:

Ok, can you confirm it is a segment that is not compacted? If so, we know it is not compaction issue

@Mayank The segment was compacted. The data was OK before compaction job ran.

You need to pick an ONLINE segment that wasn’t compacted

@Mayank I don't have such a segments. Let me try something else. I will drop a line in a minute

What we are trying to establish is if it happens when in memory CONSUMING segment goes ONLINE, or when compaction happens

@Mayank Does compaction job affects consuming segemnts?

No

The above is a very small table used for testing. What I observed: 1. I created fresh new table. 2. That table consumed my crafted Kafka topic. 3. Data was OK. 4. I ran the compaction job. 5. 1 minute later the data was broken.

Don't run the compaction job, pause and resume ingestion, that will convert CONSUMING to ONLINE. Then check if data is bad for the ONLINE

Thanks. I will do it

Please don't run any compaction in this flow I mentioned

Understood. I won't

This is just to narrow down where to look. Although, I don't really see a reason why this could happen for either case.

@Mayank I ran the above test. Data is fine after I pause and then resume consumption. I check the segments state before running test queries. Table segments state was changed from CONSUMING to Online. Anything else that I can run?

Ok, then seems like compaction may be culprit, I am tagging @deepthi ch to take a look

@Kiril Kalchev did you check the data before resuming ingestion?

@robert zych Yes. Data was OK before and after resuming ingestion. Data broke right after I manually started the compact job.

If you haven't already done so, I would recommend removing

UpsertCompactionTask

and creating an issue

Unfortunately I can't disable UpsertCompactionTask. Without this task my tables becomes useless (on the limited hardware that we are using) in a week.

I see. Sounds like your keys receive many updates, you have limited storage, and long retention period.

@robert zych And not enough money to scale CPU and storage 🙂

I am curious though @robert zych, both segment commit and the upsert compaction are using the same underlying segment generation code right? Why would one honor the maxLength and the other one not? May be some missing config wiring?

Sorry to hear that. 1st time seeing this specific issue. I haven't tried this before, but the a potential workaround might be to use

substr

to split up the large string during ingestion

@Kiril Kalchev what version are you using?

@Mayank I am using 1.1.0

@robert zych do you plan to investigate this?

Yes, I can take a deeper look into this and will let you know what I find

@Mayank @robert zych Thanks for the support. It was very helpful. I believe I find a workaround. If I change the column type from String to JSON it seems to work well.

Yes please

@Mayank Will do it tomorrow. Thanks you both!

@Kiril Kalchev You may run into issues using JSON instead of STRING. For example, Text index and string functions might not work.

@robert zych Thanks. That’s what I was expecting. I only have a very limited set of queries to run — no need for fuzzy text search in the JSON. The operations on this specific column are few, and as far as I’ve tested, they all work fine. In most of the cases its value is comma seperated list and I have to check for element existance.

Sounds like a use case for

valueIn

Oh, why are you not using multi valued column?

ValueIn should do the job in most cases yes. I am not using MV columns because the table was designed some time ago while we weren't 100% sure we will use Pinot in the long run. This is why I tried to use only simple data types and fumctions which are available in most of the modern OLAP solutions.

@Kiril Kalchev In a more recent version,

UpsertCompactMergeTask

was added with the additional capability of merging segments. Would you be able to see if it also truncates long strings?

Can you try upgrading the Pinot version to latest

1.3.0

? I didn't face any issues with the latest version https://github.com/apache/pinot/pull/13157 Looks like this PR fixed that

Thanks @deepthi ch

@Mayank @robert zych @Chaitanya Deepthi Chadalavada I tested with 1.3.0 locally and it worked as expected. I will try to prioritize 1.3.0 upgrade in production as well. Thanks everyone. I won't open new issue than as it works in 1.3.0

Yes no issue needed

fyi, commit time compaction was merged recently https://github.com/apache/pinot/pull/16344

@robert zych Thanks. I just want to confirm—does this functionality replace

UpsertCompactionTask

? Also, if I remember correctly, I’ll need version 1.4.0 to enable

UpsertCompactMergeTask

, since there was an issue with it in 1.3.0. I believe I already discussed this with @Mayank few weeks ago.

@Kiril Kalchev Yes, you can use commit time compaction instead of the

UpsertCompactionTask

if you need to compact your segments asap