Hello has anyone experienced 100 degradation in ingestion ra Apache Druid #general

Hello, has anyone experienced 100% degradation in ...

Sharmin Choksey

08/22/2024, 7:12 AM

Hello, has anyone experienced 100% degradation in ingestion rates when using kafka.timestamp as the primary time series dimension? This is on a spec with 70+ dimensions and uses protobuf decoder

Sharmin Choksey

08/22/2024, 7:38 AM

To be precise inputFormat is kafka, valueFormat is protobuf

Sharmin Choksey

08/22/2024, 7:47 AM

The ingestion rate drops by half when the ingestion spec is configured to parse kafka headers and timestamp

Benjamin Hopp

08/22/2024, 3:14 PM

What was the

__time

field previously? I wouldn't expect the

kafka.timestamp

to perform much differently than an event timestamp but if you were using a placeholder value such as

1970-01-01

things like rollup and the number of segments being generated could be drastically different.

Sharmin Choksey

08/22/2024, 4:14 PM

Let me clarify the question, its not so much related to what

timestampSpec

column is used, whether kafka.timestamp or an event timestamp, but more to do with parsing of kafka headers and timestamp.

Sharmin Choksey

08/22/2024, 4:16 PM

Configuring the ingestion spec to parse kafka headers degrades the ingestion rate by half

Copy code

"ioConfig": {
      "topic": "topic-name",
      "topicPattern": null,
      "inputFormat": {
        "type": "kafka",
        "headerFormat": null,
        "keyFormat": null,
        "valueFormat": {
          "type": "protobuf",
          "flattenSpec": {
            "useFieldDiscovery": true,

versus

Copy code

"ioConfig": {
      "topic": "topic-name"",
      "topicPattern": null,
      "inputFormat": {
        "type": "protobuf",
        "flattenSpec": {
          "useFieldDiscovery": true,

Note

inputFormat

- kafka with

valueFormat

- protobuf v/s

inputFormat

- protobuf (no kafka header parsing)

Open in Slack

Previous Next