I was having some trouble making a schema/table with transform configs, following the documentation at https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#column-transformation (=> thread)

what sort of trouble?

So I have a schema

{
  "schemaName": "test1",
  "dimensionFieldSpecs": [
    {
      "name": "id",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "amount",
      "dataType": "DOUBLE"
    }
  ],
  "ingestionConfig": {
    "transformConfigs": [
      {
        "columnName": "transform_field_1",
        "transformFunction": "Groovy({id}, id)"
      }
    ]
  },
  "dateTimeFieldSpecs": [
    {
      "name": "created_at",
      "dataType": "LONG",
      "format": "1:SECONDS:EPOCH",
      "granularity": "15:MINUTES"
    }
  ]
}

table

{
  "tableName": "test1",
  "tableType": "OFFLINE",
  "routing": {
    "segmentPrunerType": "partition"
  },
  "segmentsConfig": {
    "timeColumnName": "created_at",
    "timeType": "SECONDS",
    "replication": "1",
    "schemaName": "test1",
    "segmentPushFrequency": "HOURLY",
    "segmentPushType": "APPEND"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP"
  },
  "tenants": {},
  "metadata": {}
}

when I was uploading it to Pinot via

UpdateSchema

and

AddTable

, it looks like the

ingestionConfig

does not even make it to the table and when I run GET

/tables/test1/schema

, the schema I get back is

{
  "schemaName": "test1",
  "dimensionFieldSpecs": [
    {
      "name": "id",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "amount",
      "dataType": "DOUBLE"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "created_at",
      "dataType": "LONG",
      "format": "1:SECONDS:EPOCH",
      "granularity": "15:MINUTES"
    }
  ]
}

the ingestion config should be in the tableConfig

Oh no! It looks like the docs are probably clear about that, but I totally missed it

you’re welcome!