https://pinot.apache.org/ logo
#general
Title
# general
j

Joey Pereira

08/19/2020, 5:13 PM
I was having some trouble making a schema/table with transform configs, following the documentation at https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#column-transformation (=> thread)
n

Neha Pawar

08/19/2020, 5:16 PM
what sort of trouble?
j

Joey Pereira

08/19/2020, 5:17 PM
So I have a schema
Copy code
{
  "schemaName": "test1",
  "dimensionFieldSpecs": [
    {
      "name": "id",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "amount",
      "dataType": "DOUBLE"
    }
  ],
  "ingestionConfig": {
    "transformConfigs": [
      {
        "columnName": "transform_field_1",
        "transformFunction": "Groovy({id}, id)"
      }
    ]
  },
  "dateTimeFieldSpecs": [
    {
      "name": "created_at",
      "dataType": "LONG",
      "format": "1:SECONDS:EPOCH",
      "granularity": "15:MINUTES"
    }
  ]
}
table
Copy code
{
  "tableName": "test1",
  "tableType": "OFFLINE",
  "routing": {
    "segmentPrunerType": "partition"
  },
  "segmentsConfig": {
    "timeColumnName": "created_at",
    "timeType": "SECONDS",
    "replication": "1",
    "schemaName": "test1",
    "segmentPushFrequency": "HOURLY",
    "segmentPushType": "APPEND"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP"
  },
  "tenants": {},
  "metadata": {}
}
when I was uploading it to Pinot via
UpdateSchema
and
AddTable
, it looks like the
ingestionConfig
does not even make it to the table and when I run GET
/tables/test1/schema
, the schema I get back is
Copy code
{
  "schemaName": "test1",
  "dimensionFieldSpecs": [
    {
      "name": "id",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "amount",
      "dataType": "DOUBLE"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "created_at",
      "dataType": "LONG",
      "format": "1:SECONDS:EPOCH",
      "granularity": "15:MINUTES"
    }
  ]
}
n

Neha Pawar

08/19/2020, 5:17 PM
the ingestion config should be in the tableConfig
not in the schema
let me clarify that on the doc
j

Joey Pereira

08/19/2020, 5:19 PM
Oh no! It looks like the docs are probably clear about that, but I totally missed it
Thanks a ton! (:
n

Neha Pawar

08/19/2020, 5:20 PM
you’re welcome!
2 Views