Hi Team, can some one help me on some configurati...
# general
v
Hi Team, can some one help me on some configuration in UPSERT.
APPEND
and
UNION
strategies. how to configure it?
@saurabh dubey can you please help me on this?
s
These are partial upsert strategies. If for a given column you choose
APPEND
, the value from the new record being ingested will be appended to the last column's value.
UNION
will merge the incoming value with existing and only retain the unique set amongst them. Both strategies are only applicable for multi-value columns
Any specific question about these?
v
i am trying to configure it but getting a error
let me share you here
Copy code
"upsertConfig": {
        "mode": "PARTIAL",
        "defaultPartialUpsertStrategy": "OVERWRITE",
        "partialUpsertStrategies":{
            "uuid":"APPEND"
        }
    },
this is the configuration
Copy code
{
    "code": 400,
    "error": "APPEND merger cannot be applied to single-value column: uuid"
}'
s
Like I said, APPEND strategy is only applicable to multivalue columns. Your uuid field is a single value string column in the schema correct?
Could you describe what is it that you're trying to achieve?
v
we are not trying to achieve here but we are tying to explore features so we are just implementing it but getting above error
Copy code
{
  "schemaName": "tab",
  "dimensionFieldSpecs": [
    {
      "name": "key",
      "dataType": "INT"
    },
    {
      "name": "uuid",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "count",
      "dataType": "INT"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "ts",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ],
  "primaryKeyColumns": [
    "key"
  ]
}
s
I meant to say while ingesting data, if a new row with the same PK is seen, what do you wish the uuid should be updated to? new value / old value / some combination of new and old value?
v
we want to create new column as fullname from firstname and last name
s
don't think I fully understand it.. Is firstName and lastName columns in your schema too?
v
its not there in schema, i just want to know how to configure append and union in tablE?
r
To generate fullName you'll need an ingestionConfig...
Copy code
"ingestionConfig": {
    "transformConfigs": [{
      "columnName": "fullName",
      "transformFunction": "Groovy({firstName+' '+lastName}, firstName, lastName)"
    }]
}
https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations