wave Does `ingestionConfig transformConfigs` occur before ` Apache Pinot #troubleshooting

:wave: Does `ingestionConfig.transformConfigs` occ...

Jonathan Meyer

04/09/2021, 2:55 PM

👋 Does

ingestionConfig.transformConfigs

occur before

ingestionConfig.filterConfig

? [realtime table] I'm seeing errors on some messages due to invalid transformations (-> missing field - since this table isn't for this type of event)

Jonathan Meyer

04/09/2021, 3:02 PM

Maybe filtering isn't working Here's how it's configured

table config

Copy code

"ingestionConfig": {
    "filterConfig": {
      "filterFunction": "Groovy({type != \"com.insideboard.user-event.v1.post.like\"}, type)"
    },

and the events that should have been filtered but generated transformation errors

event

Copy code

{
  "id": "1935a299-9f7c-43d4-a571-6c5cfdcf30cb",
  "source": "urn:insideboard:microservices:user-events",
  "specversion": "1.0",
  "type": "com.insideboard.user-event.v1.document.open",
  "time": "2021-04-09T14:53:13.812Z",
  "data": {
    "name": "document.open",
    "documentId": "60705cf9106a42150318d198",
    "projectId": "60646df59c4dbcdaf6e620bf",
    "origin": "DOCUMENTATION_PROJECT",
    "status": 200,
    "userId": "6049f2c188d36401807cf526",
    "eventTime": "2021-04-09T14:53:13.705Z"
  },
  "ibcustomer": "demo",
  "x-b3-traceid": "b403e6464ab3991ba26a4aade442705e",
  "x-b3-parentspanid": "7f497c2501e745a8",
  "x-b3-spanid": "25ecdb6fec5b4840",
  "x-b3-sampled": "1"
}

Jonathan Meyer

04/09/2021, 3:03 PM

transformation_error.log

Neha Pawar

04/09/2021, 3:47 PM

Yes transformation occurs before filter.

Jonathan Meyer

04/09/2021, 3:48 PM

Interesting, thanks @Neha Pawar

Neha Pawar

04/09/2021, 3:48 PM

So that you have the option to filter on the result of a transform

Jonathan Meyer

04/09/2021, 3:49 PM

Yeah, makes sense, I just thought about that use case 🙂 So there's no way to "hide" those error messages ?

Neha Pawar

04/09/2021, 3:52 PM

Only thing I can think of is if we make the jsonPathString function null safe

Neha Pawar

04/09/2021, 3:52 PM

@Xiang Fu should we change jsonPathString to handle null input?

Jonathan Meyer

04/09/2021, 3:52 PM

Ah yes, indeed Thanks 🙂

Jonathan Meyer

04/09/2021, 3:53 PM

I've had success adding default value to

jsonPathString

- so that should "fix" my "issue" indeed

Neha Pawar

04/09/2021, 3:54 PM

Oh great

Xiang Fu

04/09/2021, 5:01 PM

I think we should fix the null input to get null output for sure

Xiang Fu

04/09/2021, 5:01 PM

please create a github issue

Jonathan Meyer

04/09/2021, 5:24 PM

@Xiang Fu so change the behavior from error (missing field) to using the null value for the given type (target column) ?

Xiang Fu

04/09/2021, 5:39 PM

right, or maybe an empty json

Open in Slack

Previous Next