Hey friends, I am trying to convert timezone based on location field with transformation functions, but the real-time table does not accept the config and after save the page goes blank..

"ingestionConfig": {

"filterConfig": null,

"transformConfigs": [

{

"columnName": "e_time_stamp",

"transformFunction": "Groovy({timestamp + ' Australia/Melbourne'}, timestamp)"

},

{

"columnName": "epoc_time_stamp",

"transformFunction": "fromDateTime(e_time_stamp, 'yyyy-MM-dd''T''HH:mm:ssZZZ')"

},

{

"columnName": "a_time_stamp",

"transformFunction": "Groovy({state == 'QLD' ? DATETIMECONVERT(epoc_time_stamp, '1:MILLISECONDS:EPOCH', '1:SECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd''T''HH:mm:ss tz(Australia/Queensland)', '1:SECONDS') : DATETIMECONVERT(epoc_time_stamp, '1:MILLISECONDS:EPOCH', '1:SECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd''T''HH:mm:ss tz(Australia/Melbourne)', '1:SECONDS')}, state)"

}

]

},

Do you have access to controller log? May be there's an error in the log?

not much in the controller log:

2021/09/12 04:49:43.585 INFO [StartServiceManagerCommand] [main] Started Pinot [CONTROLLER] instance [Controller_10.12.2.133_9000] at 49.056s since launch

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/pinot/lib/pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar) to method java.lang.Object.finalize()

WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

2021/09/12 23:08:30.327 ERROR [CompletionServiceHelper] [grizzly-http-server-28] Server: Server_10.12.2.133_8098 returned error: 404

2021/09/12 23:11:42.162 ERROR [CompletionServiceHelper] [grizzly-http-server-23] Server: Server_10.12.2.133_8098 returned error: 404

2021/09/12 23:20:55.468 ERROR [CompletionServiceHelper] [grizzly-http-server-19] Server: Server_10.12.2.133_8098 returned error: 404

2021/09/12 23:33:41.371 ERROR [CompletionServiceHelper] [grizzly-http-server-7] Server: Server_10.12.2.133_8098 returned error: 404

Seems like you are getting a 404

yeah but i can't figure out whats wrong with my transformation configs

Did you try swagger

yes this is the following error:

{
  "_code": 400,
  "_error": "Invalid TableConfigs. Missing required creator property 'schema' (index 1)\n at [Source: (String)\"{\n  \"tableName\": \"speedtest4\",\n  \"tableType\": \"REALTIME\",\n  \"tenants\": {\n    \"broker\": \"DefaultTenant\",\n    \"server\": \"DefaultTenant\",\n    \"tagOverrideConfig\": {}\n  },\n  \"segmentsConfig\": {\n    \"schemaName\": \"speedtest4\",\n    \"timeColumnName\": \"e_time_stamp\",\n    \"replication\": \"1\",\n    \"replicasPerPartition\": \"1\",\n    \"retentionTimeUnit\": null,\n    \"retentionTimeValue\": null,\n    \"completionConfig\": null,\n    \"crypterClassName\": null,\n    \"peerSegmentDownloadScheme\": null\n  },\n  \"tableIndexConf\"[truncated 2375 chars]; line: 87, column: 1] (through reference chain: org.apache.pinot.spi.config.TableConfigs[\"schema\"])"
}

You cannot use nested functions in groovy. I see datetimeconvert being used inside groovy in the last one

Thanks @Neha Pawar , we need to surface this back to the user. Do you mind filing an issue?

it does get surfaced back as a groovy format error and

throw new IllegalStateException(
    "Invalid transform function '" + transformFunction + "' for column '" + columnName + "'");

I’m not sure we can detect easily that a nested transform function is being done in groovy. We can only detect that it is invalid Groovy. Maybe the message can be enhanced

thanks @Neha Pawar and @Mayank