Has anyone else encountered the `Expression cycle`...
# general
r
Has anyone else encountered the
Expression cycle
problem when using an ingest transform function where the source field name matches the pinot column name? In my case I have geojson that has a geometry field containing coordinates (
{"geometry": {"coordinates": [-82.41959120338983, 35.61020539194915]}}
). I'm looking for a way to convert those coordinates using
toSphericalGeography(stPoint(lon, lat))
into a column also named geometry.
@Xiang Fu Are you familiar with this
Expression cycle
issue? If so, would this require significant effort to fix?
x
what's the error ? or are you able to work around it with json, something like :
Copy code
"transformFunctions": {
  "rawGeometry": "jsonPathString(geometry, '$.coordinates')",
  "geometry": "toSphericalGeography(stPoint("
      + "jsonExtractScalar(rawGeometry, '$[0]', 'DOUBLE'), "
      + "jsonExtractScalar(rawGeometry, '$[1]', 'DOUBLE') ))"
}
oh, I think that's impossible because you defined
geometry
in the pinot schema
r
How much effort would it take to make this possible?
x
it's tricky because your source column:
geometry
data type is different than the destination column:
geometry
type
technically pinot will just read the column as the defined type and may throw exception before the transform happens
I would suggest to change to a new column in the pinot schema, in query, you can always make alias
r
I can't change the Pinot column name
I want to be able to change the data type without changing the name
x
then need to check how to use
TransformPipeline
to process those rows
👀 1
r
@Xiang Fu The issue appears to be that
ExpressionTransformer.topologicalSort
treats all function arguments as pinot columns that require circular dependency checking. What are your thoughts on allowing self-referencing (argument name matches pinot column name) when the
functionEvaluator
contains JSON and other similar functions? For example,
geometry = toSphericalGeography(stPoint(JSONPATHDOUBLE(geometry, '$.coordinates[0]'), JSONPATHDOUBLE(geometry, '$.coordinates[1]')))
x
I think we can add a config to override this column
r
In which way could the column be overridden?