Hello, I am trying to use trino connector and running into following error while trying to query pinot via trino

>>> import trino
>>> conn = trino.dbapi.connect(host='<redacted>', port=8443, catalog='pinot', schema='default', http_scheme='https', auth=trino.auth.BasicAuthentication("xxx", "yyyy"))
>>> cur = conn.cursor()
>>> cur.execute('SELECT * FROM mytable LIMIT 10')
<trino.client.TrinoResult object at 0x10428d160>
>>> rows = cur.fetchall()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/trino/dbapi.py", line 558, in fetchall
    return list(self.genall())
  File "/usr/local/lib/python3.8/site-packages/trino/client.py", line 509, in __iter__
    rows = self._query.fetch()
  File "/usr/local/lib/python3.8/site-packages/trino/client.py", line 677, in fetch
    status = self._request.process(response)
  File "/usr/local/lib/python3.8/site-packages/trino/client.py", line 440, in process
    raise self._process_error(response["error"], response.get("id"))
trino.exceptions.TrinoQueryError: TrinoQueryError(type=INTERNAL_ERROR, name=GENERIC_INTERNAL_ERROR, message="Failed communicating with server: <http://pinot-broker-1.pinot-broker-headless.pinot-dev-ns.svc.cluster.local:8099/debug/routingTable/mytable>", query_id=20220603_211510_00025_9srer)

I am using the external ip of the loadbalancer i.e

service/pinot-controller-external

with port 9000 for

pinot.controller-urls

. If it helps, I am using community provided helm chart to stand up the pinot infrastructure on AWS EKS.

Trino has to be deployed in the same k8s cluster as pinot

I see, so no way to use it without deploying to the same k8s cluster?

Right, so far there is no way to do it from k8s, we are working on a pinot rest proxy solution for this

Thank you Xiang. Is there a timeline for when that is planned to be released? How can I help?

The code is pretty much completed, we are just testing it internally and discuss where to put it (shall it be a separated module or should be combined with broker, etc)

This is great. I am hoping this will also enable us to integrate with Trino / Presto that is not deployed via k8s. I am guessing this will also be an issue with Superset. Will that be taken care of via this change as well or will that need additional work?

The code is testing in our production, plan to open source it soon.

Hey Xiang, do you think external-dns might work?

hmmm, pinot internal registered instances using internal names

right, but trino only need to have an ingress gateway for broker service right? or would i need dns endpoint for every broker pod? and server pod?

no, it requires both broker and servers

i see, but just the service endpoint or would it need an endpoint for each broker and server prod? i'm going to try it out.

it needs every pod access

I see

Hi @Xiang Fu -- We are also looking for this solution. Do let us know when we can test this feature.

Hi, we are planning to open source it

That is great to hear. Is there a tentative timeline ?

I can make the proxy a startree community version out soon maybe this week or next week