datahub version : v0.9.6.1 datahub cli : 0.9.6.4 ...
# ingestion
r
# datahub version : v0.9.6.1 # datahub cli : 0.9.6.4 Hello Is there any way to specify query_max_execution while using trino source. I need to set it to 14400sec or more. Right now query gets timed out after 10mins. On asking the trino admins they said that this property is configurable on the client side. Error:
Copy code
[2023-04-30 18:24:05,254] ERROR    {datahub.utilities.sqlalchemy_query_combiner:403} - Failed to execute queue using combiner: (trino.exceptions.TrinoQueryError) TrinoQueryError(type=INSUFFICIENT_RESOURCES, name=EXCEEDED_TIME_LIMIT, message="Query exceeded the maximum execution time limit of 10.00m"

["Profiling exception (trino.exceptions.OperationalError) error 404: b'Query not found'\n(Background on this error at: <https://sqlalche.me/e/14/e3q8>)"]
Recipe yaml:
Copy code
source:
  type: "trino"
  config:
    host_port: ip:port
    database: hive_2

    username: tr
    password:

    schema_pattern:
      deny:
        - .*information_schema.*
      allow:
        - B
        - A

    table_pattern:
      allow:
        - hive_2.A.table1
        - hive_2.B.table2
   

    profiling:
      enabled: True

    profile_pattern:
      allow:
       - hive_2.A.table1
        - hive_2.B.table2

transformers:
  - type: "simple_add_dataset_tags"
    config:
      tag_urns:
        - "urn:li:tag:1_0_prod_datalake"

pipeline_name: "trino_hive_prod_to_datahub_prod"

datahub_api:
  server: "<https://gms:8080>"
  token: 
  
  
sink:
  type: "datahub-rest"
  config:
    server: "<https://gms:8080>"
    token:
🔍 1
1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: 1️⃣ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? Yes button ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? ✅ button
r
Trying this: mentioned here:
Copy code
connect_args={
      "session_properties": {'query_max_run_time': '1d'},
      "client_tags": ["tag1", "tag2"],
      "roles": {"catalog1": "role1"},
    }
This worked: Thanks to our trino admins. query_max_execution_time
Copy code
options:
     connect_args:
       http_scheme: http
       session_properties:{'query_max_execution_time': '1d'}