https://linen.dev logo
Join Slack
Powered by
Hi, Elementary! Is it possible to increase timeout...
# community-support
d
Hi, Elementary! Is it possible to increase timeout time for the send-report operation? I get a timeout after a minute. Files size is 20 - 30 mb.
Copy code
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError('The write operation timed out'))
i
Hey @Dmitrii Khriashchikov can you share with me your edr.log? I want to understand better where on the flow you are getting the timeout
d
Hi! Thank you for your involvement. It happen on 
Uploading to GCS bucket
step.
one moment, i'll attach a fresh log
edr.log
edr.log
terminal output
Copy code
(pdbt) dima@LAPTOP-230F3JMD:~/src/jw-bi-dbt/src$ edr send-report --google-service-account-path /home/dima/airflow_sa_hc.json --gcs-bucket-name jw-bi-warehouse-test-results --env prod
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2023-11-27 14:34:31 — INFO — Running with edr=0.12.0
2023-11-27 14:34:39 — INFO — edr (0.12.0) and Elementary's dbt package (0.12.1) are compatible.
2023-11-27 14:34:42 — INFO — Elementary's database and schema: '"dbt.elementary"'
2023-11-27 14:34:42 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/dbt_project
2023-11-27 14:35:11 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/dbt_project
2023-11-27 14:35:17 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/dbt_project
2023-11-27 14:35:23 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/dbt_project
2023-11-27 14:35:28 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/dbt_project
2023-11-27 14:36:21 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/dbt_project
2023-11-27 14:36:28 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/dbt_project
2023-11-27 14:36:34 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/dbt_project
2023-11-27 14:36:39 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/dbt_project
2023-11-27 14:36:47 — INFO — Uploading to GCS bucket "jw-bi-warehouse-test-results"
Traceback (most recent call last):
  File "/home/dima/pdbt/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/home/dima/pdbt/lib/python3.10/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/dima/pdbt/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1077, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.10/http/client.py", line 999, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.10/ssl.py", line 1237, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib/python3.10/ssl.py", line 1206, in send
    return self._sslobj.write(data)
TimeoutError: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dima/pdbt/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/dima/pdbt/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/home/dima/pdbt/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/dima/pdbt/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/home/dima/pdbt/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/home/dima/pdbt/lib/python3.10/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/dima/pdbt/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1077, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.10/http/client.py", line 999, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.10/ssl.py", line 1237, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib/python3.10/ssl.py", line 1206, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError('The write operation timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dima/pdbt/bin/edr", line 8, in <module>
    sys.exit(cli())
  File "/home/dima/pdbt/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/dima/pdbt/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/dima/pdbt/lib/python3.10/site-packages/elementary/cli/cli.py", line 67, in invoke
    return super().invoke(ctx)
  File "/home/dima/pdbt/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dima/pdbt/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dima/pdbt/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/dima/pdbt/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/cli.py", line 650, in send_report
    sent_report_successfully = data_monitoring.send_report(
  File "/home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/data_monitoring/report/data_monitoring_report.py", line 199, in send_report
    upload_succeeded, bucket_website_url = self.upload_report(
  File "/home/dima/pdbt/lib/python3.10/site-packages/elementary/monitor/data_monitoring/report/data_monitoring_report.py", line 241, in upload_report
    send_succeeded, bucket_website_url = self.gcs_client.send_report(
  File "/home/dima/pdbt/lib/python3.10/site-packages/elementary/clients/gcs/client.py", line 45, in send_report
    blob.upload_from_filename(local_html_file_path, content_type="text/html")
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2926, in upload_from_filename
    self._handle_filename_and_upload(
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2796, in _handle_filename_and_upload
    self._prep_and_do_upload(
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2604, in _prep_and_do_upload
    created_json = self._do_upload(
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2427, in _do_upload
    response = self._do_resumable_upload(
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2263, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport, timeout=timeout)
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/resumable_media/requests/upload.py", line 515, in transmit_next_chunk
    return _request_helpers.wait_and_retry(
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/resumable_media/requests/_request_helpers.py", line 178, in wait_and_retry
    raise error
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/resumable_media/requests/_request_helpers.py", line 155, in wait_and_retry
    response = func()
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/resumable_media/requests/upload.py", line 507, in retriable_request
    result = transport.request(
  File "/home/dima/pdbt/lib/python3.10/site-packages/google/auth/transport/requests.py", line 542, in request
    response = super(AuthorizedSession, self).request(
  File "/home/dima/pdbt/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/dima/pdbt/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/dima/pdbt/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError('The write operation timed out'))
i
Thanks for the info! Do you know after how long you get the connection error?
d
1 minute from the start of upload
i
@Dmitrii Khriashchikov I checked it and the default timeout for GCS is 60 seconds. Currently there is no way for you to change it, but I will create a fix for that.
@Dmitrii Khriashchikov I opened a PR that will support passing 
--gcs-timeout-limit
to the edr command which will allow you to expend the timeout limit. We will merge it soon, and you will be able to use it on the next version of the edr 🙂
d
Cool! Thank you!
m
Hello, I am currently facing the same issue, was this implemented?
Never mind, I used it and it works ;)
31 Views
Open in Slack
PreviousNext
Powered by