https://bentoml.com logo
Join Slack
Powered by
This message was deleted.
# ask-for-help
s
This message was deleted.
b
what platform are you on?
GKE?
k
No sorry I should have mentioned that. It is our k8s cluster.
b
On premise?
k
Yes
I found this in cert-manager troubleshooting https://cert-manager.io/docs/troubleshooting/webhook/#error-context-deadline-exceeded but I cannot pinpoint what is going wrong, since I am doing all the steps exactly as in docs.
b
which namespace did u install this on? could you output 
kubectl get svc -n <cert-manager-namespace>
?
k
Give me a min to get in touch with my devops.
Copy code
➜  ~ kubectl get svc -n cert-manager
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
cert-manager           ClusterIP   10.43.228.217   <none>        9402/TCP   34h
cert-manager-webhook   ClusterIP   10.43.253.27    <none>        443/TCP    34h
any idea on how to solve this?
b
Hmm, what does
Copy code
kubectl describe certificate -n cert-manager-test
say?
k
cert-manager-test
or 
cert-manager
?
b
You're right. 
cert-manager
😄
k
➜ ~ kubectl describe certificate -n cert-manager No resources found in cert-manager namespace.
I think this is after we create a cert-manager-test
Copy code
cat <<EOF > test-resources.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-test
---
apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1>
kind: Issuer
metadata:
  name: test-selfsigned
  namespace: cert-manager-test
spec:
  selfSigned: {}
---
apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1>
kind: Certificate
metadata:
  name: selfsigned-cert
  namespace: cert-manager-test
spec:
  dnsNames:
    - <http://example.com|example.com>
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned
EOF
b
Could u try running it again?
k
Same error as I mentioned above:
Copy code
~ kubectl apply -f test-resources.yaml
namespace/cert-manager-test unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "<http://webhook.cert-manager.io|webhook.cert-manager.io>": unexpected error when reading response body. Please retry. Original error: context deadline exceeded
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "<http://webhook.cert-manager.io|webhook.cert-manager.io>": unexpected error when reading response body. Please retry. Original error: context deadline exceeded
b
Strange ... that means it errors out when creating ther issuer?
k
From https://cert-manager.io/docs/troubleshooting/webhook/#error-context-deadline-exceeded
The trouble with the message 
context deadline exceeded
is that it obfuscates the part of the HTTP connection that timed out. When this message appears, we can't tell which part of the HTTP interaction timed out. It might be the DNS resolution, the TCP handshake, the TLS handshake, sending the HTTP request or receiving the HTTP response.
b
I assume there's nothing:
Copy code
kubectl get issuer -A
yeah that errror is almost like a catchall
k
➜ ~ kubectl get issuer -A No resources found
b
Have you tried
Copy code
kubectl -n cert-manager port-forward deploy/cert-manager-webhook 10250
k
I was just trying to do that.
b
I wonder if you have to open port 10250
k
Give me a sec, it is already in use.
I cannot do it right now, Do you any suggestions to look into ?
b
I'll check if you can talk to the webhook in the first place or not (i.e. follow the debugging steps)
Or you could try changing the port from 10250 to something like 443 to see if that changes anything or not
b
@Kostas Vasilopoulos did you get it resolved?
k
Hi Bo, thanks for reaching out. No not really. We tried debugging cert-manager using https://cert-manager.io/docs/troubleshooting/webhook/, but we had no luck. We port forward to 10250 but when we made the request.
Copy code
➜  ~ curl -vsS --resolve cert-manager-webhook.cert-manager.svc:10250:127.0.0.1 \
    --service-name cert-manager-webhook-ca \
    --cacert <(kubectl -n cert-manager get secret cert-manager-webhook-ca -ojsonpath='{.<http://data.ca|data.ca>\.crt}' | base64 -d) \
    <https://cert-manager-webhook.cert-manager.svc:10250/validate>
* Added cert-manager-webhook.cert-manager.svc:10250:127.0.0.1 to DNS cache
* Uses proxy env variable no_proxy == 'cert-manager-webhook.cert-manager.svc'
* Hostname cert-manager-webhook.cert-manager.svc was found in DNS cache
*   Trying 127.0.0.1:10250...
* TCP_NODELAY set
* Connected to cert-manager-webhook.cert-manager.svc (127.0.0.1) port 10250 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /proc/self/fd/11
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: <https://curl.haxx.se/docs/sslcerts.html>

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
and then using 
-k
arg to get our around SSL certificate error we get.
Copy code
~ curl -k -vsS --resolve cert-manager-webhook.cert-manager.svc:10250:127.0.0.1 \
    --service-name cert-manager-webhook-ca \
    --cacert <(kubectl -n cert-manager get secret cert-manager-webhook-ca -ojsonpath='{.<http://data.ca|data.ca>\.crt}' | base64 -d) \
    <https://cert-manager-webhook.cert-manager.svc:10250/validate>
* Added cert-manager-webhook.cert-manager.svc:10250:127.0.0.1 to DNS cache
* Uses proxy env variable no_proxy == 'cert-manager-webhook.cert-manager.svc'
* Hostname cert-manager-webhook.cert-manager.svc was found in DNS cache
*   Trying 127.0.0.1:10250...
* TCP_NODELAY set
* Connected to cert-manager-webhook.cert-manager.svc (127.0.0.1) port 10250 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /proc/self/fd/11
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=95.217.221.39@1634235922
*  start date: Oct 14 17:25:22 2021 GMT
*  expire date: Oct 14 17:25:22 2022 GMT
*  issuer: CN=95.217.221.39-ca@1634235922
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b718f3f2f0)
> GET /validate HTTP/2
> Host: cert-manager-webhook.cert-manager.svc:10250
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 404
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 19
< date: Mon, 14 Nov 2022 08:31:24 GMT
<
404 page not found
* Connection #0 to host cert-manager-webhook.cert-manager.svc left intact
If you have any insight it would be much appreciated, although I the bug is with cert-manager and not with bentoml.
Hi guys, we installed a couple vms to our k8s cluster to make a fresh install of everything and we managed to isntall almost everything, but we have an issue in the last step.
Copy code
2022-11-15T01:26:24+02:00 time="2022-11-14T23:26:24Z" level=info msg="Creating ingress default-domain- to get a ingress IP automatically"
2022-11-15T01:26:24+02:00 time="2022-11-14T23:26:24Z" level=info msg="Waiting for ingress default-domain-kw4qp to be ready"
2022-11-15T01:46:24+02:00 panic: Error getting domain suffix: failed to wait for ingress default-domain-kw4qp to be ready: timed out waiting for the condition
11 Views
Open in Slack
PreviousNext
Powered by