This message was deleted.

I opened up a GitHub issue and documented my findings in this thread. It seems that BentoML cannot authenticate with NewRelic, yet the OptenTelemetry collector can. Now, I'm trying a new approach where I have BentoML send traces to an OpenTelemetry collector rather than trying have the BentoML send them to NR directly, since BentoML's auth seems to be broken, while the `otel-collector`'s auth is working. Here are error logs that BentoML is emmitting when I make a request (and then wait ~60s)

2023-04-03T22:07:43+0000 [INFO] [dev_api_server] 172.23.0.1:45386 - "POST /predict HTTP/1.1" 200 (trace=7a60207579db22012547e841b045ed2c,span=c66ba9d11bcba166,sampled=1)

2023-04-03T22:07:43+0000 [INFO] [dev_api_server] 172.23.0.1:45386 (scheme=http,method=POST,path=/predict,type=application/json,length=16) (status=200,type=application/json,length=25) 2055.953ms (trace=7a60207579db22012547e841b045ed2c,span=b6e16c77395222b8,sampled=1)

I0403 22:07:46.646503465      39 <http://socket_utils_common_posix.cc:407]|socket_utils_common_posix.cc:407]>     Disabling AF_INET6 sockets because ::1 is not available.

2023-04-03T22:07:56+0000 [WARNING] [dev_api_server] Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces, retrying in 1s.

2023-04-03T22:08:07+0000 [WARNING] [dev_api_server] Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces, retrying in 2s.

2023-04-03T22:08:19+0000 [WARNING] [dev_api_server] Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces, retrying in 4s.

2023-04-03T22:08:33+0000 [WARNING] [dev_api_server] Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces, retrying in 8s.

2023-04-03T22:08:51+0000 [WARNING] [dev_api_server] Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces, retrying in 16s.

2023-04-03T22:09:07+0000 [WARNING] [dev_api_server] Transient error StatusCode.UNAVAILABLE encountered while exporting traces, retrying in 32s.