This message was deleted.

The plan sounds correct to me. Looking forward to learning the results!

Okay, I'm having an issue. (2) I believe was flawed in that we do not need an OpenTelemetry config file. We'd only need that if we were running our own collector, and in this case, NewRelic is the collector. I can't get bento to respect the settings I'm trying to use to send the OTel information to NewRelic. Here's my docker-compose file:

version: "3.8"

services:
  bento-service:
    image: dummy-service:latest
    ports:
      - "3000:3000"
    volumes:
      - .:/home/bentoml/bento/src/
      - ./bentoml_configuration.yaml:/home/bentoml/bentoml_configuration.yaml
    command: serve --port 3000 --debug # have tried with --production as well
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: <https://otlp.nr-data.net>
      # I've tried enabling/disabling this
      OTEL_EXPORTER_OTLP_HEADERS: api-key=${NEW_RELIC_LICENSE_KEY},service.name=opentelemetry-bento-svc
      BENTO_CONFIG: /home/bentoml/bentoml_configuration.yaml
    env_file:
      - .env

And here's my

bentoml_configuration.yaml

version: 1
api_server:
  tracing:
    enabled: true
    exporter_type: otlp
    sample_rate: 1.0
    otlp:
      protocol: grpc
      endpoint: <https://otlp.nr-data.net>
      grpc:
        insecure: false
        headers:
          - ["grpc-encoding", "gzip"]
          - ["api-key", "${NEW_RELIC_LICENSE_KEY}"]
          - ["service.name", "opentelemetry-bento-svc"]

But when I run

docker-compose up

, bentoml gives me this warning

[DEBUG] [dev_api_server] 'tracing.sample_rate' is set to zero. No traces will be collected. Please refer to <https://docs.bentoml.org/en/latest/guides/tracing.html> for more details.

Why wouldn't bento be respecting my

bento_configuration.yaml

?

The environment variable for your bento config path should be

BENTOML_CONFIG

Hah, that did it! Now it doesn't like the contents of the file, but that's progress 😄

you can also specify that environment variable in your

bentofile.yaml

, under the

docker.env

key: https://docs.bentoml.org/en/latest/concepts/bento.html#docker-options

Oh beautiful 🤩

I regret that I won't be much help in that regard...I haven't used New Relic in years

Oh no worries, I'm using this thread generally to book keep the progress on this, I wasn't directing this at you specifically 😅

Gotcha 🙂

For completeness, this is the context of the message:

dummy-service-bento-service-1  | 2023-03-17T20:06:44+0000 [INFO] [dev_api_server] 172.22.0.1:59704 - "POST /predict HTTP/1.1" 200 (trace=e7fde7960a744d368c34896b26d3544c,span=1b911bcd8e57b82d,sampled=1)
dummy-service-bento-service-1  | 2023-03-17T20:06:44+0000 [INFO] [dev_api_server] 172.22.0.1:59704 (scheme=http,method=POST,path=/predict,type=application/json,length=16) (status=200,type=application/json,length=25) 2068.939ms (trace=e7fde7960a744d368c34896b26d3544c,span=0387ffb4f6eb1d1c,sampled=1)
dummy-service-bento-service-1  | I0317 20:06:44.237912919      39 <http://socket_utils_common_posix.cc:407]|socket_utils_common_posix.cc:407]>     Disabling AF_INET6 sockets because ::1 is not available.
dummy-service-bento-service-1  | 2023-03-17T20:06:44+0000 [ERROR] [dev_api_server] Failed to export traces, error code: StatusCode.PERMISSION_DENIED