Hi, isn’t there any method to track the GPU scheduling of BentoML? When we run our BentoML container, it first uses GPU for incoming requests. However, after several days, it only processes requests only using CPU although the container uses the GPU memory. Because there are no error messages at all inside the container and the glitch is not raised immediately when we first run the container, it is very difficult for us to find the reason of the problem and fix it.

Could you share more about your setup and service code? Typically you can do this via something like nvtop

No, currently we are using bentoml container in self-hosted server. The BentoML version is 1.1.10. I don’t know if it could be helpful but I will share our configure.yaml file FYI. Also, we are launching the container with following command to limit the container to access for specific GPU.

docker run -d -p $HOST_PORT:3000 -v $CONFIG_SRC:$CONFIG_TRG -e BENTOML_CONFIG=$CONFIG_TRG --gpus '"device=3"' --name $CONTAINER_NAME $IMAGE_NAME:$TAG serve

version: 1
api_server:
  workers: 12 # cpu_count() will be used when null
  backlog: 2048
  # the maximum number of connections that will be made to any given runner server at once
  max_runner_connections: 16
  traffic:
    timeout: 7200
    max_concurrency: ~
  metrics:
    enabled: true
    namespace: bentoml_api_server
    duration:
      # <https://github.com/prometheus/client_python/blob/f17a8361ad3ed5bc47f193ac03b00911120a8d81/prometheus_client/metrics.py#L544>
      buckets: [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0]
      min: ~
      max: ~
      factor: ~
  logging:
    access:
      enabled: true
      request_content_length: true
      request_content_type: true
      response_content_length: true
      response_content_type: true
      format:
        trace_id: 032x
        span_id: 016x
  ssl:
    enabled: false
    certfile: ~
    keyfile: ~
    keyfile_password: ~
    ca_certs: ~
    version: 17 # ssl.PROTOCOL_TLS_SERVER
    cert_reqs: 0 # ssl.CERT_NONE
    ciphers: TLSv1 # default ciphers
  http:
    host: 0.0.0.0
    port: 3000
    cors:
      enabled: false
      access_control_allow_origins: ~
      access_control_allow_credentials: ~
      access_control_allow_methods: ~
      access_control_allow_headers: ~
      access_control_allow_origin_regex: ~
      access_control_max_age: ~
      access_control_expose_headers: ~
    response:
      trace_id: false
  grpc:
    host: 0.0.0.0
    port: 3000
    max_concurrent_streams: ~
    maximum_concurrent_rpcs: ~
    max_message_length: -1
    reflection:
      enabled: false
    channelz:
      enabled: false
    metrics:
      host: 0.0.0.0
      port: 3001
  runner_probe: # configure whether the API server's health check endpoints (readyz, livez, healthz) also check the runners
    enabled: true
    timeout: 1
    period: 10
runners:
  resources: ~
  workers_per_resource: 1
  traffic:
    timeout: 7200
    max_concurrency: ~
  batching:
    enabled: false
    max_batch_size: 2
    max_latency_ms: 90000000000
  logging:
    access:
      enabled: false
      request_content_length: true
      request_content_type: true
      response_content_length: true
      response_content_type: true
  metrics:
    enabled: false
    namespace: bentoml_runner
tracing:
  exporter_type: ~
  sample_rate: ~
  excluded_urls: ~
  timeout: ~
  max_tag_value_length: ~
  zipkin:
    endpoint: ~
    local_node_ipv4: ~
    local_node_ipv6: ~
    local_node_port: ~
  jaeger:
    protocol: thrift
    collector_endpoint: ~
    thrift:
      agent_host_name: ~
      agent_port: ~
      udp_split_oversized_batches: ~
    grpc:
      insecure: ~
  otlp:
    protocol: ~
    endpoint: ~
    compression: ~
    http:
      certificate_file: ~
      headers: ~
    grpc:
      headers: ~
      insecure: ~
monitoring:
  enabled: true
  type: default
  options:
    log_config_file: ~
    log_path: monitoring

I am serving as self hosted right now. What we did here was: We followed this guide here. https://cloud.google.com/stackdriver/docs/managed-prometheus/exporters/nvidia-dcgm Then we know the GPU status on prometheus.