https://bentoml.com logo
Join Slack
Powered by
This message was deleted.
# ask-for-help
s
This message was deleted.
👀 1
a
yes, we support arize as a built-in monitoring data collectors
the default is always to json file
cc @Jiang
j
Is there something that works on-premise? I just want to have a small application outside of cloud on my hardware.
j
Both custom metrics and Monitor API are suitable for on-premises deployment. The former allows you to directly expose the metrics you need, which are collected by your Prometheus along with other built-in metrics of the server. You can query them directly. The latter is used to collect raw data, which is saved in the form of JSON logs in a path when working locally. This allows you to perform more advanced analysis.
@Jan Palasek
It seems Custom Metrics are what you are looking for
The Doc should have give an example. I'd like to provide more details if needed
j
Thank you. I need to compute some drift metrics. I think that doing that per request is going to be too slow and therefore I need the “custom analysis”. If I understand correctly, the raw data will be shipped out of bentoml and I need to provide it as metrics for prometheus on my own 🙁
@Jiang Is there some kind of tutorial for that? I'm struggling hard to get this working. I probably need to: 1. Set-up monitor so it sends metrics to some database (elasticsearch, mongodb,...). 2. Iteratively compute statistics based on the content of the database. The second bullet I can do. The first proves to be very difficult to get working.
j
I think that doing that per request is going to be too slow and therefore I need the “custom analysis”.
That's true.
Set-up monitor so it sends metrics to some database
Do you have preferred analyze or drift detection platform/solution?
j
@Jiang Thanks for help. I read the article. I ended up using evidently for drift detection. My stack is currently; BentoML, Logstash + Elasticsearch to read the bentoml logs, metaflow (pipeline) + evidently to periodically compute metrics, store it and detect drift. However this wont work on aws lambda, since i wont be able to read the bentoml monitoring files. I tried to use jaeger to cooperate with bentoml but didnt succeed so far 🙁
j
@Jan Palasek Hi. Glad for these progress. We have solution for cases when you cannot access the file system
Copy code
monitoring:
  enabled: true
  type: default
  options:
    log_path: path/to/log/file
The config above will put logs to file system. But if you change it to this:
Copy code
monitoring:
  enabled: true
  type: otlp
  options:
    ...
The logs will be shipped with otlp protocol
https://docs.bentoml.org/en/latest/guides/monitoring.html#through-a-otlp-endpoint
Copy code
For some deployment platforms, it's not easy to collect log files. For example, AWS Lambda doesn't support log files. In this case, BentoML can export monitoring data to an OTLP endpoint. Some log collectors like fluentbit also supports OTLP input.
Docs here
j
Thanks. I've read that all. It's not really easy to get that working. Do you have some kind of POC that gets it working? Some docker-compose would be most welcome.
Or do you know about any?
j
Yes. We are currently using fluentbit by our own.
fluentbit acts as a pipe, supporting otlp input and tons of outputs
You can start a small ec2 in the same vpc of your lambda functions, running fluentbit, and fill the endpoint info to your bentoml config like this
Copy code
monitoring:
  enable: true
  type: otlp
  options:
    endpoint: <http://subnet-ip-of-your-ec2:5000>
    insecure: true
    credentials: null
    headers: null
    timeout: 10
    compression: null
    meta_sample_rate: 1.0
@Jan Palasek Feel free to @ me if you have further questions. I'd like to provide more details. I think our discussion could contribute to the official doc to make monitoring feature easier to on-board. 🍱
Open in Slack
PreviousNext
Powered by