This message was deleted.

Some context: • I'm runner a single model (in a single runner) that is running a classification inference. • # of API workers is 2 I've tried to track down the following to help with this: • Looked at CPU usage. These errors originally appeared to come from that single runner when CPU usage peaked above ~230%. ◦ The

cpu

amount given to the runner was 1, and I also tried changing it to 0.5. In both cases, the runner was still able to use > 200% of CPU 🤔 . ◦ It seems when I allow for two runners of that same type, CPU percentage is able to hold above that ~230% for both runners and not throw these errors. • Looked at memory usage - there's plenty of free RAM left both on the machine as a whole but also in the individual runner process (it uses ~10% of RAM). • These errors don't occur when I turn off batching. ◦ I still get errors, but they are not errors with response code 0 and EOF (which seem somewhat non sensible).

cc @Sean @sauyon

@Sean @sauyon any thoughts?

Sorry, haven't had time to really dig into this one. Would you be able to share a reproducer?

@sauyon all good! Just wanted to make sure it wasn't lost among all the things. Sorry, but do you mean a full script to reproduce, or something else?

I think both the k6 script and as much of the service code as you're able to share would help! If you can't give us the full service code at least the model arch / setup would be helpful!

Right on! Here is the k6 script:

import http from 'k6/http';
import { sleep } from 'k6';
import { Counter } from 'k6/metrics';

export const CounterErrors = new Counter('Errors');

export const options = {
    // Key configurations for avg load test in this section
    thresholds: {
        'Errors': [
            {
                threshold: 'count<5',
                abortOnFail: true,
                delayAbortEval: "1s"
            }
        ]
    },
    stages: [
        { duration: '4800s', target: 480 }
        { duration: '60s', target: 480 }
        { duration: '60s', target: 0 }
    ],
};

export default () => {
    const urlRes = <http://http.post|http.post>('<http://127.0.0.1:8000/analyze>', JSON.stringify({
        "s3Path": [insert path to S3 URL],
        "modelsToRun": [insert list of strings that specify what models to run in BentoML]
    }), { headers: { 'Content-Type': 'application/json' }, timeout: '120s' });
    const contentOK = (
        urlRes.status === 200
    )
    if ( contentOK === false ) {
        console.log(urlRes.status, urlRes.body)
    }
    CounterErrors.add(!contentOK)
    sleep(0.001);
};

As for the service code, I'm looking at it and it looks like I'd have to cut ~90% of it. Are there questions about it I could answer that would help? Also, FWIW, I did file an issue with

k6

: https://github.com/grafana/k6/issues/3597. I haven't had a chance to track down what they had proposed - wanted to try to lump in any suggestions y'all had and do all the work at the same time. But want to at least pass that on here.

Oh, no problem! Sorry again about the delay 😅 What model architecture are you using? Re: cpu limits: we don't actually enforce CPU limits in our server; we just set well-known environment variables and assume that the model will respect that. Maybe you know something off the top of your head @larme @Jiang?

Oh, yes...sorry about that! @sauyon we're using ResNet18 here

Thanks Sean! Could you also clarify if it just includes the model inference code for resnet18 in the service api? Or are there multiple model/runners or complex pre/post processing code?

I think they said it's a single model runner!

@Chaoyu yep, it's a single runner!