BentoML

So this is happening in <https://github.com/bentoml/BentoML/blob/main/src/bentoml/_internal/bento/docker/entrypoint.sh#L38>

I've tried Bento 1.05 to Bento 1.07 and it's still the same problem

I've tested the service via `bentoml serve`  and it's working fine

You're deploying via yatai? And it looks like 2 of the runners failed and the rest are pending? But serving locally works fine?

Maybe try:
```kubectl -n yatai-deployment logs -f deploy/yatai-deployment```

```yatai                                 Active        42d
yatai-builders                        Active        42d
yatai-components                      Active        42d
yatai-operators                       Active        42d
yatai-system                          Active        42d```

```k logs -f deploy/yatai-yatai-deployment-operator -n yatai-components```
Maybe this?

Is there a way to test this command locally?
```    - ./env/docker/entrypoint.sh python -m bentoml._internal.server.cli.runner . --runner-name
      orientation_fixer --bind <tcp://0.0.0.0:3000> --working-dir .```

Ok cool I think I can reproduce this in a Pod and shelling into it

So seems like the above command is the one causing the `Illegal Instruction`

Where is `bentoml._internal.server.cli.runner` ?

Interesting ...

```bentoml start-runner-server  --runner-name orientation_fixer  --working-dir . --verbose
Illegal instruction```

I guess this is the bare minimum arguments for this to work

But:

```$ bentoml serve
Illegal instruction```

what type of runner is that runner? "Illegal instruction" sounds like some type of incompatibility at the driver/os level... but that's a complete guess

Might need to run this on an Ubuntu node again...

Have you tried running it locally with docker? It's not exactly the same thing, but it could help narrow down the issue.

You do: bentoml containerize bento_name

Then it will give you a docker command to run to start it

I have also deployed a previous  version of this in Knative with the same dependencies and that worked fine. :thinking_face:

Ah! Might be because of intel-tensorflow-avx512 on a node that doesn't have it. I'll try and report back

one way or another, we should find a way to bubble that error up. I wonder if we looked in the node's /var/log if we would see anything weird

I shall poke around more in a bit. And ya that would be cool :)