https://bentoml.com logo
Join Slack
Powered by
hi, I am trying to run the following service: ```@...
# ask-for-help
p
hi, I am trying to run the following service:
Copy code
@bentoml.service(
    # workers="cpu_count",
    resources={
        "memory": "8Gi",
        "cpu": "4",
    }, 
    workers=4, 
    http={"port": 8181},
    tracing={
        "exporter_type": "jaeger",
        "sample_rate": 1.0,
        "jaeger": {
            "protocol": "thrift",
            "collector_endpoint": "<http://localhost:14268/api/traces?format=jaeger.thrift>",
            "thrift": {
                "agent_host_name": "localhost",
                "agent_port": 6831,
                "udp_split_oversized_batches": True,
            }
        }
    }
)
class AnchorModels:
    def __init__(self):
        self.cross_encoder = CrossEncoder(
            "cross-encoder/stsb-distilroberta-base", 
            num_labels=1, 
            device="cpu",
        )

    @bentoml.api()
    async def rank(self, query: str, documents: list[str], ctx: bentoml.Context):        
        scores = self.cross_encoder.rank(query, documents)
        # turn score to float instead of numpy float that causes serialization issue
        scores = [
            {"corpus_id": score["corpus_id"], "score": float(score["score"])}
            for score in scores
        ]
        return scores
but it doesn’t seem to parallelize the requests. I set the number of workers to 4 but it processes at most 2 requests. here is a trace with 
jaeger
.
j
Hello, may I know what machine you are currently running at and how many cores do you have?
Copy code
resources={
        "memory": "8Gi",
        "cpu": "4",
    },
Does not set 4 workers nor uses 4 core of the CPU, this is a BentoCloud related feature and does not do anything within BentoML. Also, if you managed to run 4 workers locally, do you see 4 process spun in the access logs?
p
macbook 32 GB m2 10 cores.
Copy code
> lsof -i :8181
COMMAND     PID         USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
python3.1 82313 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82475 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82476 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82477 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82478 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82479 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82480 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82481 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82482 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82483 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82484 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82486 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82488 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82490 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82497 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82498 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82499 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82500 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82506 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82547 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 82548 pavelschudel    3u  IPv4 0x8286e06ddbdf4a25      0t0  TCP *:8181 (LISTEN)
python3.1 97014 pavelschudel  103u  IPv4 0x8286e06ddcb514b5      0t0  TCP localhost:65306->localhost:8181 (CLOSED)
python3.1 97014 pavelschudel  104u  IPv4 0x8286e06ddde114b5      0t0  TCP localhost:65304->localhost:8181 (CLOSED)
python3.1 97014 pavelschudel  106u  IPv4 0x8286e06dde0c94b5      0t0  TCP localhost:65307->localhost:8181 (CLOSED)
python3.1 97014 pavelschudel  107u  IPv4 0x8286e06ddef55f45      0t0  TCP localhost:65308->localhost:8181 (CLOSED)
python3.1 97014 pavelschudel  108u  IPv4 0x8286e06ddef67465      0t0  TCP localhost:65309->localhost:8181 (CLOSED)
this is what is see listening on port 8181 where my service is running. I don’t see logs of number of workers started. I remember in the old ranners API there was logging for number of workers.
j
Copy code
import bentoml

@bentoml.service(
    workers=4
)
class svc:
    @bentoml.api
    def predict(self, inputs: str) -> str:
        # print current python process id

        import os
        print(f"Current process id: {os.getpid()}")

        import time
        print("Sleeping for 5 seconds")
        time.sleep(5)
        return f"{inputs}"
if you run the code above and run 
ps aux | grep python
you should see there is a main process and 4 subprocesses running. and that should handle up to 4 concurrent request
p
I have changed the code a little to fit my server:
Copy code
@bentoml.service(
    workers=4,
    http={"port": 8181},
)
class AnchorModels:
    @bentoml.api
    def rank(self, query: str, documents: list[str]) -> str:
        # print current python process id

        import os
        print(f"Current process id: {os.getpid()}")

        import time
        print("Sleeping for 5 seconds")
        time.sleep(5)
        return f"{query}"
image.png
there is no parallelism
Copy code
Current process id: 16870
Sleeping for 5 seconds
2024-07-29T11:36:11+0300 [INFO] [entry_service:AnchorModels:2] 127.0.0.1:56576 (scheme=http,method=POST,path=/rank,type=application/json,length=3076) (status=200,type=text/plain; charset=utf-8,length=42) 5018.664ms (trace=5104f3ea590d1bb70bb48871d6750789,span=789b36608e2791b0,sampled=0,service.name=AnchorModels)
Current process id: 16870
Sleeping for 5 seconds
2024-07-29T11:36:16+0300 [INFO] [entry_service:AnchorModels:2] 127.0.0.1:56580 (scheme=http,method=POST,path=/rank,type=application/json,length=3076) (status=200,type=text/plain; charset=utf-8,length=42) 10028.010ms (trace=9163fdac36c903c27a0eada96112bb98,span=ebf5042d9024ef4d,sampled=0,service.name=AnchorModels)
Current process id: 16870
Sleeping for 5 seconds
2024-07-29T11:36:21+0300 [INFO] [entry_service:AnchorModels:2] 127.0.0.1:56581 (scheme=http,method=POST,path=/rank,type=application/json,length=3076) (status=200,type=text/plain; charset=utf-8,length=42) 15048.424ms (trace=963fb6e2b24b4ce33f9b0c2284a86df5,span=7fd6dcdf3919dc32,sampled=0,service.name=AnchorModels)
Current process id: 16870
Sleeping for 5 seconds
they are all proceessed by the same process
j
one small detail for the load balancer, it is randomly distributed instead of round robin.
how are you starting your server? do you run 
bentoml serve
?
p
bentoml serve models.service:AnchorModels
j
image.png
p
strange
my be because I am using bentoML client to make the calls?
Copy code
class AsyncCrossEncoder:
    async def rank(self, querie, documents):
        async with bentoml.AsyncHTTPClient(env.MODELS_URL, timeout=300) as client:
            print("sent request")
            return await client.rank(query=querie, documents=documents)  # type: ignore
j
what is the output of 
ps aux | grep python
p
Copy code
pavelschudel     32189   0.3  0.8 37485340 256108   ??  Ss   11:42AM   0:02.56 /Users/pavelschudel/opt/anaconda3/envs/anchor/bin/python -m _bentoml_impl.worker.service models.service:AnchorModels --fd 3 --service-name AnchorModels --backlog 2048 --worker-id 2 --prometheus-dir /Users/pavelschudel/bentoml/prometheus_multiproc_dir --ssl-version 17 --ssl-ciphers TLSv1
pavelschudel     32188   0.3  0.8 37501724 255800   ??  Ss   11:42AM   0:02.59 /Users/pavelschudel/opt/anaconda3/envs/anchor/bin/python -m _bentoml_impl.worker.service models.service:AnchorModels --fd 3 --service-name AnchorModels --backlog 2048 --worker-id 1 --prometheus-dir /Users/pavelschudel/bentoml/prometheus_multiproc_dir --ssl-version 17 --ssl-ciphers TLSv1
pavelschudel     32191   0.3  0.8 37501724 257560   ??  Ss   11:42AM   0:02.57 /Users/pavelschudel/opt/anaconda3/envs/anchor/bin/python -m _bentoml_impl.worker.service models.service:AnchorModels --fd 3 --service-name AnchorModels --backlog 2048 --worker-id 4 --prometheus-dir /Users/pavelschudel/bentoml/prometheus_multiproc_dir --ssl-version 17 --ssl-ciphers TLSv1
pavelschudel     32190   0.3  0.8 37500740 262232   ??  Ss   11:42AM   0:02.56 /Users/pavelschudel/opt/anaconda3/envs/anchor/bin/python -m _bentoml_impl.worker.service models.service:AnchorModels --fd 3 --service-name AnchorModels --backlog 2048 --worker-id 3 --prometheus-dir /Users/pavelschudel/bentoml/prometheus_multiproc_dir --ssl-version 17 --ssl-ciphers TLSv1
looks like there are workers
j
Could you try using a normal async client like httpx or aiohttp, or even firing multiple curl at the same time and see if it has parallelism
p
I ran the following:
Copy code
import asyncio
import httpx

async def send_request(client, url, data):
    try:
        response = await <http://client.post|client.post>(url, json=data, timeout=100.0)  # Set a timeout of 10 seconds
        return response.text
    except httpx.ReadTimeout:
        return {"error": "Request timed out"}

async def send_multiple_requests(url, queries, documents, num_requests):
    async with httpx.AsyncClient() as client:
        tasks = []
        for i in range(num_requests):
            data = {
                "query": queries[i % len(queries)],
                "documents": documents[i % len(documents)]
            }
            tasks.append(send_request(client, url, data))
        responses = await asyncio.gather(*tasks)
        return responses
    
url = "<http://localhost:8181/rank>"
queries = ["example query 1", "example query 2"]
documents = [["doc1", "doc2", "doc3"], ["doc4", "doc5", "doc6"]]



await send_multiple_requests(url, queries, documents, 2)
got results in 5 seconds.
Copy code
Current process id: 52413
Sleeping for 5 seconds
Current process id: 52412
Sleeping for 5 seconds
2 distinct processes for 3 requests
Copy code
Current process id: 52412
Sleeping for 5 seconds
Current process id: 52413
Sleeping for 5 seconds
Current process id: 52413
Sleeping for 5 seconds
still only 2 processes and took 10 seconds
4 requests:
Copy code
Current process id: 61456
Sleeping for 5 seconds
Current process id: 61454
Sleeping for 5 seconds
Current process id: 61455
Sleeping for 5 seconds
Current process id: 61455
Sleeping for 5 seconds
(restarted the server)
j
Yeah this is expected as the default load balancing is random instead of round robit.
p
is there a way to change it?
for 20 requests got
Copy code
Counter({'76679': 15, '76677': 3, '76680': 2})
not very random I must say.
p
@Jian Shen Yap is there a reason why this is randomly distributed instead of a round-robin? I agree with @Pavel Schudel that being able to choose would be very helpful.
Open in Slack
PreviousNext
Powered by