https://bentoml.com logo
Join Slack
Powered by
This message was deleted.
# ask-for-help
s
This message was deleted.
j
I haven't been able to figure that out yet, either. If I try to instantiate the service in my tests or reference one of the bento service API methods, it throws the error that the runners haven't been initialized. I ended up putting everything in functions and tested those individually outside of the API methods and it works ok.
n
Hey Jim, thanks for the response! That's a good alternative, actually. However, I think it would be good if they provide a way to test the "compiled" bentos alone. Because bentofile will also configure some parts of the service behaviour it should be better for integration tests.
s
U can dive in their code space, all cli based u can find corresponding function behind it, if I remember correctly there is a serve function, in that case no diff to normal pytest case
c
We are working on a recommended testing setup for BentoML users, as for now, you can use the runner.init_local method to initializes all runners within the current python process
For example, with the demo service from quickstart guide:
you can remove the 
await
part if this is not an async API
j
Oh that's very cool, thanks!
j
Is there a way to include the input and output descriptors (as described by the svc.api decorator) in the local testing? I’m able to test the pydantic models and the endpoint logic, but not sure how to test the bentoml io descriptors. E.g., my testing follows:
Copy code
for runner in svc.runners:
    runner.init_local()

test_json = {<dict with features>}
input_data = PydanticInputDescriptor(**test_json)
result = svc.apis[service_name].func(input_data)
output_data = PydanticOutputDescriptor(**result)
^ this all works, locally, but when I make requests to the API when serving, I get errors during Bentoml JSON’s 
to_http_response()
. I’d like to be able to catch this in my tests, but I’m not sure how. If I define 
JSON(pydantic_model=PydanticOutputDescriptor)
, how do I test the functionality that’s used in the svc.api decorator? Or am I thinking about this wrong?
c
Could you share your service definition? Is the output of your api function a pydantic model instance or a dict?
j
Copy code
input_descriptor = (
    JSON(pydantic_model=PydanticInputDescriptor))
output_descriptor = (
    JSON(pydantic_model=PydanticOutputDescriptor))

@svc.api(input=input_descriptor,
         output=output_descriptor)
def predict_proba(
        input_data: PydanticInputDescriptor) -> dict:
    # run pipeline to get results_df
    results = {"data": results_df.to_dict(orient="records")}
    return results
it’s a dict
I’m attempting to test the full validation with:
Copy code
for runner in svc.runners:
    runner.init_local()

test_json = {<dict with features>}
input_data = PydanticInputDescriptor(**test_json)
result = svc.apis[service_name].func(input_data)
output_data = PydanticOutputDescriptor(**result)
output_descriptor = (
    JSON(pydantic_model=PydanticOutputDescriptor))
blah = asyncio.run(output_descriptor.to_http_response(output_data))
assert blah.status_code == 200
but when I run the test, I pass with a status code of 200, yet when I send a request to my localhost, I get: 
ValueError: Out of range float values are not JSON compliant
status code 500
(using the same exact input)
@Chaoyu , I may have figured it out. Related to your question, “is the output of your api function a pydantic model instance or a dict”, my output is in fact in fact a dict, and I was developing under the thought that Bento (via the svc.api decorator) would pass the output of the api function would through the output validator (which would convert the dict into the pydantic model), but it seems as though that wasn’t happening and that the the output descriptor is expecting the data to already be the pydantic object? So when I add
Copy code
results = {"data": results_df.to_dict(orient="records")}
results2 = PydanticOutputDescriptor(**results)
return results2
my endpoint works. (the 
PydanticOutputDescriptor
handles the out-of-range floats, mentioned above). So, that is workable. But I still feel like I’m not understanding it completely. Is it true that the output descriptor expects the api output to already be the pydantic object?
👍 1
c
Yes that’s the expected behavior, with the current design, API function should return the Pydantic instance.. now you mentioned it, I’m wondering if we should support the other route as well, helping user to construct the Pydantic instance with API function’s return value 🤔
j
Hmm, I’m not sure. Because the decorator handles the input validation, intuitively I thought it would handle the output validation, too. But the current way is fine, too.
👍 1
Open in Slack
PreviousNext
Powered by