https://bentoml.com logo
Join Slack
Powered by
This message was deleted.
# ask-for-help
s
This message was deleted.
l
plus: And I and a new guy in the aiml.
also swagger:
c
Could you share your model save code again?
πŸ‘ 1
l
Sure! Thanks for you response!
Copy code
import bentoml
from transformers import AutoTokenizer, SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan

processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts")
model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts")
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")

bentoml.transformers.save_model("speecht5_tts_processor", processor)
bentoml.transformers.save_model("speecht5_tts_model", model, signatures={"generate_speech": {"batchable": False}})
bentoml.transformers.save_model("speecht5_tts_vocoder", vocoder)

%%writefile service.py
import bentoml
import torch

from <http://bentoml.io|bentoml.io> import Text, NumpyNdarray
from datasets import load_dataset

proccessor_runner = bentoml.transformers.get("speecht5_tts_processor").to_runner()
model_runner = bentoml.transformers.get("speecht5_tts_model").to_runner()
vocoder_runner = bentoml.transformers.get("speecht5_tts_vocoder").to_runner()
embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")
speaker_embeddings = torch.tensor(embeddings_dataset[7306]["xvector"]).unsqueeze(0)

svc = bentoml.Service("text2speech", runners=[proccessor_runner, model_runner, vocoder_runner])

@svc.api(input=Text(), output=NumpyNdarray())
def generate_speech(inp: str):
    inputs = proccessor_runner.run(text=inp, return_tensors="pt")
    speech = model_runner.generate_speech.run(input_ids=inputs["input_ids"], speaker_embeddings=speaker_embeddings, vocoder=vocoder_runner.run)
    return speech.numpy()
And also I run these pip install script
Copy code
!pip install -r <https://raw.githubusercontent.com/bentoml/BentoML/main/examples/quickstart/requirements.txt>
!pip install git+<https://github.com/huggingface/transformers.git>
!pip install sentencepiece
!pip install torchaudio
!pip install datasets
!pip install cchardet
!pip install tensorflow
and also souce code is:
Copy code
!pip install git+<https://github.com/huggingface/transformers.git>
!pip install sentencepiece
!pip install soundfile

from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech

# load the pre-train model
processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts")
model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts")
inputs = processor(text="Hello word.", return_tensors="pt")

# load the embeding data
from datasets import load_dataset
embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")

# embeding data voice data
import torch
speaker_embeddings = torch.tensor(embeddings_dataset[7306]["xvector"]).unsqueeze(0)
spectrogram = model.generate_speech(inputs["input_ids"], speaker_embeddings)


# load the voice
from transformers import SpeechT5HifiGan
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")

# run the predict
speech = model.generate_speech(inputs["input_ids"], speaker_embeddings, vocoder=vocoder)

import soundfile as sf
sf.write("./tts_example.wav", speech.numpy(), samplerate=16000)
c
cc @larme (shenyang) @Frost Ming could you help take a look?
πŸ‘€ 2
l
em....any guys can help me ?
l
Hi Lucas, I'm trying to reproduce this issue in the jupyter notebook. Meanwhile I saved your codes in 
service.py
, then serve the model using 
bentoml serve service:svc
and it seems to work fine. Maybe you can try this routine first?
ok I also reproduce this bug when using 
bentoml serve service:svc
, I will dig deeper for this issue
l
Thanks @larme (shenyang) But I try to re-run this code. I just get the same error. i can share my notebooks. And I create a conda env for it.
bentoml.ipynbenvironment.yml
And again thank you for your quickly response.
And one suggestion maybe in example we can share the conda env file will be better for everyone can quickly start from example no pain.
l
Ok the cause of the problems is here: 
speech = model_runner.generate_speech.run(input_ids=inputs["input_ids"], speaker_embeddings=speaker_embeddings, vocoder=vocoder_runner.run)
where you are passing 
vocoder_runner.run
to another runner call. 
vocoder_runner.run
is not a simple function and is not picklable. (notes: every runner call's arguments are serialized and then send to runner server to be executed here, by default pickle is used to serialize the argument)
if I just use 
speech = model_runner.generate_speech.run(input_ids=inputs["input_ids"], speaker_embeddings=speaker_embeddings)
then the code runs
Do we have any work around to not passing 
vocoder_runner.run
?
never mind I may post an invalid solution
l
em....sorry for my side I just follow the the bentoml validation the example. https://docs.bentoml.org/en/latest/frameworks/transformers.html my target just want to try bentoml running the transform model. you can check this notebooks.
speecht5.ipynb
this example code from the bentoml website.
l
Ye that is bug in our side. I'm trying to fix this
πŸ‘ 1
l
Thanks.And I think if on your side these code is work. Maybe you can share your conda env file. If you do mind.
l
I think we also met this bug. It's not about the environment.
πŸ‘ 1
l
Got it Thx bro. And have a good day
Hi. guys any update on this issue?
l
The issue is not fix yet. But if you want to try the example, you may change 
speech = model.generate_speech(inputs["input_ids"], speaker_embeddings, vocoder=vocoder)
to 
speech = model.generate_speech(inputs["input_ids"], speaker_embeddings)
and the codes should work.
l
Cool! Thanks for your workaround. I will try to run it. Thanks a lot. @larme (shenyang)
I has been try to use mlserver But from my side. I think bentoml is more powerful and readable.
l
Thanks! Your feedback also help us to improve the product!
πŸ‘ 1
l
HAHAHHA . I will keep going to build my product using the bentoml. and I want to with the bentoml become better.
πŸ™Œ 1
Open in Slack
PreviousNext
Powered by