so sorta like parentheses i guess

OK got it - so that's weird - seems like hi-fi gan would need a training that was phoneme based, to match. so that would be talknet?

hifi-gan is just an addon to tacotron

tacotron turns text into a spectrogram, which then has to be turned into audio

this can be done using regular algorithms (e.g. griffin-lim) but then it sounds bad hifi-gan on the other hand is a neural network optimized to turn spectrograms of speech into audio

uberduck uses a "universal" hifi-gan, which i think means it was trained on a bunch of voices? so it performs decently on about anything you give it but of course if you finetune it on specifically the one type of speech it does better

though i'm not sure if it's "hey here's spectrograms gotten from the audio please make the audio from them" or "hey here's tacotron's spectrogram output for

Apple.

please turn that into the training data we have" can't remember if the training notebook asks for the tacotron model as input

okay what the heck is finetuning, and how do you do it. i've hear finetuning mentioned a lot. is this mainly a tweak to the synthesis process or the training process too?

it's just training, but instead of starting from scratch you start from a network that was trained on a ton of data

in this case 24 hours of public-domain audiobooks spoken by one lady, which is enough for the model to learn how english works

yeah! i ran into that earlier. i wondered about the links to the audiobooks on one site where they were giving directions on tacotron. so let me get this straight. you train from scratch, probably test it in synthesis, and then you use this audiobook thing to finetune it?

no i got that wrong didn't i? so you apply your trained model to another pretrained model data?

the opposite actually

you take the model that was trained on audiobook data, probably took them a few weeks or something so it knows how to speak really well

and then you tell it "hey great job now can you do this voice instead", aka resume training but with different data

and since it already knows how to speak it only needs a fraction of the data and a couple hours of training

okay so that feels somewhat similar to the Voice Clone thing i tried, because it used pretrained data as a starting point. i would think that the pretrained data wouldn't have to be redone from scratch and could be reused, but i guess sometimes people need something specific?

yeah it's absolutely reused

i believe it starts from the same model every time

@[voice clone] if you mean

real time voice cloning

that's different, i believe it's actually trained to do speaker recognition (as in "who's speaking that") and the final output for the recognition is a thousand neurons each corresponding to a speaker, but on the layer before that there's like 100 neurons that together encode the characteristics of a voice so they ignore the final layer and take the 100 neurons, and then they use them to synthesize voice somehow with a special version of tacotron that is trained to work together with the network but the point is you can then take a clip of anyone's voice, get the characteristics of it, and then synthesize using those characteristics codeparade did pretty much that but with music and faces and fursonas go check it out

i might be very wrong though who knows

its that thing that Corentin J made and various folks have made notebooks of. it uses SV2TTS as i recall but the notebooks seem to have a simplified version. it was the easiest thing i could use, as it's already trained and you just generate texts to complete it but after a whil i could tell that i overtrained it.

yeah sv2tts is that thing

though i've only ever used it without finetuning, just "here's 5 seconds of audio go do the 100 neurons thing"

me too - well someday i'll run it off the Win10 laptop on the 1070 and see if it's much of an improvement since the local version can utilize more than one file. in terms of getting clips out of videos i have to say i was amazed at what spleeter was able to accomplish with pulling out cartoon voices from a music background.

yeah spleeter's pretty good in part because it was trained by deezer who have access to like a billion copyrighted songs and their stems

though i'd say vocal remover 5 is better maybe not next-level better but incrementally better

interesting - that can be run off of notebook as well? have to say i'm a bit new to this method of running code but i'm warming up to it. way easier that typing CLI yourself.

yeah there's a notebook for that the preset isn't perfect though, it's optimized for making instrumentals (hence the name) and the vocals are just a byproduct there's an alternate model that was trained on mix-vocal pairs instead of mix-instrumental, here's how use

though it's not like you have to use it in command-line on desktop either, there's no gui for v5 yet but there are batch files that you can drag your audio onto and have it process