yea, i'm interested in how to resume training too. Both for tacotron and talknet

also, on the plus side, also, having the transcription.txt just point to the mounted google drive wav folder is so much better than uploading the wavs to the colab

they should the transcription.txt pointing to a google drive folder for wavs as the default on how the notebooks work. It'll save a ton of headache for larger datasets

You have to re-run all of the cells of the notebook.

So if you rerun it the next day, i'm assuming it'll look at recent checkpoints and start at the very latest checkpoint then?

In the case that you still have the checkpoints, yes.

ah, that means just save the latest checkpoint somewhere first and then readd them the next day. alright, that makes sense

The checkpoints are automatically saved in the "talknet" folder in your Google Drive.

i'm using the tacotron one though. Does this work with tacotron? or does this only work with talknet?

Yeah, the checkpoints are saved in your Tacotron 2 model, too.

cool. that's good to hear. I have an instance running just to test whether it resumes or not before I commit to trying to do 2000+ wavs

I couldn't believe what Tacotron model I made today! Somehow, some way, I made Jevil speak English with only 21 seconds (14 wavs) of data! It kinda sucks quality wise, but considering almost all of his data is because of the japanese and him saying "Oh." and "Ah ha!" I'm surprised it can speak more than one word at all!!! "The Fitness Gram Pacer Test is a multistage aerobic capacity test that progressively gets more difficult as it continues."

i don't think its worth putting on uberduck tho

but it public

so am I missing something here? Checkpoints are being saved to the checkpoint folder. I waited until epoch 240 to grab a copy of the checkpoint and saved it. Then I restarted the colab instance and uploaded the checkpoint back to see if it would restart from 240. Instead, it started at 0. Am I suppose to do something else to have it restart at 240 ? Screenshot clearly shows the 240 checkpoint, but it then started from scratch and created the 0 checkpoint. And config file exactly the same with no changes

Huh, that's odd. I dunno how to answer that, my apologies.

dang

@User how to warmstart in the new uberduck pipeline

actually i'm sure @User could help as well

i hope so. well, I'll just wait and see if anyone has any suggestions then

thanks for sending them a ping

i think the talknet version restarts from a checkpoint, but not for the tacotron one. I'll try the talknet version

normal tacotron restarts from checkpoints

uberduck pipeline is just for weirdos

does resuming talknet will always redo the spectrogram generator?

There's a "warm_start_name" parameter that can be passed to indicate the checkpoint to load from

So instead of "warm_start_name": "tacotron2_statedict.pt", its something like "warm_start_name": "tacotron2_180.pt"

where tacotron2_180.pt is a checkpoint from your previous run (here, run for 180 epochs)

oh nice, yea, that worked. Thanks Samkoelle

what happened here??