Another attempt to request details regarding base model training.

[codename0og]

Hey boss ~ I see you're active and yeah, back to work.

I realize that maybe some information is deliberately kept a secret or perhaps, you don't quite want your work reproduced
( I can totally understand that. ), so I don't quite ask for end-to-end guide on how to train the said base models,
but instead I'd really appreciate if you could explain few things or at the very least provide basic and crucial info we all lack.

Because really.. Like I said in some other issue where I too requested more information,
it's been almost 2 years now and we still have no reliable way to train better base models.

I think I should first name all the datasets we've tried ( including mixtures in various proportions. ):

• VCTK ( Our main baseline / reference point since you'd often mention that this is what you used. (( Did you actually? )) )
• M4 Singer
• Expresso
• GT Singer
• Genshin ( in any case, game / gacha rips. ~ That's what I use atm. )
• EARS
• Utau/Vocaloid/SynthV-based
• Some private korean ( 1000h I believe? or at least 200-500h, can't remember.. )

Now.. There's so many unclear things and questions, and to be honest I really wish you could answer these:

1. Is your project more grounded in VITS? I know the deal with doing phone -> contentvec features and re-adjustments to some layers but, is it really closer to VITS? or is it actually closer to early so-vits-svc iterations? ( Because depending on what it is, we could have had a bad reference point. )

2. Your base-models can sing reliably well ( ofc, it is still fine-tune set dependent, but I speak of the baseline capabilities )
   which is really suspicious from research point of view.. VCTK does absolutely not provide any sensible pitch-range so,
   Is there any chance you performed some of these approaches?:

• A) Dataset wasn't just VCTK, instead it was VCTK + some private / copyrighted singing dataset?
  and the explanation of why og models are aligned with vctk speakers-wise is because you pruned embedding table?
• B) You first trained a vocoder and discriminators ( Standard GAN framework, pure adversarial ~ no vae.) using some singing datasets first, then you'd use the resulting G/D for rvc, meaning:
  conv_pre from: [ mel_bins -> 512 ] to: [ 192 -> 512 ] change, adding spk cond, load_strict False..
  so seemingly, you'd be doing a representation-shift / domain-shift along with vae? ( posterior, flow, text ),
  I guess in a way it's a warm-start ish where decoder has to re-align with posterior inputs and learn mapping?
• C) You were doing some kind of 2-phase learning? First training on singing-set only and then fine-tuning on VCTK?
  ( but then.. this approach wouldn't provide any good results.. the model is forgetting how to sing reliably.
• D) You indeed only used VCTK but were performing pitch augmentation? Something similar to what SingingVocoders repo does?

3. How is the kl meant to behave? This part is a lil bit unclear to me.
   I generally notice that on smaller sets ( say, a prototyping-focused dataset which is 30 hours )
   the kl seems to start collapsing relatively early.. meaning within 15 to 35k steps ( batch 16, 1 epoch being roughly 2.5k steps )
   but then, I am not sure if I can even call it collapsing, easiest explanation I can give is via this image:

I tried to mitigate by implementing free-bits but, I doubt it's really the solution.
There must be something I don't quite understand ( bare in mind, I am not a machine learning researcher, just a hobbyist. )
Maybe it is related to dataset size? for example, we need just enough of data to introduce just enough variety
so vae-parts won't go " ahead " of gan / or simply collapse?

4. Was there 2-D-updates-1-G-update rule utilized? I noticed the discriminator doesn't seem to learn the real/fake boundary good enough. Or perhaps was D and G employed different learning rate? Was there differential lr per module / layer / component?

5. How come there's such a discrepancy in the epoch counts ( iteration metadata in base models you provide )?

G/D 48khz:  iteration; 392
G/D 40khz:  iteration; 77
G/D 32khz:  iteration; 636

Are these genuine or spoofed? Like I don't understand. Why would you train seemingly easiest 32khz for 636 epochs
but much more difficult 48k for almost half of it? and then.. 40k being so absurdly low?
Could it be you first trained the 32k as foundation and then re-used the same dataset + 32k model and froze everything but decoder+posterior ? so representation re-alignment training?

6. Assuming 636 epochs is a real number.. and given that you mentioned using DDP
   ( b16 = b4 x 4 ( 4 x v100 ) it'd be roughly ~397,500 steps ( 1.6m steps if 1 gpu ).
   Is that correct in terms of training duration? I don't think you ever provided exact steps count, instead it was training-days?

7. is DDP crucial for generalization? I am aware that 4x4 Is analogous to bs16 but not exactly the same.

8. What about schedulers? was exponential lr decay used for base models as well? with gamma 0.999875?
   or was it something cyclical / faster in decay?ssssss

9. What was the learning rate used exactly? I know RVC comes with 1e-4 but, the codebase is meant solely for fine-tuning.
   2e-4 causes issues, differential ( 2e-4 for gan, 1e-4 for vae ) doesn't go well either.
   ( But then an important factor, for prototyping I run stuff in fp16 ~ rtx 3060. )

10. Lastly, did you train in mixed-precision? or was it FP32 and the models got converted to half only before distributing them?

Phew.. yeah I know, that's a lot of things but hopefully you could answer these ( or at the very least, most important ones. )
I'll be honest with you boss, we don't want RVC to be forgotten or put aside.
yeah it's been 2 years, a lot of compute and time went into experiments and trials, but I am afraid without your input or feedback,
we are really stuck and won't go that far which is sad..

There's so many new ideas and potential of upgrades for RVC.. for example:
For example, take a look at these:

• Eta-WavLM
• R-Spin
  Worthy competitors for contentvec
  ( Some people already tried contentvec training/finetuning but, something is wrong with the codebase.. stuff just goes wrong. )
  But unfortunately, even something so trivial as embedder swap requires us to know things we simply do not know or can't reliably deduce.

Oh yeah and just so we're clear, I am not affiliated with AI Hub nor I support them by any means.
So if certain information bits are too delicate to be discussed publicly or something?
I do provide both my discord and email for contact:
dsc.: .codename0.
email: tmpcmz@gmail.com

Let's keep RVC alive man, thank you in advance🙏

[codename0og]

@RVC-Boss bumping to ensure you see it