Recommended settings for transformers?

[OhadRubin]

Are there any recommended settings for Transformer Language modeling?

[wassname]

I'd also like to know this please. Even if it's just some informal insights you might have picked up.

For anyone who's new, the author has said the MadGrad mode is better for transformers. And also states that they have done a bunch of experiments some of which are documented here.

@OhadRubin Until the author arrives we can share our own tips? Here's mine

I've noticed that transformers sometimes can't handle the default learning rate (1e-3 in the readme images), they will "break". So you sometimes need a lower max learning rate (perhaps 1e-4 or even 5e-5) if they "break" them. This happens when the loss goes up and wont come down.