One management server keeps stopping the console proxy every 30s #14390

problem

We run 5 management servers. After a restart of one of them, that server keeps stopping the console proxy vm about every 30 seconds with:

INFO  [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[...]) Stop console proxy Console VM instance {"id":35,...}

Because we are currently in ResetSuspending management mode the other management servers start it again, so the console proxy keeps flapping between Up and Disconnected. we counted 200+ stops in about 2hours. The agent connection itself is fine, StartupProxyCommand and ReadyCommand get answered, then it gets stopped again.

The db is fine while this happens:

• consoleproxy.management.state = Auto
• consoleproxy.management.state.last = Auto
• both global only, no zone level values, and isdynamic: false in listConfigurations

The other 4 management servers dont log any ResetSuspending. restarting the affected management server once more fixes it, 0 stops after that.

We had this twice now, both times on the same management server right after it restarted. the first time was shortly after we uploaded console proxy certificates with uploadCustomCertificate. The second time we can't link it to an upload, so we dont know what had ResetSuspending in the db when that server read it.

Looks like the regression from #11415: these settings are read through ConfigKey now and are not dynamic, so every management server caches the value once and never reads it again. In a cluster one server can stay stuck in ResetSuspending while all the others are on Auto, and they keep fighting over the console proxy.

versions

• ACS 4.22.1.1, 5 management servers
• KVM on AlmaLinux 9.8, agent 4.22.1.0
• MariaDB Galera cluster

The steps to reproduce the bug

Not 100% reproducible for us, what we think happens:

1. Run more than one management server
2. Upload a console proxy certificate, this sets consoleproxy.management.state to ResetSuspending in the database
3. Restart one management server before the state is back on Auto, so it reads ResetSuspending on startup
4. The state goes back to Auto in the database, but the restarted server keeps ResetSuspending in memory and stops the console proxy every scan

What to do about it?

#13202 makes these two settings dynamic, which should fix this since the stuck server would read Auto again. It has been approved but has merge conflicts since september, would be good to get it into 4.22.2.

For now the workaround is: after restarting management servers, grep the logs for "currently in ResetSuspending" and restart the server that shows it.