Correct, but hacking into the hypervisor is harder than hacking an administration interface. In the end the only secure server is one in a vault at the bottom of the ocean, but there are ways to prevent certain attack vectors.
I don't think you have that sidechannel with AES-NI. Besides, as a physical attacker cold boot attack would be much easier
Or if the server has any interfaces with DMA, like PCI or something, that's even easier.