Get server image from http://www.ubuntu.com/getubuntu/downloadmirrors#mirrors
Create new VM. Edit it’s settings:
General > Advanced: enable PAE/NX
Name: the interface of your host that is connected to the internet
Install as usual. Select only “OpenSSH Server”.
Configure router for static dhcp, i.e. <mac-of-VM> –> <some-ip>.
ssh to <some-ip>.
If you really need X, install these packages (tested with hardy):
xorg gnome-core gdm human-theme legacyhuman-theme ubuntu-artwork