Installing VMware Tools on Windows Server 2008 Server Core has to be done slightly differently than a full installation of the OS.
Because Server Core is command line based and does not have a GUI, the VMware Tools installer has to be run via the command line with msiexec.
1. First right click the virtual machine and choose "Install VMware Tools".
2. Then at the command prompt (assuming the CD drive is D:) run:
msiexec /i d:\VMware Tools.msi /qn
3. The computer will reboot once the installation is complete.