Microsoft has yet to release an iso that allows you to install Nano Server onto a server. The standard method seems to be to create a VHD and then use that to run Nano.
I was at a conference the other day, and managed to work through the process of getting a server up and running in VirtualBox 5.
On a Windows 10 Client or Windows 2016 TP4 server with GUI, mount the normal Windows 2016 ISO and copy the nano server folder to c:\nanoserver.
In an administrative powershell window, type the following commands:
This enables the cmdlets to create the vhd file we want.
The new command creates a new image, puts the oem Drivers (the same drivers that would up on a core install), the Hyper-V role, the IIS role, and the containers role (I want to play with that in another article).
new-NanoserverImage -MediaPath d:\ .\Base -TargetPath ‘c:\nanoVMs\nano.vhd’ -computerName myNano -oemDrivers -Compute -Storage -Defender -Packages microsoft-nanoserver-iis-package -containers
The script will ask for a password and start the creation process which looks something like this:
Now in c:\nanoVMs, there is a vhd file called nano.vhd
Copy this to your computer hosting Virtualbox. Create a Windows 10 VM, but instead of creating a hard disk, point the hard disk section at the nano.vhd file that was created/copied.
Now turn it on, jobs a goodun.