We have studied the magnetic states of self-organized Fe islands on W(001) by means of spin-polarized scanning tunneling spectroscopy (Sp-STS). Single-domain and vortex states in the nanoscale islands have been observed. The high-resolution magnetic images enable to experimentally determine the boundary between the single-domain and vortex states. The single-domain state was always found below a thickness of 6 nm and a diameter of 120 nm in Fe islands. The boundary directly observed with Sp-STS is consistent with theoretical predictions.