We combine spin-polarized scanning tunneling microscopy (SP-STM) and first-principles calculations to demonstrate the control of the wavelength of helical spin textures in Fe nanoislands by varying their atomic structure. We make use of the complexity of submonolayer growth of Fe on Cu(111) to prepare nanoislands characterized by different thickness and in-plane atomic structure. SP-STM results reveal that the magnetic states of different nanoislands are spin helices. The wavelength of the spin helices varies strongly. Calculations performed for Fe films with different thickness and in-plane atomic structure explain the strong variation of the wavelength by a subtle balance in the competition between ferromagnetic and antiferromagnetic exchange interactions. We identify the crucial role of the effectively enhanced weak antiferromagnetic exchange interactions between distant atoms.