The spin texture of the unoccupied bands of the surface alloy Bi=Ag(111) is investigated with spin- and angle-resolved inverse photoemission and first-principles calculations. Surprisingly, the measured spin character does not always reflect the calculated spin texture of the bands. With the help of photoemission calculations within the one-step model, however, the discrepancy is traced back to the influence of the orbital symmetry of the respective states in combination with the experimental geometry. In particular, the calculations show that the spin texture of a surface band with mixed orbital symmetries may neither be recovered with s- nor p- nor unpolarized light. In general, spin information from direct or inverse photoemission experiments on electronic states with mixed orbital symmetries at spin-orbit-influenced surfaces has to be taken with a pinch of salt, while it remains reliable for states with pure symmetry.