Another characteristic of sound, in addition to its spectrum, is the presence of formants, or peaks in the spectral envelope.
The formants describe certain regions in the spectrum where there are strong resonances (where the amplitude of the spectral components is considerably higher).
As an example, pronounce aloud the vowels ``a'', ``e'', ``i'', ``o'', ``u'' while keeping the same pitch for each. Since the pitch is the same, we know the integer relationship of the spectral components is the same.
The formants are what allows us to hear a difference between the vowel sounds.