Linker histones bind to the nucleosome and regulate the structure and function of chromatin. We have previously shown that the globular domains of chicken H5 and Drosophila H1 linker histones bind to the nucleosome with on- or off-dyad modes, respectively. To explore the determinant for the distinct binding modes, we investigated the binding of a mutant globular domain of H5 to the nucleosome. This mutant, termed GH5_pMut, includes substitutions of five globular domain residues of H5 with the corresponding residues in the globular domain of Drosophila H1. The residues at these five positions play important roles in nucleosome binding by either H5 or Drosophila H1. NMR and spin-labeling experiments showed that GH5_pMut bound to the nucleosome off the dyad. We further found that the nucleosome array condensed by either the GH5_pMut or the globular domain of Drosophila H1 displayed a similar sedimentation coefficient, whereas the same nucleosome array condensed by the wild-type globular domain of H5 showed a much larger sedimentation coefficient. Moreover, NMR and spin-labeling results from the study of the nucleosome in complex with the full-length human linker histone H1.0, whose globular domain shares high sequence conservation with the corresponding globular domain of H5, are consistent with an on-dyad binding mode. Taken together, our results suggest that a small number of residues in the globular domain of a linker histone can control its binding location on the nucleosome and higher-order chromatin structure.