Histone chaperones are involved in a variety of chromatin transactions. By a proteomics survey, we identified the interaction networks of histone chaperones ASF1, CAF1, HIRA, and NAP1. Here, we analyzed the cooperation of H3/H4 chaperone ASF1 and H2A/H2B chaperone NAP1 with two closely related silencing complexes: LAF and RLAF. NAP1 binds RPD3 and LID-associated factors (RLAF) comprising histone deacetylase RPD3, histone H3K4 demethylase LID/KDM5, SIN3A, PF1, EMSY, and MRG15. ASF1 binds LAF, a similar complex lacking RPD3. ASF1 and NAP1 link, respectively, LAF and RLAF to the DNA-binding Su(H)/Hairless complex, which targets the E(spl) NOTCH-regulated genes. ASF1 facilitates gene-selective removal of the H3K4me3 mark by LAF but has no effect on H3 deacetylation. NAP1 directs high nucleosome density near E(spl) control elements and mediates both H3 deacetylation and H3K4me3 demethylation by RLAF. We conclude that histone chaperones ASF1 and NAP1 differentially modulate local chromatin structure during gene-selective silencing.