Asymmetric division of stem cells results in both self-renewal and differentiation of daughters. Understanding the molecules and mechanisms that govern differentiation of specific cell types from adult tissue stem cells is a major challenge in developmental biology and regenerative medicine. Drosophila follicle stem cells (FSCs) represent an excellent model system to study adult stem cell behavior; however, the earliest stages of follicle cell differentiation remain largely mysterious. Here we identify Castor (Cas) as a nuclear protein that is expressed in FSCs and early follicle cell precursors and then is restricted to differentiated polar and stalk cells once egg chambers form. Cas is required for FSC maintenance and polar and stalk cell fate specification. Eyes absent (Eya) is excluded from polar and stalk cells and represses their fate by inhibiting Cas expression. Hedgehog signaling is essential to repress Eya to allow Cas expression in polar and stalk cells. Finally, we show that the complementary patterns of Cas and Eya reveal the gradual differentiation of polar and stalk precursor cells at the earliest stages of their development. Our studies provide a marker for cell fates in this model and insight into the molecular and cellular mechanisms by which FSC progeny diverge into distinct fates.