The correct establishment and maintenance of unidirectional Notch signaling are critical for the homeostasis of various stem cell lineages. However, the molecular mechanisms that prevent cell-autonomous ectopic Notch signaling activation and deleterious cell fate decisions remain unclear. Here we show that the retromer complex directly and specifically regulates Notch receptor retrograde trafficking in Drosophila neuroblast lineages to ensure the unidirectional Notch signaling from neural progenitors to neuroblasts. Notch polyubiquitination mediated by E3 ubiquitin ligase Itch/Su(dx) is inherently inefficient within neural progenitors, relying on retromer-mediated trafficking to avoid aberrant endosomal accumulation of Notch and cell-autonomous signaling activation. Upon retromer dysfunction, hypo-ubiquitinated Notch accumulates in Rab7+ enlarged endosomes, where it is ectopically processed and activated in a ligand-dependent manner, causing progenitor-originated tumorigenesis. Our results therefore unveil a safeguard mechanism whereby retromer retrieves potentially harmful Notch receptors in a timely manner to prevent aberrant Notch activation-induced neural progenitor dedifferentiation and brain tumor formation.