In Drosophila, aversive olfactory memory is believed to be stored in a prominent brain structure, the mushroom body (MB), and two pairs of MB intrinsic neurons, the dorsal paired medial (DPM) and the anterior paired lateral (APL) neurons, are found to regulate the consolidation of middle-term memory (MTM). Here we report that another prominent brain structure, the ellipsoid body (EB), is also involved in the modulation of olfactory MTM. Activating EB R2/R4m neurons does not affect the learning index, but specifically eliminates anesthesia-sensitive memory (ASM), the labile component of olfactory MTM. We further demonstrate that approximately two-thirds of these EB neurons are GABAergic and are responsible for the suppression of ASM. Using GRASP (GFP reconstitution across synaptic partners), we reveal potential synaptic connections between the EB and MB in regions covering both the presynaptic and postsynaptic sites of EB neurons, suggesting the presence of bidirectional connections between these two important brain structures. These findings suggest the existence of direct connections between the MB and EB, and provide new insights into the neural circuit basis for olfactory labile memory in Drosophila.