In this work we compute, at the 'one-loop-dressed' level, the nonperturbative contribution of the ghost loops to the self-energy of the gluon propagator, in the Landau gauge. This is accomplished within the PT-BFM formalism, which guarantees the gauge-invariance of the emerging answer. In particular, the contribution of the ghost-loops is automatically transverse, by virtue of the QED-like Ward identities satisfied in this framework. Using as nonperturbative input the available lattice data for the ghost dressing function, we show that the ghost contributions have a rather sizable effect on the overall shape of the gluon propagator, both for d=3,4. Then, by exploiting a recently introduced dynamical equation for the effective gluon mass, whose solutions depend crucially on the characteristics of the gluon propagator at intermediate energies, we show that if the ghost loops are removed from the gluon propagator then the gluon mass vanishes. These findings strongly suggest that, at least at the level of the Schwinger-Dyson equations, the effects of gluons and ghosts are inextricably connected, and must be combined suitably in order to reproduce the results obtained in the recent lattice simulations.