Vacuum leak across the diaphram is allowing air into the vacuum line and
then to the inlet manifold.

The leak may be the diaphram itself or the valving which vents the back
of the booster into the front of the booster when you take your foot off
the brakes.