Ants, Acacia trees, and herbivores

It's well known that acacia trees produce food bodies and offer nesting sites for ants, which help it to defend against herbivores that feed on the tree's leaves. But what happens when the herbivores are removed? The trees paradoxically grow sicker and lose leaves. Apparently when herbivores are not browsing on the leaves, the trees reduce their investment in providing resources to the ants, and the ants start to leave the trees. This allows the establishment of wood-boring beetles which bore tunnels into the wood of the trees, causing them to weaken. Amazing! It's like playing Jenga.