It is narrated from Sayyiduna Abu Juhayfah (may Allah be pleased with him) that the Messenger of Allah (peace and blessings be upon him) was sitting one day, and in front of him (peace and blessings be upon him) some people were engaged in disliked talk and clamour. It was said: O Messenger of Allah, why do you not forbid them? He (peace and blessings be upon him) said: "Even if I were to forbid them from going to (the mountain of) al-Hajun, one of them would surely be about to go there without having any need for it."
Hadith Referenceالمعجم الكبير للطبراني / باب الواو / 18539