לזכות ר' משה גבריאל בן יהודית הוא וכל אשר לו לברכה והצלחה וכל טוב סלה
The Rosh says [as does Tosfos] that one need not rebuke his fellow if he is certain that he will not listen and is only obligated to rebuke if there is at least a chance that his friend will listen. He adds that this is only true if we are dealing with a law that is learned from a drasha such as adding on to the time of afflicting oneself on Yom Kippur. However, if the law is explicit in the pasuk then he must give rebuke until the person listens.
The Ritva in Makos [20a] writes that the rule מוטב יהיו שוגגין ואל יהיו מזידין - Better people be shogeg and not meizid [meaning don't rebuke them so that they remain unwitting sinners] only applies to איסורי עשה ["negative positive commandments" where it's phrased as a positive but requires one to refrain from a certain action] such as adding on to the time of Yom Kippur but not to לא תעשה's. A לא תעשה requires one to give rebuke no matter what and we don't prefer that people remain ignorant.
The achronim ask a strong question on these opinions from our gemara that implies clearly that if the tzadikim had been certain that the people would not have heeded their rebuke they would have been allowed to withhold it [they were only held accountable because they weren't certain, so they should have at least tried]. The problem is that we are talking about the sins of the first Beis Hamikdash which were the three cardinal sins of גילוי עריות שפיכות דמים and עבודה זרה which are both explicit and לא תעשה's which means that they were obligated to give rebuke even if they were sure that it would not have been heeded. So how do we reconcile the Rosh and Ritva with our gemara?
עיין אריכות דברים בס' מאור ישראל עמ' קט לגאון הרב עובדיה יוסף שליט"א