I once spent Shabbos in a community in New York. The shul has hundreds of people who daven there. Shabbos afternoon the Rov gives a shiur. The Shabbos I was there, about 3 and a half people showed up.
I was troubled by the bizayon Talmid Chachom and Bizayon Ha-Torah. The Rov deserves more respect than hundreds of people who are too busy doing nothing so they can't come to a shiur. I don't believe this Rov even takes a salary as he has a very good job outside the shul.
I would guess that in many communities there is scant attendance at the shiurim of the Rov. To remove all doubts in people's minds - when someone gives a shiur and almost nobody shows up - it doesn't feel good...:-). [I have given countless shiurim where either barely anyone showed up, barely anyone was interested - or both. Hard to get people interested in a Torah shiur unless it is a daf yomi shiur and they are daf yomi aficionados. It is hard for me to give a daf yomi shiur because I have yet to experience such a superficial level of learning that equals the typical daf yomi shiur. That is without discounting the many benefits.]
To borrow a line from our zmiros - ולמזהיר ולנזהר שלומים תן כימי נהר. Maybe start going to a shiur if only to give the Rov a good feeling.
And you might learn something as well 
.
. 
