There are numerous discussions in the gemara about the status of mitzva objects of an עיר הנידחת such as a lulav and mezuza. How odd that a city of idolaters would have such mitzva objects. The lesson is that despite the fact that these people are keeping all of the mitzvos, they still went off and served idols. When it comes to the correct hashkafos one must be very careful not to veer from the straight and narrow path.
[Heard from the Rebbe Shlita - Motzaei Shabbos Parshas Re-eh 5773]
לע"נ ר' אלכסנדר זושא בן ר' יוסף