לע"נ ר' מרדכי גימפל בן ר' אברהם משה
"You are passing by the border of your brother Esav". [Devarim 2/4]
The Ramban explains that Esav is also our brother because he descends from Avraham unlike Yishmael who is no longer the seed of Avraham from the pasuk כי ביצחק יקרא לך זרע - only Yitzchak is the seed off Avraham and not Yishmael.
This is very strange because the gemara in Nedarim excludes Esav from the descendants of Avraham from the word ביצחק implying that only part of Yitzchak is Avraham's seed but not all of it which excludes Esav. The Chasam Sofer and many others were bothered by this Ramban.
Maran HaRav Hutner [קונטרס ברכת אבות מאמר ד] suggests that the Ramban is making a distinction between the exclusion of Esav and the exclusion of Yishmael. Yishmael was EXPLICITLY excluded by the pasuk - Yitzchak and not Yishmael. Esav was never mentioned by the pasuk. All it says is that part of Yitzchak's seed will be excluded and Esav with his free choice placed himself in that category. But it didn't have to be him. It could have been somone else as well. The gemara calls Esav a Yisrael Mumar - an apostate. We never find Yishmael being described that way. This is because a Mumar is one who does a מעשה מומרות as only Esav did. Yishmael's exclusion was not because of any specific מעשה but because of a גזירת הכתוב.
That is why the pasuk calls Esav a brother.
The question intrigued me so I looked around and here is what I saw....:-)
זאב יטרף דברים סי' ד', חבצלת השרון דברים עמ' לח, ותירוץ מאד נחמד ומחודש בס' חקרי לב לג"ר א"ל היימן עמ' 188, ס' לב ציון להרה"ג רא"ל לפיינסקי שליט"א עמ' 269, להורות נתן דברים עמ' י"ח, משנת חיים על דברים סימן ה', שיעורי חומש לג"ר שמחה מיימון עמ' שנ"ה, תכלת מרדכי לג"ר מרדכי גימפל