When Rosh Chodesh is on Sunday, the regular haftorah is usually replaced with "machar chodesh", the story of Dovid and Yehonoson and the latter's father's pursuit of Dovid. This is [only] usually the case because sometimes a different haftorah such as parshas ha'chodesh or of the three weeks etc., takes precedence.
There is however another question of halacha that arises every time erev Rosh Chodesh falls out on Shabbos: The prevalent custom in k'lal Yisroel (especially in winter months) is to begin seudas shlishis (preferably) before shki'ah and recite birkas ha'mazon only when Shabbos is more or less over. In a normal Shabbos, we tend not to question whether or not we should be reciting “retzai” (the additional paragraph for Shabbos). After all, this is the third of the Shabbos meals, and as such it is most appropriate, if not imperative, to say it. However, when Sunday is also Rosh Chodesh, we must ask ourselves if we should be adding “yaaleh v’yavo”. After all, in a matter of minutes, we will begin Maariv when we will be adding it. Reciting both might be somewhat of a contradiction. On the other hand, it is already Rosh Chodesh, but how could we leave out retzai; after all, this is one of the three obligatory Shabbos meals? Indeed, many people try to make an extra effort to eat an earlier seudas shlishis on a Shabbos such as this one. However this can be especially challenging in the winter months and even in the summer time, requires some forethought and advance planning. What, then, should one do when faced with this type of situation?
The Rosh (in a teshuva) writes that even on an ordinary Shabbos, if one did not bentch before nightfall, he should omit retzai from birkas ha’mazon. As proof, he cites the gemara (which is also the halacha) regarding a person who forgot to daven Mincha on Shabbos afternoon: When he makes up his missing tefilah at Maariv, he recites two weekday shemoneh-esrei’s. Although the second one is meant to serve in place of his Shabbos Mincha, since it is no longer Shabbos, he davens a weekday tefilah. So too, reasons the Rosh, since it is no longer Shabbos, how can we allow him to recite retzai?
The Hagaos Maimonios however takes issue with this ruling. After all, one who accepts Shabbos early recites a Shabbos shemoneh-esrei although it is still Friday afternoon. It seems that the same applies to birkas ha’mazon, even if he eats quickly and bentches before sunset. Clearly, since for all intents and purposes it is Shabbos for him, he recites retzai. The same should apply to a meal eaten at the end of Shabbos, so long as he has not yet called an end to Shabbos by davening Maariv or reciting havdala. As for the Rosh’s proof from the person who forgot to daven Mincha, on the contrary, the reason why he doesn’t recite a Shabbos tefilah is because he has ended Shabbos with his first shemoneh esrei. His second one, is a make-up tefilah, but it is definitely no longer Shabbos.
Both the Shulchan Aruch and Rema (in Darkei Moshe) write that the halacha follows the Hagaos Maimonios. However, they speak only of a regular Shabbos when retzai is the only addition that should possibly be added. Our question of what to do if motzei Shabbos is Rosh Chodesh still remains. The Tosefta (Berachos) however, rules that one who began eating on erev Shabbos (a weekday meal, without accepting Shabbos early) and bentched on Shabbos, must recite retzai in birkas ha’mazon “because he recites bentching on Shabbos”. As such, it would seem that in our scenario, one should recite ya’aleh v’yavo and not retzai. After all, if is certainly Rosh Chodesh now (especially if he recites bentching after tzei ha’kochavim). Indeed this is the ruling of the Magen Avraham who writes that reciting both is not an option for how can one contradict himself by saying it is Shabbos and at the same time Rosh Chodesh which isn’t until Sunday?. The Bach however maintains that the ruling of Shulchan Aruch (O.C. 188) is absolutely correct; everything depends on when he began eating. As such, retzai is most appropriate for this meal. This would seem to be especially true since, as mentioned above, this is a Shabbos seudah. (The reason why one who forgets retzai during seudas shlishis is not required to repeat bentching is not because it might already be motzei Shabbos, but rather because we rely – b’dieved – on the opinions that do not require one to eat bread for seudas shlishis, and the halacha remains the same even if one eats and bentches well before sunset. As such, l’chatchila, this is a Shabbos meal.) The Taz, takes a third approach and says that while it is true that the beginning of the meal is the main determining factor, one should also recite that which is appropriate at the time birkas ha’mazon is recited, since “the additional kedusha of the day has begun”. It is for this reason that the Tosefta requires the addition of retzai for a meal that began on Friday. Therefore, when Rosh Chodesh falls on Sunday, one should recite both retzai and ya’aleh v’yavo. There is no need to be concerned that this appears to be a contradiction just like havdala is recited as part of kidush when yom tov is on motzei Shabbos. The Mishna Berura cites both the opinion of the Magen Avraham and the Taz without making a clear ruling. It is important to remember that this is all provided a person ate bread both before and after nightfall; if he ate before shki’ah and merely continued (by eating other foods or drink) into the night the poskim rule that he should recite only retzai. Similarly, if didn’t wash until after shki’ah, reciting retzai (in this case) is questionable and many poskim recommend following the ruling of the Magen Avraham and not the Taz.