Bsrgvty

Say, that Nation A absolutely destroys Nation B in a battle and annexes them. Can Nation A realistically demand that the conquered people either convert to their religion or be expelled?

If we assume that the conquered people are extremely zealous and would rather be expelled than convert, wouldn't that render the conquest fruitless as the conquered region would become empty and need to be colonized?

Say, that Nation A absolutely destroys Nation B in a battle and annexes them. Can Nation A realistically demand that the conquered people either convert to their religion

Sure. Cuius regio, eius religio was specific to Western Europe in the aftermath of the Protestant Revolution, but the concept is very old.

or be expelled?

If we assume that the conquered people are extremely zealous and would rather be expelled than convert, wouldn't that render the conquest fruitless as the conquered region would become empty and need to be colonized?

This reminds me of the Roman conquest of Palestine.

Things went pretty smoothly because the upper caste of Jews (the Sanhedrin mentioned in the Bible) had been Helenized, and the Romans allowed Judaism to be practiced without molestation.

But when Caesar declared himself a god, and ordered that busts of his likeness be installed in all temples, you can imagine that the monotheistic Jews were a tad irked.

They revolted in 66 A.D., the Romans stomped them down and scattered them across the Empire. (This was the third Jewish Diaspora.)

They revolted again 66 years later, and this time, the Emperor kicked all the Jews out, forbidding them reentry.

Yes, of course that can happen. It has happened frequently in our history. How do you think Islam spread to much of Asia and North Africa, for example? Or Christianity to much of sub-Saharan Africa? However, your second scenario doesn’t happen — most people don’t care enough about their religion to face exile, and will just pay lip service to whatever religion their rulers demand.

Ethical considerations apart, why expelling them, when they can be simply eliminated? It happened in many cases in the past, for example during the Ottoman invasion of Otranto:

On 28 July 1480, an Ottoman fleet of 128 ships, including 28 galleys, arrived near the Neapolitan city of Otranto. The garrison and citizens of Otranto retreated to the Castle of Otranto. On 11 August, after a 15-day siege, Gedik Ahmed ordered the final assault. When the walls were breached the Turkish army methodically passed from house to house, sacking, looting and setting them on fire. A total of 12,000 were killed and 5,000 enslaved. [...] 800 men were given the option of Islam or death, and chose death.

If you want to go larger scale, look at how North and South America have been colonized by the European colonists.

Consider that survivors can actively seek revenge, while deads don't, and that an almost empty land is a good starting point for setting colonies. Moreover expelling large groups from a place takes a larger logistic effort than simply burying the corpses.

It depends on what A is willing to do to B to make it happen. If A is of comparable size and has total military dominance over B, A can simply give each member of B the choice to convert or die. Assuming A is willing to go through the trouble to enforce this, there is no reason for this to be unrealistic.

But this might be unrealistic if, say, A was a smaller nation that only won because of a few clever victories in battle, but doesn't actually have the manpower to completely occupy B from top to bottom. If this is the case, then A would realistically have to seek some sort of compromise; even expelling B from their lands is probably not feasible.

Even if B is not willing to convert and gets removed as a result, the conquest can still be useful for A. A gets new lands and resources, and also removes B as a potential threat.

So ultimately it's a question of motivation: Desire to spread religion A or Dilution of Nation B's culture?

As you can tell by the other answers, there are plenty of historical precedents for forceful conversions and expulsion from land. Ignoring the exertion of power over the populace via expulsion, extermination or enslavement - I don't believe there is a strong deterrent against forced conversion or prolonged enforcement of acceptance of Religion A is indeed hard.

Perhaps outlawing Religion B and/or seizing control over Religion B's institution/authority would be acceptable?

e.g. there are many different localised denominations of Christianity with distinct belief systems. There is also a historical precedent of amalgamation of Pagan rituals into Christianity. So perhaps there is room for an enforcement of conversion into pseudo-religion AB that is ultimately absorbed/administered by religion A.

This is more a comment to Mike Scott's answer, but I can't comment. "However, your second scenario doesn’t happen — most people don’t care enough about their religion to face exile, and will just pay lip service to whatever religion their rulers demand."

While not in a war situation, the early members of the Church of Jesus Christ of Latter-Day Saints (aka Mormons by many) did just that. They were told to denounce their beliefs or leave, so they left or were killed. This happened at least 3 major times in their history: Kirtland, OH; Independence MO; and Nauvoo IL. In this case the rulers were the mobs, local government, state government, and to a lesser extent the US government who all had a hand in many of the actions that followed.

The only minor discrepancies if you will, is that in this case you could argue the "conquesting people" wanted them to abandon a religion rather than join one.