Bsrgvty

The Two-Weapon Fighting rule says:

When you take the Attack action and attack with a light melee weapon
that you're holding in one hand, you can use a bonus action to attack
with a different light melee weapon that you're holding in the other
hand. You don't add your ability modifier to the damage of the bonus
attack, unless that modifier is negative.

If either weapon has the thrown property, you can throw the weapon,
instead of making a melee attack with it.

With Two-Weapon Fighting, I can only add my ability modifier to the damage of the first attack. Even if I get the Dual Wielder feat (PHB, p. 165), it doesn't say anything about adding the ability score to the damage.

The Crossbow Expert feat (PHB, p. 165) says:

Thanks to extensive practice with the crossbow, you gain the following
benefits:

• You ignore the loading quality of crossbows with which you are proficient.


• Being within 5 feet of a hostile creature doesn’t impose disadvantage
  on your ranged attack rolls.


• When you use the Attack action and attack with a one-handed weapon,
  you can use a bonus action to attack with a loaded hand crossbow you
  are holding.

When I attack using the bonus action granted by Crossbow Expert, do I add my ability modifier to the damage? I don't know whether I should follow the Two-Weapon Fighting rules, or roll damage as for a normal weapon attack.

You can add your ability modifier to damage from the bonus action attack

The general rule for damage rolls is:

When attacking with a weapon, you add your ability modifier--the same modifier used for the attack roll--to the damage.

In short: You can add your ability modifier to your crossbow attack because nothing tells you that you cannot.

Unofficial guidance from Jeremy Crawford (5e's lead rules designer), which some may consider authoritative, agrees with this interpretation:

When making a weapon attack, you add your ability modifier to the damage, unless a feature tells you not to.

So far, so good.

But, does this rule for two-weapon fighting, which also grants a bonus action, confuse matters?

You don't add your ability modifier to the damage of the bonus attack, unless that modifier is negative.

Dont worry - this rule can be safely ignored. The bonus action granted by the Crossbow Expert feat is not the same as the bonus action granted by two-weapon fighting. Thus the two-weapon fighting rules are completely irrelevant to your situation.

Yes, because it's not Two-Weapon Fighting.

As you correctly point out, the rules for using a hand crossbow with Crossbow Expert don't mention losing out on your stat bonus. Since using your off-hand crossbow in this way doesn't actually use the Two-Weapon Fighting rules, you don't need to refer to them at all when using Crossbow Expert.

Since Crossbow Expert effectively gives you an entirely new way of spending your bonus action to attack, you get your full ability modifier to damage.

The other answers are correct but they fail to explicitly state the factor that makes them correct: The rules for 2-weapon style do not apply in this case because they only apply to melee weapons. The only way that the rules for dual weapon could apply to your hand crossbow would be if you swung the crossbow itself at somebody as an improvised club.

So long as you are using the crossbow as a ranged weapon to fire crossbow bolts at enemies, the 2 weapon style rule does not apply and you only need to worry about the rules of the feat that allows you to fire it as a bonus action.