Well, in all but lawless and warfare condition, you can't prance around in full armour without a very good reason, so assuming the armour isn't all non-intrusive (augments, diplovests, etc.) put them in situations where you can't just go barging in, a Casino Royale, not a Monte Cassino.
If it's powered armour, there's also maintenance issues, infected onboard computers. And armour piercing ammunition. Or artillery. Or, well, power and air supply isn't infinite...
It doesn't need to be adversarial between you and the Traveller. It could also be that you put them in situations where only that powerful armour has a chance of saving the day... as long as they don't fall into the volcano, star, black hole, monofilament thrashing machine, etc.
Finally, if you fly into a solid object (wall, tree, starship, asteroid) at high velocity and come to a sudden stop, the armour might not crack, but whatever is inside still goes splat, though people might not notice until they open the suit and the occupant oozes out.