Has anyone played a fifth columnist, a mole working within a group of otherwise loyal players but taking orders from the enemy they are fighting against?
Agreed - this can be difficult to do right.
You never want one of your players to be actively trying to sabotage the other's main goal* as this just leads to ill-feeling.
The way we've managed to do this and strike a happy medium in some games is as suggested in FF's
Black Crusade - essentially a common primary objective but some optional personal objectives which are different for each player and may be mutually contradictory.
e.g. In
Pirates of Drinax you're trying to set up an imperial trade convoy by feeding doctored jump data so the merchants end up in a different location to the warships, leading to much looting and yelling of "yaar!". This is the primary objective, common to everyone, and makes sure that betrayals and conspiracy don't get too far out of hand or have someone trying to make them fail deliberately.
However, secondary objectives might be different - one player may have done a deal on the quiet to ensure ship X in the convoy isn't attacked (and must ensure that the proportion of the convoy the others get time to loot doesn't include it) whilst a 'rival' might want to sell the location to some less successful pirates to let them scavange the scraps from the attack, others may take the opportunity to frame someone in the escort fleet for doing the data doctoring, and may have different personal desires as to who to replace them with if they can.
The one time it can be interesting is - if the players are friends - to do a "hostage situation" type scenario where one player is actively trying to sabotage the other players efforts because he has no choice. The thing held hostage should be something equally important to all parties - ideally, the ship and/or the other players. Something akin to a 'there's a bomb on your ship, it's controlled by an intellect programme on your ship's computer and it's wired into all the internal sensors - if you don't do what I say or tell anyone then 'boom' and goodnight vienna' type threat. Make sure there is a good way for the other players to figure out what's going on, though!
*
Obviously this doesn't apply to paranoia, where all the players are plotting against everyone else, and their employer, all the time. Friend Computer is watching you.