Heh, aye

The problem arises when folk takes ages checking every possible move and every enemy ship move before shifting anything (and the same before firing). OK with a couple of ships per side, irritating with a full fleet and not really in the spirit of the game IMO

Placing spare dice to mark possible moves helps a bit, as does adopting a time limit for movement if players are really abusing the privilege.
As to the original question: the trouble with taking one or two larger ships over a swarm (if you can call 3-4 frigates a swarm

) is that you're in danger of getting critical hit'd to death (or at least impotence) if a bunch of smaller ships get lucky with their shield penetrations, whereas the larger ships may have to hold back a lot of their firepower for defence and may have to "waste" firepower overkilling smaller ships or do small amounts of damage to the group (depending on weapons fit).
Fleet choice matters a lot, I'd think. A couple of agile Klingon cruiser types that can dance around at Disruptor range while still throwing out a respectable amount of damage are a much more viable choice against a swarm than a couple of battlecruisers with limited arc Reload heavy weapons and not-so-great phaser suites (yes Plasma-boats, I'm looking at you

). While I'd imagine a bunch of Kzinti Frigates and Scouts could be real trouble.
A mix of both types is probably the best choice in a small game IMO, gives you a "heavy hitter" and some initiative sink/IDF cover options if required. Which is pretty much essential against some fleets, at least until a particular pair of empires appear with (hopefully/eventually) their signature fighters and gunboats in tow.