Sword Worlders are more of a cultural group. Their separation from the Solomani is only a couple thousand years along, if that, with no geneering or millennia-long environmental adaption.
Compare that to the Darrians, Suerrat, Geonee, Vilani, etc (300,000 years), or the Nexxies and Jonkeereen...