I am all for allowing different commoners of different backgrounds different types of class skills. A slaver though could just be a nomad or soldier who goes around and captures people, then takes them to market, allowing a commoner to sell them for him. I don't know what "special class abilities" a slaver would need, other than an appropriate feat and skill selection to catch people (ranks in Craft (trapmaking) would be good for a potential slaver). Any PC could do it without taking a special class.
A fence is just someone with a lot of contacts (ie. reputation), ranks in Bluff, Gather Information, Search, etc. Any class would do. A scholar could sit at the centre of a vast fencing web... Any PC could do it without taking a special class.
Smugglers could be nomads, soldiers, borderers, scholars - they just have to be willing to hide things from the law. Again, it is just a matter of skill choices. Any PC could do it without having to take a special class.
It is all a matter of skill selection and appropriate role-playing.
An adventuring merchant?