Personally, I think it should probably transfer to weapons as well (or at least weapons from races that value their privacy. I can't imagine a Brakiri weapon, for example, not being constructed to have the widest possible market appeal). That being said, I also think the book specifically states the skills that the -4 penalty applies to, and I don't believe that attack rolls are part of that. Or armor check penalty, for that matter. You'd think that Drazi armor, for instance, wouldn't be the most comfortable for humans.
No reason not to house rule it in, but for simplicity's sake it's probably easiest to limit it to certain skill checks.