Let me help clarify things. Let's take your example of swimming in plate armour with a -9 AP penalty; here's what the rules say.
Swimming: Divide the base Movement by two and subtract the Armour Penalty. If the result is zero, the Adventurer cannot move and barely keeps himself afloat. If the Movement is negative, then the Adventurer sinks. For example, the Adventurer wearing full plate armour, with a –9 penalty, has a Swimming Movement of 4–9 = –5: the Adventurer will sink if he attempts to swim in his armour.
So yes, AP inhibits your movement depending on the type of activity you're undertaking. Swimming in full plate is impossible for a standard human because the armour's weight and bulk will pull you under. Your armour type will affect how far you can move under certain activities and may make some kinds of movement activity impossible.
Why did we get rid of the old method of applying a penalty to skills? Well, it wasn't popular for one thing, even though its not necessarily unrealistic, and many people were house-ruling the way the skill penalty worked. It also made it messy for reflecting skills in stat blocks.
However, someone who's trained in combat is likely to have trained wearing armour. Pete, who's the expert on these matters because he fights in heavy armour on a regular basis, looked carefully at how to reflect the fact that trained combatants can, actually, be quite dexterous in armour, but not necessarily as fast, because the bulk slows you down and fatigue builds. A trained fighter, though, won't have his attack, parry and evasion skills necessarily compromised through his armour type.
So we re-engineered the armour penalty to work on speed of initiative and movement to better reflect circumstances. We tried several mechanisms, including an Armour skill that reduced the skill penalty armour imposed. In the end we found the way we have in the rules the most efficient and best reflecting what really happens. I don't think its necessarily
completely true to life, but is a reasonable compromise in terms of game system.
Does this help?