Maybe you can add a BAB penalty, and maybe a move penalty, whit the previous presented:
At middle age, –1 to Str, Dex, and Con; +1 to Int, Wis, and Cha. -1BAB
At old age, –2 to Str, Dex, and Con; +1 to Int, Wis, and Cha. -2BAB, -5ft movement.
At venerable age, –3 to Str, Dex, and Con; +1 to...