I believe that the division of the fleet between the three castes had to do with the maintenance of the power balance. Remember that before Valen and without the Grey Council, the Minbari people were very internally adversarial.
Also, in terms of controlling the fleet, there is no indication as to what types of ships were controlled by which castes. And control doesn't necessarily equal being crewed by them, perhaps only commanded by them. Worker caste may simply have controlled the third of the fleet that circulated around logistics and support. Religious all of the research, exploration and diplomatic vessels (although obviously they also had some warships).
Fact remains, Minbari built the Whitestars, crewed them & commanded them and the WS fleet before handing them to the ISA. And the Rangers worked for the Grey Council until it was dissolved and Delenn took over exclusively (the better part of season 3). Back to the point of the Victory, shouldn't the ISDs be shifted? Victory herself was destroyed as was the shipyard that made the first 2. Time taken to rebuild the facility and new ships pushes their ISD to the right somewhat?