I would say that the time to orbit is at least partially dependent on a maximum atmospheric speed. It's best to limit speed so that the hull doesn't get too hot; even if it can handle the heat, many cycles of hot and cold lead to fatigue that's an issue for a vehicle meant to fly as long as a B-52.
Even if we assume that the hull and even external features are sufficiently heat-resistant that they don't have much of a speed limit, there's the matter of atmospheric ionization that blinds most external sensors, including radio. If your spacecraft is only capable of entering atmosphere at close to orbital velocity, like the Space Shuttle or Apollo capsules, that radio blackout is a necessary difficulty. But a spacecraft with maneuver drives doesn't need to do that, and probably doesn't want to.
In the lower atmosphere, sonic booms are very annoying to people on the ground, and that argues for a speed limit around 85% of the local speed of sound.
Yet another consideration is whether the spacecraft has enough maneuver to directly levitate out of a planet's gravity (maneuver drive G rating higher than the planet's gravity) or requires some help from aerodynamic lift.