Referees discretion as the demands of each situation is different.
Broadly speaking, there are two types of fatigue: physical fatigue or mental fatigue. Perhaps, since some skills are more mentally challenging, use a INT check instead of END, as a test for fatigue.
Could also limit Timeframes...