Most people have a default −3 skill in everything that they don't have specific training in (though there may some skills with an even worse default, such as Astrogation for a caveman).
People with Jack of All Trades-1 have a default of −2 in everything for which a default is possible; Jack of All Trades-2 grants a −1 default and Jack of All Trades-3 grants a −0 (or just plain 0) default. There is no Jack of All Trades-4.
So, I would rule that rudimentary training (or practice) elevates the standard −3 default to −2, but for people with Jack of All Trades-1 it's just a credit toward training up to −1. More extensive training further elevates the −2 of rudimentary training to −1, and thorough training elevates that −1 to skill-0 (with analogous effects for Jack of All Trades-2 and -3).
It's only after a lot of intensive training that one advances from skill-0 to skill-1, and getting to skill-2 or higher requires even more. Most player characters are too busy adventuring to have time for training above skill-0, but in troupe style play a player could designate that one of his or her characters is on sabbatical from adventuring to seek advanced training (skill-1 or higher).
I'm not sure how to annotate a skill between the −3 default and skill-0. "Skill-−2" and "Skill-−1" just don't read very well on a character sheet, even with the distinction between the hyphen ("-") and minus ("−") characters I've used here. Maybe "Skill-(2)" and "Skill-(1)"? Parenthesized numbers are accounting notation for negatives, and they're visually distinct. Following the accounting notation, the numbers could be red too, to emphasize the distinction between skills worse than 0 but better than the standard −3 default. Another option would be to place them in a "Skills in training" block on a character sheet, maybe with the associated hours or uses toward improvement tracked there too.
I'd be inclined to say that skills short of skill-0 fade with time unless training continues, just to avoid accounting overload.