because it is not the gloss that fades, it is the pigment below. Will turn milky/chalky underneath a layer of gloss. You are more than welcome to come look at some sculpture that I can illustrate this with.
When this happens, it is harder to match the paint for touch up. the worst colors for this are red, yellow and black... blue is bad as well, but mainly the lighter shades.