Average speed can be lost or gained quite quickly early on in a ride but the more miles you're into the ride, the slower the loss/gain will be.
Not only that, but you need more fast riding - at least in distance - to make up for the slow to bring up the average. Consider a hill 2 miles long, one mile up, and one down. Further suppose that you can ride at 10mph up it and 30mph down it. You will take 6 minutes to go up and 2 to go down. Total 8 minutes. That's an average not of 20mph (half way between the uphill and downhill speeds) but rather 15mph (4 minutes per mile). The reason for this is of course that you spent three times as long at 10mph as you did at 30, even though you covered the same distance at both speeds. Indeed, since 20mph is 3 minutes per mile and therefore 6 for two miles, to make a 20mph average after going up at 10mph you'd need to travel infinitely fast downhill. It's quite hard to bring the average up because the fast sections, by definition, don't last very long
It's very annoying. You can lose .2, .3 mph off the average crawling through a slow junction bit by bit and need to bust a gut for the next three miles to get it back!