I wonder if you might have some bronchial constriction. If I were you I would try go get my hands on a blue inhaler (Ventolin/salbutamol) and try using it before your rides and see if that improves things. If it has no affect, no harm done it's not a dangerous drug**, but if it helps, you can talk to your doctor about asthma management.
I have very low level of asthma, that is rarely triggered by anything other than an chest infection. When I was a child, this sort of asthma was called "chronic bronchitis" Ventolin is a big help to me at these times.
**I am not medically qualified in any respect. Talk to a doctor or pharmacist before following this course of action. You need instruction on how to use it, and confirmation that it is safe for you.