I think that is true!
I used to go on a cycling training camp in Spain every March. I would do a total of (say) only 100 miles from December to February but then do 750 hilly/mountainous miles in 13 rides over the course of the holiday. I'd come back to the UK at the end of March and every single time I caught a bad cold within days and then spent a couple of weeks feeling awful.
I didn't get ill in March very often before I started taking those holidays, and I haven't very often since I stopped.
(One year I only lasted 4 days on the holiday before getting ill. I felt really rough for a couple of days but I recovered quickly by resting in the sun, eating lots of fruit and not touching the bike.)
PS I'm not suggesting that moderate exercise causes problems. I just completely overdid things on those holidays. I'd hammer myself the first week and get stronger and stronger but then the fatigue would set in during week 2 and that must have been when my immune system was struggling.
The organisers suggested taking easier rest and recovery days but I wanted to make the most of my time in the sun so I usually ignored the advice. The holidays were great, but there was a price to pay for a few weeks after I got home.