my mother (had 4), grannie (had 3) and an ex-aunt (had 5 last time I saw her +15 years ago) all said the same thing, adding a 3rd on makes not that much difference, adding a 4th makes even less difference (and in the case of my aunt a 5th, devote catholic who still thinks contraceptives are against his will before anyone asks and for all I know if has probably already got more...) My husband is 1 of 4 as well which probably goes a long way to explain why I hate family get togethers...
I understand the organisation required after 2 does not increase anywhere near as much - however getting them all into 1 vehicle does limit your car options somewhat (whilst I have no kids, I did bring my brother & sister up (17 & 14 years between us) so have done the have work, heart ache & joy without actually having had the kids...) it's rather like dog sitting you can have the dog for a day/week or two and then send it home again when you are tired of it!