It looks to me as though you are adding 35km to lose 900 metres. I work in miles so that's 20+ miles to lose 3000 feet.
IF the variation is all in one section, that is you lose it all over 35km, it will make some difference. If the difference is spread over the entire route I'd guess your ride times will be the same.
You mentioned this is a 3/4 day trip. So that's 50/60 km day plus a lot of sightseeing. I can't see altitude having a big impact on your day, possibly an extra hour riding at the very most.
Japans scenery seems to be either horizontal or at 45°: the coastal regions are generally flat as a pancake, unless the mountains reach the coast. The inner areas are extremely hilly jungle.
The difference in altitude is therefore usually concentrated in a very short stretch of extremely steep road, so we have a choice of going around and having an extra easy day, or two to three extremely gruelling hours climbing. (To give you an idea,
look at the contortions made by this highway in the mountains- and that tunnels through the worst: We'd have to climb on the old road to the pass)
@Andy in Germany you also mention the need to acclimatize. I don't know your local terrain but here in Lancashire we have a wide variety from flat as a pancake to very challenging.
Your highest point looks to be +/- 380 metres? Today I'll be at 350 metres on an ordinary cafe run of 100km. Are you sure acclimatisation is a factor?
Ah, poor communication on my part. I mean to Japan's climate. We'd aim to go at a cooler time of the year but it is sub-tropical, warm and extremely humid. We'd have to get used to that.
We would also be eight hours Jet lagged. I find a bike tour is the perfect cure for Jet lag: it gets my body in tune with the local time within 24h, but that may be a minority view.
I also want Elder Son to remain enthusiastic about riding, and come on day rides with me while in the country: if his first impression of Japan is murderous hills then he may lost his enthusiasm fairly quickly.
There's another factor too: away from major 'a' type roads, Japan doesn't bother with direction signs, and one village looks like another.
I think the length of the day in time is more important and of course that factors in the slower pace of climbing. Flexibility is key too. Planning a shorter day or a shortcut or indeed, using public transport if needed. The key to any tour is flexibility and interest, it shouldn't be a chore. Of course you don't necessarily need to mention those contingency plans but keep them up your sleeve for when you need them. Apart from that, hard to say, you know your own family best.
This is another problem: Japan is huge, and most of that is barely populated mountains: The urban bits we see in the media with a bazillion people crossing massive roads represent about 30% of the country. The mountains are extensive and remote: there's no plan 'B' and public transport, at least public transport that carries bikes, is non-existent. By taking the flatter routes we at least will be travelling through villages where we can get something to drink and rest under a roof if need be.
Overall, an easier if much less interesting idea is probably to fly to Nagoya airport instead, catch the ferry to Tsu, then ride the 40 very flat km to Ise:
Not as much of an adventure though, nor does it allow as much conflict free Introvert time.
If we did this we could then go on shorter tours, maybe with overnight stops, which would enable us to explore into the hills a bit...