I changed all my bikes from 172.,5 to 165 last year. TBH I always thought 172.5 was a bit long, but it came with my bike, then I had a 4iii left crank power meter in that size, so stuck with it for a long time...
Anyway I'm a massive convert now. Main benefit is I get less DOMS after hard rides, which if I'm going away for a cycling trip, is really useful indeed. It definitely benefitted my fitness as I didn't need to take so many easy days.
WRT to the less leverage and struggles on higher %... yes you can definitely feel the reduction in leverage but modern gearing has so many options now you should be able to work around (especially 12 speed).