I can't say worry or don't worry but I do think it be easy to overreact.
When I lived in France much was Tuffeau stone which is very porous erodes easily so soon end-up with large holes from softer bits. Other bits flint/clay very thick and with a river running through the house so a lot of drying out, saturation through the year meaning a lot of stuff moves around.
When I purchased my house on Oxfordshire they survey report came back with "it doesn't have foundations" which the surveyor informally comment that it never had foundations and has been standing for hundreds of years.
Depending on construction, etc. if I stripped off old wallpaper and saw that I'd not be concerned. eg it was "old" walpaper yet showed no signs of having torn (or stretched?) so maybe been stable for years.