The marking on the rim refers to the rim size (2.125 is the outside rim distance), not the tyre. That helps explain why you are having trouble finding 2.125 tyres: it's not a 'normal' size/width.I need a new back tyre for my Raleigh Mustang which should be a 26x2.125 but I can't find one for sale.
The tyre in the image looks to be about 26x2.5. Those rims are wide (wider than 'normal' MTB/26" rims) - I wouldn't mount anything less than 1.9 on it. 26x 1.9 up to what your frame will allow would be my suggestion, and choice of width and tread depends on its intended use (as articulated above).