The reason they are so harsh is the same reason they are puncture proof - instead of one big air chamber, there are lots of little ones (bubbles, usually).
This means the air can't move away from the point of any pothole impact, but just compresses. If your 25mm tyre is compressed to 12mm, the pressure between the rim and the road is doubled. If it compresses to 6mm, it is quadrupled. In contrast, depressing a normal pneumatic tyre has hardly any effect on the pressure.
What you end up with is a relatively soft tyre that suddenly becomes very hard when you do hit a pothole, with all the possibilities of wheel damage that implies.
There are also other drawbacks. In the interest on making fitting them possible, they aren't as tight a fit on the rim as an inflated pneumatic tyre. There have been cases of people braking in wet weather, and finding that although the wheel stops going round, the tyre doesn't.