I think most panniers are right and left handed. I've never seen a pair which are both the same, or one of them would be snafu for the very reason you give.
Mine are both identical. The previous pair I had (which were exactly the same) weren't. May have been a manufacturers error.