Swapping to panniers was one of my best decisions, you won't regret it! It will change the handling of your bike, but I certainly don't miss the back, shoulder and neck pains!
Don't worry about weight distributions though... I only use one pannier for work so I'm very uneven, but it's never caused me any problems.