I feel exactly the same way - about classical music. There are bits I like - the kind of 'bits that everyone likes' that you tend to find on '100 Classical Greats', CD's - £4.99 from Sainsburys, sort of thing. But take me far outside that 'easy' selection (which as you say I'm sure a true afficionado would regard with pity if not contempt) and I find myself sitting there thinking: 'I know I'm supposed to be enjoying this...but I'm not, really.' Brahms 4th symphony t'other day was the latest. I listened to one side, but just couldn't be bothered flipping the record.
As you say, I'm not knocking it...I just don't get it.
Now jazz on the other hand....a lot of people seem to like dissing it, but I for one just cannot understand how anyone could listen to this and not enjoy it:
View: https://www.youtube.com/watch?v=u2xx3YTu2ac