That might very well be because the party you vote for has a good 35 members who are rabid on the issue and by being rabid make any sensible move on your relationship with the EU impossible. It doesn't matter very much who is leading your government; as long as the Conservative Party selects people disregarding their extremism on Europe that party can never deliver on Europe.