To vote or not to vote: expat edition

There has been much heart-searching and hand-wringing over the Ontario Court of Appeal's recent ruling that it is indeed constitutional that Canadian citizens who live outside the country for more than five years forfeit their right to vote in Canadian elections, a status quo that actually goes back to 1993, despite a short hiatus after a Superior Court ruling last year.
The argument on one side is that Section Three of the Charter of Rights and Freedoms - which reads: “Every citizen of Canada has the right to vote in an election of members of the House of Commons” - guarantees votes for expats, regardless of how long they live away from Canada. But the court ruled that another clause in Section 1 of the Charter overrules this absolute right.
Many people seem absolutely incensed at what they see as an attack on the very roots of our democracy, and are bridling at the creation of "second-class citizens" in this way. Personally, I have swung back and forth on the issue.
Some argue that granting expats a vote is wrong because would allow them to shape Canadian policies that don’t directly affect them. Others argue that people do not necessarily vote for policies that directly and immediately affect them as individuals, but that they deserve a chance to affect the broader long-term direction of their country whether they live there or not. I can see both sides of that.
Neither is it as simple as to say that, because they pay taxes, they should be able to vote. The logical extension of that argument is that homeless citizens and even some retirees should lose their vote because they don't pay taxes.
Some have even suggested that special Members of Parliament be established to directly represent Canadians who live abroad, in much the same way as in Italy and France, which just seems kind of weird and pointless to me.
My gut feeling is that, by choosing to live abroad, expats are not affected by Canadian laws to the same extent, and so don't bear the same responsibilities of citizenship. As I see it, they have effectively withdrawn from the “social contract” of citizenship, and so forfeited their democratic rights. The ability to vote with, as the judge in the case put it, “no practical consequence for their own daily lives”, while affecting the lives of other people who do have to suffer the consequences, seems intuitively wrong to me. When they move back, they are welcome to vote again. In the meantime, I have no problems with them voting in their country of residence.
Another way of looking at this is that, if someone were to move from, say, Toronto to, say, Edmonton, they would not be eligible to vote in their old Toronto riding. Why, then, would someone moving from Toronto to Beijing expect to have that right? Looked at this way, five years is actually pretty generous.
What I reassure myself with in my prevarications, is that, when all's said and done on the theory and the ethics of the issue, it has almost zero practical importance: of the million or so expats who were actually entitled to vote in the last federal election, a grand total of 6,000 actually bothered.