Why is a Speaker expected to be impartial anyway?

It's not that long since House of Commons Speaker Greg Fergus narrowly avoided official censure for a partisan video he put out for a long-time Liberal friend. But now he is in hot water again, although this time his apparent partisanship was not even his fault.

The Conservatives, jumping on any opportunity to stomp on the Liberals while they are well and truly down, have presented a motion to remove Fergus as Speaker, claiming that he lacks impartiality. This is after an event he was to preside over was advertised using a partisan criticism of Conservative leader, Pierre Poilievre. 

It turns out that Fergus knew nothing about the posting, which was actually generated by Liberal Party head office using "the auto-populated standard language we use for events on our website". The Liberal Party has already apologized for this mistake, but no doubt the Tories will try and make something out of nothing anyway, and are forcing a vote on it. Hyper-partisanship is the new normal, after all.

Two things occur to me with regard to this issue. 

One, why does the Liberal Party use "auto-populated standard language" in its advertising? Is there no-one that can come up with individualized content? And does their "auto-populate standard language" have to include ad hominem attacks against the opposition, rather than something a bit more positive and edifying?

But two, and perhaps even more importantly, why is the Speaker of the House of Commons expected to be impartial anyway? He is, after all, an MP, elected for his Liberal beliefs and policies. Is he then expected to completely subsume and deny all those beliefs and inclinations once assigned the position of Speaker? He still has to somehow carry out the (necessarily partisan) duties of an elected MP in other respects.

I understand that the same would be expected of a Conservative or NDP representative in the same situation. My point is: wouldn't it be better if the Speaker of the House of Commons were a non-elected official - say, a judge or a civil servant - whose sole duty is to arbitrate the increasingly fractious legislature, with no partisan baggage to get in the way?

Otherwise, it's a bit like asking a hockey or soccer player to referee a game in which their own team is competing, and expecting them to do it impartially.