Trump "shithole" outburst offensive, stupid, but not intrinsically racist

Loath as I am to make yet another entry about Donald Trump, I have been trying to figure out why his latest outburst about "shithole countries" is being considered racist. Yes, the man is a racist in general terms - not to mention thoughtless, bigoted, supercilious and gauche - but I'm still not convinced that this particular remark is actually racist in and of itself.
During an all-party meeting on DACA and other immigration issues on Thursday, Trump is reported as having said, "Why are we having all these people from shithole countries come here?" The context of the comment is not entirely clear: it could refer to El Salvador, Haiti and African countries in general, all of which had recently been discussed, or it could just refer to the African countries, which was the last item to have been mentioned. Either way it clearly refers to poor, downtrodden countries populated by non-white people. This was made even more clear by his subsequent suggestion that the US should instead be bringing in more people from countries like Norway, which just happens to be as close to Aryan purity as possible in this day and age. Trump himself, predictably enough, is denying using such language, although Democratic Senator Dick Durbin, who was at the meeting, has confirmed it (and even maintained that Trump used the word "repeatedly").
So, minor contextual quibbling aside, Trump does seem to have called some poor black countries "shitholes". Is this actually a racist thing to say? The Guardian and Vox, among others, certainly seem to think so. But I find their arguments a little thin. Xenophobic, yes. Offensive, sure. Stupid, oh yeah. Ill-advised, definitely (although some are arguing that Trump says these kinds of things deliberately from time to time to shore up his core support - who knows?) But intrinsically racist, probably not, insofar as it does not directly claim that one racial group is superior to another.
Semantics, shmemantics? Perhaps. But if we slide into sloppiness, then arguably we are no better than Sloppy Steve Bannon.