Why is cellphone tracking to identify virus contact a major privacy concern?

I am still trying to understand what civil liberties campaigners and "privacy advocates" are so worried about when considering cellphone tracking apps as a means of identifying and alerting when people have been in close proximity to someone who later turns out to have tested positive for COVID-19.
Such contact tracking has already proven very effective in managing the coronavirus outbreak in countries like South Korea, Singapore and Taiwan, and some western countries are (very belatedly) considering their use. But there has been a lot of push-back from the civil liberties people, and companies like Apple and Google are having to tread very carefully, despite having a working model of an app that could prove very useful in the fight against COVID-19.
At its simplest, such an app would merely log down when Bluetooth connections identify that two otherwise unconnected people came within two metres of each other. This is done using anonymous identifying keys, not names, and the data is held on the phones themselves and not by governments or companies. Nothing happens to this data at all unless one of those people is later shown to have the virus. In that case, anyone who has been close to that infected individual over the previous couple of weeks is alerted, so that they can self-isolate and/or get tested. The data can then be deleted.
I fail to see what is so nefarious about this. Granted, the app and Bluetooth need to be running all the time and it may use a bit more battery (hardly a deal breaker, I would suggest), and its efficacy does depend on the availability of accurate, timely and widespread COVID testing. But neither of  these objections constitute gross invasions of people's privacy and personal security, are they? So, what is the privacy issue? There are much more flagrant and invasive privcy risks in use right now, although there are usually ways that the paranoid can avoid them (e.g. by turning off location services, turning off cookies, etc).
I often think that "privacy advocates" have a knee-jerk reaction to anything that might possibly, under certain obscure circumstances, be putatively used to track someone's whereabouts, or be used to advertise to them products or services that they may or may not be interested in. In most commercial cases, I see that as just the price of a more-or-less free internet. Where privacy is "invaded" in the interests of the greater good - for example, CCTV cameras that may help to catch thieves and rapists, or, as in this case, a cellphone app that might help manage a killer virus - I for one am willing to put up with that minor infringement. It seems to me that if you have nothing to hide, why would it be a concern.