Sometimes it's not about what you're doing, but what you can be seen to be doing.

Objections about camera phones have subsided because people understand them better. Unless someone is pointing the phone at you, which is usually an obvious action and not something that is widely done covertly, you aren't being photographed. But there are still concerns: I'm about to go vote in an election where there will almost certainly be "no camera" signs up at the polling station and specific advice against taking selfies that might inadvertently contravene laws about disclosing who voted for whom.

You have no such guarantees with a camera that by design is always looking out with the wearer and doesn't advertise whether it's recording or not.