It's pretty straight-forward. The left can't stand Rogan because he allows the non-left (not just the alt-right; the non-left) to have a platform. They don't actually care what the material is, what the issue is, they can't tolerate anything that isn't their own bubble, their own world views. The left controls ~95% of all media and it still isn't enough. Any platform that gives a voice to anything other than their own views, is fundamentally bad and is not to be tolerated. That's why most of the censorship push is coming from the left today, they're the party of intolerance and it's not just about the alt-right or far-right. Comedians broadly are terrified of the left's crazy censorship push, they're speaking out about it on a constant basis; comedians are almost always the canaries in the speech coal mine.

An obvious example of this in action, is Jordan Peterson, who the left almost universally detests. He's neither alt-right nor far-right, and yet the left tries to slander and control him with that label constantly (bury him under fake labels). If you're not in total agreement with them, you're the enemy; that's widely the ethos of the intolerant left today. Rogan has given people like Peterson a large platform to speak through. That's the sort of thing the left hates Rogan for.