Well... aren't these attacks only a concern if you have adversarial code running on adjacent cores? So for a desktop user the threat model really doesn't change. Just do your best to keep malicious programs off of your machine. If someone can already run a speculative execution attack against you're local machine, there are much easier attacks to run.
How closely have you followed the layers of mitigations which the major browser engines have implemented? I think a more interesting form of this question is asking how often people are exposed to malicious code which isn't covered by the browser sandboxes.
Browser vendors have been pretty quick to add mitigations, disable attack surface, etc. Someone comes up with a new way to attack systems via JS, but vendors are all headed in a safer direction.
