It isn't my assertion. The crime statistics in both the UK and the USA back this up. It is something like 96% of all prosecutions of sexual assault are men.
EDIT: In fact it is higher at least in the UK
> In 2011, males accounted for the vast majority of prosecutions for sexual offences
(98.2 per cent).