Many activities popular among men involve danger or risk. I would postulate that after the death of utilitarian and military horse-riding, the connotations of it changed. It became more of a leisure activity, and was largely supplanted by the faster, more utilitarian, more military car/truck/bike culture (at least in America). To your point about mid-20th century, I would suspect WWII had something to do with it.

Another poster also mentioned that it is less than comfortable for men. This is true. I would imagine it to be a serious motivation to switch to cars.