Game mode being latency-optimized really is the saving grace in a market segment where the big brands try to keep hardware cost as cheap as possible. Sure, you _could_ have a game mode that does all of the fancy processing closer to real-time, but now you can't use a bargain-basement CPU.
On my Samsung OLED game mode has an annoying effect that turns (nearly) copletely black screens into gray smudge garbage that you can only turn down but not completely off, making that mode entirely useless.