It’s good to somewhat limit the amount of memory that gets paged to disk, as it is incredibly slow memory. There should be backpressure on memory allocation. Getting around the OS on this matter would be silly, IMO. We’re talking about an interactive UI environment where massive page files would be doubly bad, swallowing up surprising amounts of disk space in order to provide incredibly slow memory.

However, the type of solution you’re talking about would apply well to a caching server, and Varnish (on FreeBSD I believe) uses it quite successfully.