Unsafe is literally made for that. For cases when your wants can only be checked by your brain saying "I did the logic and it's all good". Ten lines of unsafe does not mean your program will blow up. If anything, it's here to encourage you to just be much more careful here.
Right, you don't have to be a purist to get safety benefits from Rust. If a Rust program is 1% "unsafe" code, it still has a far lower vulnerable surface area than an equivalent C program.