> After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com
Everyone is talking about backups, but why not about this? How is it even possible to delete the production database by accident? Why does he have SSH access there? Why do they test their database replication in production? Why are they fire-fighting by changing random database settings _in production_?
I know that all of this is common practice. I am questioning it in general.
Because when something goes wrong he has to be able to discover what it was.
> Why do they test their database replication in production?
Many places do that. With free databases there is simply no reason to, but the practices are inherited from the non-free databases best practices arsenal.
People, if you are using postgres and have a cluster in production, make sure you also have the script that creates this cluster in a VCS and is able to create an equal cluster on virtual machines in your computer.
> Why are they fire-fighting by changing random database settings _in production_?
Oh, that's because they are fire-fighting. The problem is that when you get a problem you know nothing about, you don't really know what to replicate in another environment to reliably test things there. The most you can do is verify that your changes aren't harmful, the real test is always in production.
Everyone is talking about backups, but why not about this? How is it even possible to delete the production database by accident? Why does he have SSH access there? Why do they test their database replication in production? Why are they fire-fighting by changing random database settings _in production_?
I know that all of this is common practice. I am questioning it in general.