Skip to main content

Posts

Showing posts from June, 2024

How to handle big issue on live server that already broke a lot of user data

How to handle big issue on live server that already broke a lot of user data Have you ever caused a big problem on production and panicked, not knowing what to do to remedy the shitty situation you've just caused? Well, I did. So here's what I've found. 1. Find what caused the issue Its probably the most recent changes you've just pushed. If not, then try to remember whatever code you're ever had doubt in, it's usually one of these. If you cant find it, tough luck. Lets move to the next step. Don't spend too much time on finding the rootcause. 2. Stop the function that caused the issue (eg:cron that cancels invoice) In my case, it was a cronjob. Easy fix, just disable the cronjob on both the server and in the codebase. If it was a crucial add/update process, then just disable it in the code, returning an alert 'something went wrong, we're currently fixing it' should suffice. Just make sure to apologize to your support person later. 3. Update curr...