How to handle big issue on live server that already broke a lot of user data

Have you ever caused a big problem on production and panicked, not knowing what to do to remedy the shitty situation you've just caused? Well, I did. So here's what I've found.

1. Find what caused the issue

Its probably the most recent changes you've just pushed. If not, then try to remember whatever code you're ever had doubt in, it's usually one of these. If you cant find it, tough luck. Lets move to the next step. Don't spend too much time on finding the rootcause.

2. Stop the function that caused the issue (eg:cron that cancels invoice)

In my case, it was a cronjob. Easy fix, just disable the cronjob on both the server and in the codebase. If it was a crucial add/update process, then just disable it in the code, returning an alert 'something went wrong, we're currently fixing it' should suffice. Just make sure to apologize to your support person later.

3. Update current live database with data from backup

That is, if you even have a backup in the first place. If not, then you've just learned a valuable lesson in database management. Always have backup. Also good luck trying to remap/fix the broken data.

4. Find the underlying issue & its solution

Now that everything have stopped breaking itself, its the perfect time to find the rootcause & solve the problem. Make sure you imprint this problem in your memory, we don't want to cause another problem on production now do we?

5. Fix code & deploy changes

After you've fixed it, make sure to have another programmer or your tech lead to review your changes. Best case scenario they see that it can be even further improved. After that, deploy and test the function again on live. Hopefully, it wont break anything this time.

Comments

Setup existing IONIC project in local

Setup existing IONIC project in local Steps: clone git repo install ionic - npm install -g @ionic/cli masuk folder repo project install npm dependencies - npm install run ionic project - ionic serve buka android studio / xcode - ionic cap open $var - $var = ‘android’ atau ‘ios’. ada dua je option, replace $var dgn dua option tu sync changes vscode & dkt android studio / xcode - ionic cap sync good luck Common issues: dependency conflict check https://www.npmpeer.dev/ utk tengok version yg compatible try naikkan/turunkan version dependency yg keluar dkt error. tembak je sampai hilang error g radle issue try upgrade gradle. kalau tak boleh, try remove folder android & build semula e rror cocoapod make sure install xcode make sure install cocoapod error java home not found utk mac, buka ~/.zshrc & masukkan chang es dkt VSC tak masuk android studio / xcode try ionic cap sync try quit & buka semula cordo va.variables.gradle no...

🪄 Useful git spells

Navigating the world of Git can be akin to mastering an arcane art. For the savvy developer, knowing the right incantations can mean the difference between seamless collaboration and catastrophic code conflicts. Here's a compendium of Git commands that are essential for those critical moments. Handle them with care, for they wield great power. Reverting to a Specific Commit (Non-Shared Branches Only) When you need to undo changes and return to a known good state: `git reset --hard <commit-hash>` - This will reset your branch to the specified commit. `git push -f origin` - Force push the changes to overwrite the remote branch. Merging Branches Like a Pro To incorporate changes from one branch into another, follow these steps: `git checkout <branch-to-merge-to>` - Switch to the branch you want to update. `git fetch origin` - Fetch the latest changes from the remote. `git pull` - Pull the latest changes into your local branch. `git status` - Check the status of your branch...

Vulnerabilities in dependencies used in projects

Vulnerabilities in dependencies used in projects The case of 4Chan never updated any dependencies since 2009 one dependency used was publicly-known to have multiple critical vuln Ghostscript - https://security.snyk.io/vuln?search=ghostscript Local File Inclusion - process pdf files but never checked if the pdf is actually pdf Remote code execution - after injecting. can run scripts on the server The modern wake up call - this event reminded everyone to stay up to date with dependencies updates Dependencies we’re using: mPDF - fixed vuln for versions lower than 7.1.8 deserialization of untrusted data - similar with how 4chan got hacked aws/aws-sdk-php - fixed path traversal vuln for versions lower than 3.288.1 can go back and open directories that are not public arbitrary code execution vuln for versions lower than 3.2.1 Best practices: stay up to date with the tech world join forums like reddit and stackoverflow check websites that publish security reports snyk security - https:/...

Firdaus Safari

Search This Blog