How to handle big issue on live server that already broke a lot of user data

Have you ever caused a big problem on production and panicked, not knowing what to do to remedy the shitty situation you've just caused? Well, I did. So here's what I've found.

1. Find what caused the issue

Its probably the most recent changes you've just pushed. If not, then try to remember whatever code you're ever had doubt in, it's usually one of these. If you cant find it, tough luck. Lets move to the next step. Don't spend too much time on finding the rootcause.

2. Stop the function that caused the issue (eg:cron that cancels invoice)

In my case, it was a cronjob. Easy fix, just disable the cronjob on both the server and in the codebase. If it was a crucial add/update process, then just disable it in the code, returning an alert 'something went wrong, we're currently fixing it' should suffice. Just make sure to apologize to your support person later.

3. Update current live database with data from backup

That is, if you even have a backup in the first place. If not, then you've just learned a valuable lesson in database management. Always have backup. Also good luck trying to remap/fix the broken data.

4. Find the underlying issue & its solution

Now that everything have stopped breaking itself, its the perfect time to find the rootcause & solve the problem. Make sure you imprint this problem in your memory, we don't want to cause another problem on production now do we?

5. Fix code & deploy changes

After you've fixed it, make sure to have another programmer or your tech lead to review your changes. Best case scenario they see that it can be even further improved. After that, deploy and test the function again on live. Hopefully, it wont break anything this time.

Comments

Setup existing IONIC project in local

Setup existing IONIC project in local Steps: clone git repo install ionic - npm install -g @ionic/cli masuk folder repo project install npm dependencies - npm install run ionic project - ionic serve buka android studio / xcode - ionic cap open $var - $var = ‘android’ atau ‘ios’. ada dua je option, replace $var dgn dua option tu sync changes vscode & dkt android studio / xcode - ionic cap sync good luck Common issues: dependency conflict check https://www.npmpeer.dev/ utk tengok version yg compatible try naikkan/turunkan version dependency yg keluar dkt error. tembak je sampai hilang error g radle issue try upgrade gradle. kalau tak boleh, try remove folder android & build semula e rror cocoapod make sure install xcode make sure install cocoapod error java home not found utk mac, buka ~/.zshrc & masukkan chang es dkt VSC tak masuk android studio / xcode try ionic cap sync try quit & buka semula cordo va.variables.gradle no...

🗑️ Clear storage Mac OS

🗑️ Clear storage Mac OS 1: Clear system cache: Go to Finder > Go > Go to Folder, then type in "~/Library/Caches" and hit enter. Select all the folders inside the Caches folder and delete them. 2: Clear system logs: Go to Finder > Go > Go to Folder, then type in "/var/log" and hit enter. Select all the files inside the Log folder and delete them. 3: Remove unused language files: Go to Finder > Go > Go to Folder, then type in "/Library/Languages" and hit enter. Delete all the language folders you don't need. 4: Uninstall unused apps: Go to the Applications folder and delete the apps you don't use. 5: Clean up system files: Use a system cleaning tool like CleanMyMac X to scan and remove unnecessary system files. 6: If you have npm installed, clear the caches once in a while with ‘sudo npm cache clean --force’ 7: If you have ionic projects, open the ‘.angular’ folder and delete the ‘cache’ folder inside it.

Vulnerabilities in Dependencies

Vulnerabilities in Dependencies - A Lesson from 4Chan Case Study: The 4Chan Security Breach The Neglect 4Chan had not updated its project dependencies since 2009. The Vulnerability One specific dependency was publicly known to have multiple critical vulnerabilities: Ghostscript. Ghostscript Vulnerability Reports (Snyk) Attack Vectors: Local File Inclusion (LFI): The system processed PDF files without verifying if the uploaded file was actually a PDF. Remote Code Execution (RCE): After a successful injection, attackers could run malicious scripts directly on the server. The Wake-Up Call This event serves as a modern reminder of the critical importance of keeping dependencies up to date. Dependencies Audit & Status These are some of the dependencies commonly used. mPDF (Fixed) Vulnerability: Deserialization of untrusted data (similar to the 4Chan/Ghostscript exploit). Risk: Affects all versions lower than 7.1.8. aws/aws-sdk-php (Fixed) Path Traversal Affects versions lower t...

Firdaus Safari

Search This Blog