Linux@lemmy.mlbyMindfulMaverick@piefed.zip
6 days

How to diagnose a complete system freeze (no REISUB, no mouse/kb, have to hard reset)?

English

Hey everyone,

I’m running into a frustrating issue and could use some guidance on how to pinpoint the faulty component.

My system completely locks up every few hours. It’s not just a DE crash; the entire machine becomes unresponsive. The mouse and keyboard are completely dead (no cursor movement, Caps Lock key doesn’t toggle). I’ve tried waiting 10-15 minutes to see if it recovers, but it never does.

REISUB does not work. Holding Alt + SysRq and pressing the keys in order does nothing. The only way out is a hard reset using the case button.

The last time this happened, I ended up buying components for a new computer and replaced them one by one until I found the faulty one. I’d rather try a more targeted approach this time. Though if it takes too much effort, I do have another computer I can fall back on.

Any advice on how to diagnose this efficiently? Logs to check, stress tests to run, or hardware to suspect first?

Thanks in advance!

16
    • klangcola@reddthat.com
      6 days

      I had / have a similar issue that started at some point on my Ryzen 7 laptop with Kubuntu 24.04. I haven’t tried REISUB yet, but otherwise same symptoms.

      RAM is the usual suspect. I ran memtest for 24h++ with no errors Also tailed dmesg and journalctl to a remote machine, and checked journalctl after reboot. No errors reported. Presumably because the system hard locked before it had a chance to log the error.

      I never found a root cause, but after I changed the KDE Power Profile from Eco to either Balanced or Power (I don’t remember which) the random freezing reduce from 1-3 times per day to once every few feels of continuous uptime.

      So my guess is some kernel driver bug relating to power states of the CPU ( or GPU nVidia 3060 with 590 drivers)

      • klangcola@reddthat.com
        6 days

        For actual advice:

        • run memtest to verify RAM. Do multiple passes, at least overnight.
        • Check cooling? RAM and other things can overheat and cause locks, not just CPU.
        • Can you throw a couple of different distros on there, to try different permutations of kernel and drivers, old and new.
        • journalctl --follow (with sudo), dmesg -w. I ran these over ssh from a remote machine. Even better if you can run it on a local 2nd monitor. The point is to have them open the whole time since it’s too late to change once the system is locked up
    • brownmustardminion@lemmy.ml
      6 days

      I’ve had these issues during high intensity GPU usage on an nvidia gpu. It’s the only times REISUB didn’t work and I’ve had to do a hard reset.

      Not much I can contribute other than don’t rule out a nvidia driver problem.

    • RedstoneValley@sh.itjust.works
      6 days

      I had this happen when a game at random times filled up the available memory so quickly that it froze completely before any OOM watchdog could catch it.

      • Minnels@lemmy.zip
        4 days

        Same happend to me. I just made the swap file really big. Haven’t had the problem since. It was 4gb default, tried 16, was better and now on 32gb it is all good. I have 32gb ram.

    • Bricked@feddit.org
      6 days

      The two times something like this happened to me, it was always a RAM issue. One time it was XMP, the other time I forgot to install swap, so the memory simply ran out

    • shoveler@piefed.socialEnglish
      6 days

      This does sound like a hardware problem:

      1. Find the motherboard brand/model # and memory brand/model.
      2. Check the motherboard’s manufacturer’s support page for compatible memory with that specific motherboard. Also check their forum for similar problems and solutions.
      3. Unplug all non-critical peripherals ( might be a driver issue).
      4. Swap the two (or more) memory sticks, see if that changes the freezing somehow.
      5. Check mother board manufacturer for updated BIOS, especially if the new BIOS addresses memory concerns.
      6. If the BIOS doesn’t have a memory test, Try MemTest86+.
      7. If it’s not the hardware, the BIOS, or the BIOS settings: Boot a live USB stick, see if the problem still exists ( might be a corrupt install somewhere; backup data and install a different distro; on a different drive if available; or stick (a backup of?) the boot drive in a different machine).
      8. Dig into the logs, as mentioned elsewhere.
    • linuxguy@piefed.caEnglish
      6 days

      Setting up a watchdog and crash kernel might help or let you diagnose it

    • Brickfrog@lemmy.dbzer0.com
      6 days

      My system completely locks up every few hours. It’s not just a DE crash; the entire machine becomes unresponsive. The mouse and keyboard are completely dead (no cursor movement, Caps Lock key doesn’t toggle).

      Before you rule out a DE (or Wayland issue), are you 100% sure the entire system is unresponsive? Like is it still online and responding to ping or SSH? Just to be sure try enabling SSH on the system - then set up a spare laptop/computer on the same network that can normally ping or SSH to your Linux system. Next time the issue occurs test to see if the Linux system is truly unresponsive by checking if it is still responding to pings and allowing you to SSH into it.

      If you don’t have a spare laptop/desktop but do have an Android phone you could do the same with Termux.

      Also if you can SSH into it you should be able to force logout your own user, that would bring your Linux system back to the login screen and you’d then be able to use mouse/keyboard normally again. (run “who” to view logged in users, run “pkill -u your-username” to kill and logout the user, may need to run those with sudo)

      Only reason I mention it is that I have an ancient desktop that exhibits similar behavior occasionally but my system is still alive on the network. So far for me it seems like it might be a Wayland + Nvidia + GNOME issue. Once I switched back to X11 it doesn’t seem like the issue occurs anymore.

    • Kory@lemmy.ml
      6 days

      First I always check with sudo journalctl -r Check journalctl --help for more options or do sudo journalctl --since "2015-06-26 23:15:00" --until "2015-06-26 23:20:00" Then search errors online or come back with more questions.

      • klangcola@reddthat.com
        6 days

        Doesn’t work in my experience, or I’m typing it wrong. I can use the journalctl boot filter to show the current boot, the 2 boots ago, but not the previous boot where the system crashed.

        So I end up filtering by time instead with --since

      • pinball_wizard@lemmy.zip
        6 days

        To pile on to your (excellent) suggestion, OP might try enabling POST (power on self test) in BIOS. It takes awhile on modern RAM, but I’ve had it identify failing RAM sticks for me.