Desktop System Failure

Help Support Homesteading Forum:

Peanut

Awesome Friend
Neighbor
Joined
Dec 16, 2017
Messages
14,163
Location
Bama
Hey folks, looking for assistance. I had a total system failure last night. Its a Dell-5yr old. 10months ago I installed a new harddrive and loaded linux mint 20.1. It was a clean load with partitions. System ran great. It was updated last on Tuesday of this week.

The screen froze - power off reset

This morning I tried to boot to a thumb drive ISO. I have 5 versions of mint on iso's. Tried 3, the system does not see the thumb drives.

I get a root menu command prompt and the keyboard works.

I do not know the commands to navigate or trouble shoot from the root menu. Anyone do phone support? @Haertig

Help? "You are in emergency mode..."

system failure.JPG
 

TeeJ

Awesome Friend
Neighbor
Joined
Oct 28, 2021
Messages
349
Location
Nestled in the Rocky Mountains
I'm not as up on Linux as I used to be, but I'm pretty sure that happens when there's corruption in system files and the OS can't load.
As far as booting from USB, have you tried to pull up the boot menu at startup? Should be the F2 or F12 key (I'll just tap them alternately during startup until the menu opens). From there, hopefully you can pick the thumb drive!
 

Peanut

Awesome Friend
Neighbor
Joined
Dec 16, 2017
Messages
14,163
Location
Bama
Found a list of commands. checked hardware configuration and usb connections. Looks like there are no hardware failures. Looks like the OS is toast. I started organizing for a full data back up this week. If I can boot to a thumb... Maybe I can save my data, then wipe the drive and reload

hard configa.JPG
 

Magus

The Shaman of suburbia.
Neighbor
Joined
Dec 13, 2017
Messages
2,986
Location
Somewhere over here to the south.
When you reboot, tap F8 repeatedly, it should give you a list of drives to boot from, in linux you can try this:
If your computer uses BIOS for booting, then hold down the Shift key while GRUB is loading to get the boot menu. If your computer uses UEFI for booting, press Esc several times while GRUB is loading to get the boot menu.
 

Magus

The Shaman of suburbia.
Neighbor
Joined
Dec 13, 2017
Messages
2,986
Location
Somewhere over here to the south.
And Linux shat itself like this? I'm no Bill gates, but that resembles damage done by either a direct hack attack or a worm! if it was Windows, I'd say someone nuked your System 32 folder. when my son gets back in here ask him, he's a bit of a propeller head when it comes to Linux,
 

Magus

The Shaman of suburbia.
Neighbor
Joined
Dec 13, 2017
Messages
2,986
Location
Somewhere over here to the south.
 

Peanut

Awesome Friend
Neighbor
Joined
Dec 16, 2017
Messages
14,163
Location
Bama
And Linux shat itself like this? I'm no Bill gates, but that resembles damage done by either a direct hack attack or a worm! if it was Windows, I'd say someone nuked your System 32 folder. when my son gets back in here ask him, he's a bit of a propeller head when it comes to Linux,
Exactly, Tha'ts the impression I got last night while it was happening (not the first attack I've seen). I thought I was getting nuked. My network connection froze first, I couldn't close it or my browser (logged in here, only window open). In fact I was responding to a post.

I opened the system monitor and was trying to open other system tools when the start menu froze. At that point I did a hard power off reset. system wouldn't reboot. It was late so I shut it off.
 

Magus

The Shaman of suburbia.
Neighbor
Joined
Dec 13, 2017
Messages
2,986
Location
Somewhere over here to the south.
Who do you figure? Government, an enemy or just some prick?
I haven't needed this in a decade, but ALL us admins swore by it!


Not only did it stop attacks dead, it told the IP of the little B-tard and we'd either email him a google earth pic of his house or post his IP on a certain place we knew of and listen for the shrieks of agony as his computer melted through the floor. There was this one guy from England, he had a virus that ever time you went to a certain website, your computer auto rebooted, and after a week, it fried your BIOS
 

Neb

Jack of all trades master of some
Neighbor
Joined
Sep 4, 2020
Messages
5,627
Exactly, Tha'ts the impression I got last night while it was happening (not the first attack I've seen). I thought I was getting nuked. My network connection froze first, I couldn't close it or my browser (logged in here, only window open). In fact I was responding to a post.

I opened the system monitor and was trying to open other system tools when the start menu froze. At that point I did a hard power off reset. system wouldn't reboot. It was late so I shut it off.
There was a unix command...

Sync

I would issue that commando twice before shutting down. That command forces the disk to synchronize with memory. Unix used to defer updating disk directory info to improve performance.

Ben
 

Peanut

Awesome Friend
Neighbor
Joined
Dec 16, 2017
Messages
14,163
Location
Bama
I figure it was just some jerk with nothing better to do. Since I was only connected to our forum someone randomly picked my ip or they were watching the forum. (Unless it was the USDA. I was doing research on a family of plants earlier and using their data base.)

If this was a windows system I’d have the tools needed for something like this. I haven’t used unix for work since the 90’s. I played around with linux during the same time frame. Didn’t use either after 2000.

I’ve been running linux for about a year now. It runs so well I’ve had no reason to dig into it and memorize 100 commands.

I have a vpn I use for general browsing. I don’t use it when I’m just logged into out forum. We’ve had enough issues with speed lately. I might have to rethink that.
 

Neb

Jack of all trades master of some
Neighbor
Joined
Sep 4, 2020
Messages
5,627
I figure it was just some jerk with nothing better to do. Since I was only connected to our forum someone randomly picked my ip or they were watching the forum. (Unless it was the USDA. I was doing research on a family of plants earlier and using their data base.)

If this was a windows system I’d have the tools needed for something like this. I haven’t used unix for work since the 90’s. I played around with linux during the same time frame. Didn’t use either after 2000.

I’ve been running linux for about a year now. It runs so well I’ve had no reason to dig into it and memorize 100 commands.

I have a vpn I use for general browsing. I don’t use it when I’m just logged into out forum. We’ve had enough issues with speed lately. I might have to rethink that.
There were only 2 command that needed memorized.

Man - opens the help manual
Apropos - finds subject in the manual related to topic

I managed some unix systems when I worked in the physics labs at Pitt 30 years ago.

Ben
 

Haertig

Awesome Friend
Neighbor
Joined
Dec 6, 2017
Messages
3,798
Location
Colorado
The fact that your system froze is not indicative of anything specific. Could be many causes. This is a rare happening in Linux, but it does happen. Often times it is the "window manager" that freezes. Your entire system is not frozen, but it feels that way if you don't know what to do.

The overall plan is to:
(1) Get out of your locked state
(2) Reboot from a thumbdrive or a CD - anything but your normal hard drive (which we assume is possibly corrupted at this point)
(3) Run a "file systems check" on your hard disk
(4) Attempt to reboot from your hard disk again to see if everything is fixed

To address (1) above:

You can try opening up a new "console window". Your system should be configured to have several of these. You access them by holding down "control" and "shift" and a function key ... F2, F3, etc. You can try hitting ctrl-shift-F2 when things appear frozen and see if that brings up a new console window. A console window is just a login prompt. You can use your login/password to get a command prompt. But if you don't know the command line, then this hasn't done you any good. You can go back to your (frozen) GUI window with ctrl-shift-F7. It may not be F7 on your system, but it will be some function key. Try them all. I am running LinuxMint, same as you, and my GUI is on F7, so yours probably is too.

It's too involved to tell you what you can do from the command line in this single post if you don't already have some knowledge of that. Remember that when you bring up a new console window, you have to login first, before you are given a command prompt (which will probably end in a $ sign). Once you get that command prompt, you can blindly try this (without knowing what it's doing):
Code:
sudo su -
It will ask you for a password, enter yours, even though you just entered it a few seconds ago when you logged in.
You will now get another command line prompt, but this time it will probably end in a # sign, not a $ sign. Type the following:
Code:
sync;sync;sync;sleep 10;shutdown now
Sit back and do nothing more for a minute or so and you should hear your system power down after that. Hopefully. If everything went as planned.

Now, this is hitting a locked up screen with a big hammer, but it's a safe hammer, and one that can be easily communicated. I would do things differently, but it's harder to explain what you would do (it's more interactive).

You got out of your locked state in a different way than I described above. You killed the power to your computer. Doing that could have corrupted your hard disk. Linux is very good at recovering from a "crash", which is what happens when you yank it's power cord. But it is not perfect and it can't always recover gracefully. I can count on one finger how many times I have seen a crash corrupt Linux, so it is very rare. But it does happen. One thing you can do to help protect yourself when you decide to do a human-initiated crash, is let the computer sit there, untouched, for five minutes before you yank it's power. That means don't touch the keyboard, don't move the mouse, just let it be as idle as possible. This gives the disk(s) time to sync if the system is still sane enough to run that operation as it would normally do on a routine basis.

To address (2) above:

You have already managed to solve this one. You can boot from a thumbdrive. There are lots of potential problems here. If you are booting UEFI, it may be set up for "Secure Boot". If you are booting legacy (BIOS), then it may be set to try booting from the hard drive before it tries to boot from the thumbdrive. Which defeats the purpose if your hard drive is temporarily dead. But, you said you got it booted from a thumbdrive, so you're all good. All the potential problems didn't affect your case. So you're in good shape.

I'm going to break this post in half and hit "Post reply" now. And then I'll post the second half of my troubleshooting tome in a followup post (which will take me a few minutes to type...
 
Last edited:

Haertig

Awesome Friend
Neighbor
Joined
Dec 6, 2017
Messages
3,798
Location
Colorado
I had typed in a bunch more information, but then I realized I was targeting a Linux command line guru, not a normal person. So I need to re-think and re-write a bit to make it more understandable. A phone call might be better if you are really interested in pursuing this. No guarantee that I can save your system, but I'm willing to try. Booting issues are very difficult to remotely troubleshoot compared to other problems. Sheesh - you could have a hard drive that hard drive that went belly up and nothing we do will save it. But that would be unusual, so don't lose heart. It's probably a simple thing. Well, simple if you happen to have a Linux guru standing next to you with physical access to your computer. I will take this offline for now, and PM you my cellphone and email info. Contact me if you are so inclined. I am in Mountain timezone. And I will need to be cooking dinner shortly as my wife is getting off work soon. If you would rather keep working from the HCL forum in this thread, I will do that as well, as long as the mods don't kick our butts for getting way off the basic forum purpose of HCL.
 

Haertig

Awesome Friend
Neighbor
Joined
Dec 6, 2017
Messages
3,798
Location
Colorado
One other thing... That info you posted above - the pictures of your screen - you were booted via a thumbdrive at that time. So that is information related to your thumbdrive, not your hard disk! So when you saw that error about not being able to run fsck, that had nothing to do with your hard drive. You can see this illustrated where in your pictures it's talking about a 15Gb disk. I dare say, I hope your hard disk is larger than that. Also note, when booting Linux identifies your hardware at boot time. So it does things trying to figure out what you have. Many of those things it checks fail, and that is how it learns what your hardware situation is. In other words, "Try this, if it fails, then my assumption was wrong, so try something else". In other words - don't get overly concerned because you see the word "FAIL" in boot output. It may or may not be meaningful.

Interesting point: Linux determines your hardware situation at boot time. Windows determines it at installation time. This is one reason why you can install Linux on a hard disk, then remove that hard disk and put it into a totally different computer, and Linux will boot just fine. Try that with Windows. Guaranteed to fail. You can't remove the hard disk (the bootable one) from a Windows computer and install it into a different computer and expect it to boot. Not without some technical manual work. And even with that, it's very iffy, and your system may never be "right in the head".
 

Peanut

Awesome Friend
Neighbor
Joined
Dec 16, 2017
Messages
14,163
Location
Bama
Thanks to all if you. And your offer heartig! I was having trouble getting out of the hole this morning so gave a shout for pointers. Couldn’t remember how to change to boot sequence on a Dell, to allow booting from a usb port. The bios setup utilities don’t allow it, only the F12 key allows it during initial power up. A weird little quirk that kept me stumped for hours.

Once I could boot from a thumb drive I was home free. I’ve been backing up data for 7hrs now. Hopefully I’ll be done by midnight.

Currently I’m not interested in repair/restoring the OS and using it. I’ve actually been thinking about doing a wipe for over a month, then reload with mint 20.2. A full data backup was the first step. I started moving files this past monday. This event caught me in the middle of that process so I have to back up everything as is, a mess. And backup Gigs that I wouldn't have otherwise.

Once I get my data saved I’ll try to repair the OS just to see if I can do it. Then I’m going to wipe and reload. I don’t like how I originally partitioned the drive anyway, to much unusable space.
 
Last edited:

Magus

The Shaman of suburbia.
Neighbor
Joined
Dec 13, 2017
Messages
2,986
Location
Somewhere over here to the south.
Invest in a 2GB external and backup constantly.
I have 5 of them in a shielded and insulated 50 cal can.

And when in doubt, ask the admin.
You might look up my son too. He g33ks for red hat last I heard.
 

Peanut

Awesome Friend
Neighbor
Joined
Dec 16, 2017
Messages
14,163
Location
Bama
I had the 2TB drive partitioned. Prt 1 - partition table, Prt 2 - operating sys, Prt 3 - Home folder, Prt - 4 unmounted (previously).

What ever occured... Prt's - 1&2 are now unmounted. whether I just have to mount them and the files are still there - unkown.

Prt 3 is mounted, all my data is there.

I'm taking my time, making good backups. Afterwords I'll try to repair the problem. I'm in no hurry, I haven't kept my laptop updated so I'm taking care of that and making a 2nd hard backup on it also.

(I applied text)

partition tblab.JPG
 

Neb

Jack of all trades master of some
Neighbor
Joined
Sep 4, 2020
Messages
5,627
I had the 2TB drive partitioned. Prt 1 - partition table, Prt 2 - operating sys, Prt 3 - Home folder, Prt - 4 unmounted (previously).

What ever occured... Prt's - 1&2 are now unmounted. whether I just have to mount them and the files are still there - unkown.

Prt 3 is mounted, all my data is there.

I'm taking my time, making good backups. Afterwords I'll try to repair the problem. I'm in no hurry, I haven't kept my laptop updated so I'm taking care of that and making a 2nd hard backup on it also.

(I applied text)

View attachment 86469
Fsck ????


Ben
 

Haertig

Awesome Friend
Neighbor
Joined
Dec 6, 2017
Messages
3,798
Location
Colorado
Your first partition is used for booting from your hard disk in EFI mode. Leave it alone. It is known to your system as /dev/sda1 (per the output you screen-shotted and posted).

The second partition is your OS, LinuxMint. It is probably known to your system as /dev/sda2. Using whatever utility it is you are using to display your partitions, you can probably just click on that second volume in the display and it will tell you what it is known as (down below, where it says "Device"). While you're at it, look at the other volumes too and see what they are know as. Probably /dev/sda3 and /dev/sda4. That second volume, /dev/sda2, is not needed when you booted from the thumbdrive. You can mount it, look at it, check it for errors (fsck - but you have to unmout it before running fsck), and back it up if you want. If you are planning on installing a fresh copy of the OS, you probably don't need anything from this filesystem. However, if you had previously installed/configured a lot of software and want a record of what you configured so you can duplicate that later, there will be stuff in this filesystem that you want. It won't hurt to back it up - if you don't need anything from it later, no harm done, and you can eventually delete it's backup. If you have enough space to backup the whole thing, just do that. But if backup space is tight, probably the only thing you really need out of it is /etc. If you don't know what I'm talking about, or don't know if you need more than just /etc, then you probably don't.

A few commands that may be useful to you:

To see if you are "root" or not, use the command "id" or "whoami"

If you are not "root" and want to be, use "sudo su -"

To see what is mounted, use "mount"

To mount something that isn't: First you have to create a "mount point". This is just an empty directory. Do this as "root". You probably already have a directory named /mnt, and it may or may not be being used for mounted things already (that would depend on which specific LiveCD you booted from). So, since I don't know if you have this directory, or if it's currently used, we'll just make a new one. As "root", type "mkdir -p /my-mount/part1", then "mkdir -p /my-mount/part2", then "mkdir -p /my-mount /part3", then "mkdir -p /my-mount/part4". Now you have four mountpoints to play with - one for each of the partitions on your hard disk.

Now, say you want to mount your fourth partition (which you said is currently unmounted). As "root", type "mount /dev/sda4 /my-mount/part4". When you want to unmount it, the command would be "cd /; umount /dev/sda4" or "cd /; umount /my-mount/part4". Note that the command is "umount", not "unmount". Also, the "cd /" part is to get you out of that filesystem in case you had previously cd into it. If you are in the filesystem, you can't unmount it. Hence the "cd /" to get out of it.

To look at something that is mounted, run "cd /my-mount/part4" (or whatever partition number you mounted and want to look at). Then you can run "ls" to see what's there. Or, you can use the file manager GUI program to look at things. The caveat above about having to "get out of" a filesystem before you can unmount it holds here as well.

If you want to backup a partition, using a file-based backup method, that partition will have to be mounted for you to do that. If you are doing an image (bit-for-bit backup of the whole partition) then you want it to be unmounted.

If there is damage to the filesystem, you will want to clean it up. If the damage is bad, you wouldn't have been able to mount it above. But sometimes you can mount a damaged filesystem, sometimes you can't. To check it for errors (and to fix them), you need to unmount the filesystem first. This is important. All of this will need to be done as "root". As an example, say you tried to mount your fourth partition, /dev/sda4, and it wouldn't mount. You may get some failure message complaining about a "dirty" filesystem, or telling you "the filesystem needs to be checked", or something else. First, we want to make dang sure it's not mounted. So type "umount /dev/sda4". If it wasn't mounted in the first place, it will complain about you trying to do this, but just ignore that complaint. Now that we are positive that it is unmounted, you should run "e2fsck -fpy /dev/sda4". Note that "e2fsck" is specific for Linux filesystems of type ext2/ext3/ext4. Your 2nd, 3rd and 4th partitions are ext4. So "e2fsck" is appropriate for these three partitions. Your 1st partition, that has the UEFI boot stuff on it, is NOT ext4 though, so don't use the "e2fsck" command on that 1st partition. We're not messing with that 1st partition anyway. I specified the options "-fpy" to the "e2fsck" program. The f tells it to run the check even if it thinks it doesn't need to. The p tells it to automatically correct any errors it finds. And the y tells it to automatically answer "yes" to any question it might want to ask you. In other words, "just fix everything that's broken and don't bother me".

"fsck" programs can take a long time to run. Sometimes hours. It depends on how bad things are, what needs to be fixed, how big the filesystem is, etc. Just run the command and walk away until it finishes. Once it has completed, try to mount the filesystem again. Hopefully this time it will mount and you can do your backups or whatever else you want to do with it.

Note that after running e2fsck" on your second partition, /dev/sda2, your OS partition - you may be able to boot from your hard drive again. That was kind of the goal. Do a normal shutdown or restart from your thumbdrive-booted system and see what happens when you try to boot it normally from the hard drive. Cross your fingers!
 

TeeJ

Awesome Friend
Neighbor
Joined
Oct 28, 2021
Messages
349
Location
Nestled in the Rocky Mountains
Invest in a 2GB external and backup constantly.
I have 5 of them in a shielded and insulated 50 cal can.

And when in doubt, ask the admin.
You might look up my son too. He g33ks for red hat last I heard.
I agree with @Magus on keeping an external HD backup in a safe place. I keep one in an EMP proof bag in a portable fire safe in my home office and another in the same setup at work about 25 miles away. I figure anything that destroys both backups will take me out too, so I'm not worried about it beyond that. The drive I use is: https://www.amazon.com/Seagate-Portable-External-Photography-STDR5000100/dp/B01LZP2B23?th=1 - available between 1 TB & 5 TB. I've used hundreds of these for work and have yet to have one fail that wasn't physically abused...
 

Magus

The Shaman of suburbia.
Neighbor
Joined
Dec 13, 2017
Messages
2,986
Location
Somewhere over here to the south.

I have had one of these going on 6 years, its been rained on, gnawed on by a retarded raccoon, stepped on by an alien, and lost and found several times, its a tank! I only would add 1 thing, a piece of tape labeling what's on it. In my case, a decade worth of my web comics.
 

Latest posts

Top