everything is "invalid file format" after upgrade from 11.4 to 12.0-RELEASE

noodlefling

noodlefling

I was running 11.4-RELEASE-p12 and ran

freebsd-update -r 12.0-RELEASE upgrade

It seemed to run normally, I did the merges, then rebooted.

When it came back up I did this...

# freebsd-update install
Installing updates...Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
etc...

That went on for a bit, it complained a lot about missing directories, then finished. Now I can't do much.

# uname -a
FreeBSD [REDACTED] 12.0-RELEASE-p13 FreeBSD 12.0-RELEASE-p13 GENERIC amd64
# freebsd-version -uk
ld-elf.so.1: /lib/libedit.so.7: invalid file format
# freebsd-update fetch
ld-elf.so.1: /lib/libedit.so.7: invalid file format
# pkg info
ld-elf.so.1: /lib/libm.so.5: invalid file format


Feeling a bit nervous. Suggestions?
 
Quite a few problems with lib files. I would reboot in single user mode and fsck / to make sure the file system is okay. See if fsck gives errors. See what you have then, replace mangled files or consider the rollback option of freebsd-update.
 
I'd use the rollback option right now if freebsd-update were working. It will take a while to get on-site, but I guess I'm going in!

# freebsd-update rollback
ld-elf.so.1: /lib/libedit.so.7: invalid file format
 
You know you can upgrade straight to 12.2? There is no need to install any intermediate versions.

If you've only done a single freebsd-update install, then rebooted, only the kernel was updated. I suspect something went wrong with the new kernel. Your old kernel should be in /boot/kernel.old. Not sure when the kernel selection was added but you might be able to select it in the 'beastie' menu. If not, drop to the loader prompt, unload and load /boot/kernel.old/kernel. If you're booting from ZFS you also need to load ZFS; load /boot/kernel.old/opensolaris.ko and load /boot/kernel.old/zfs.ko. Then you can boot (normal mode) or boot -s (single user mode).
 
Wasn't sure about which to update to, but up until a few hours ago, I've always been pretty comfortable with version upgrades. Thanks for the 12.2 tip, though.

That said, I did the first freebsd-update install, rebooted, and things seemed fine. Then I did the second freebsd-update install and that's when things got bad.

If I try to boot to single user with the new kernel, it doesn't accept any keyboard input. If I use a different kernel (from the kernel options on the beastie menu), it recognizes the keyboard, but gives me similar "invalid file format" errors.

I have UFS, not ZFS. I will try the unload thing, but am not optimistic.

I have recent backups, but not sure how to get them from tape if I can't even get into single user. Wasn't looking for Remedial Update School today, but I guess we can't pick our pitfalls!
 
Yeah, the unload thing had the same result as trying to boot from the kernel in the menu. I guess we're down to backups. Yikes! I suppose I'll need to burn an 11.4 disc? Haven't done this in ages...
 
Hmm. It doesn't want to seem to boot from the DVD drive. Getting a little panicky.

Are there any other tricks you can do from the loader prompt? Such as replacing all those libraries that don't seem happy? If the backup kernel is available, are all those .so files hanging around somewhere too?
 
Don't panic - it will only make things worse (easy to say when not my server/machine). But try and slow down and take a few deep breaths.

Can you boot off USB key?

Once you get something you can boot off, you'll be able to have enough of an environment to backup (if not already done) and copy files across (like the .so files)/try and rebuild it.

But if you can boot it's also worth checking (as others have advised) that you don't have hardware issues like a failing drive.
 
Ha! Somehow the first ISO didn't burn correctly. So, there is hope. It's just that the system has many redundant chunks of hardware and it takes it forever to reboot, while I sit there nervously hoping it isn't frozen.

Will try the USB key next. It's a server grade system and kinda pigheaded, so I don't know if that will work. Second DVD worked in a second system, did not work on first try with the broken system, but will give it another go.

Assuming at some point I get a shell, I'm guessing I should try put 11.4 back on there (so everything else has a greater chance of working). I partitioned things the old way, so I'm guessing replacing / might do it, if not / and /usr, and maybe /var.

But I'm getting ahead of myself. At least I have the tape, but I need to get to a place to access it. All I want to do is put / where it was a few hours ago so at least I can talk to it!
 
Last edited:
If server-grade you might also have remove console/BMC/iDRAC with Virtual Media boot option? Any logs, event viewers etc. that might show hardware issues?

Get a bit of a plan together - what do you want to achieve? Have you got another machine you can experiment with - do the steps on the experimental machine first.

Is your goal to try and rollback to 11.4? If yes, how are you going to do that? Again, that's where an experimental machine comes into play. Build it with 11.4, do the first upgrade step to 12.0, then see if you can get it back to 11.4 using a boot disk/Live Shell.

Or if you want to push onto 12.x, then have you got what you need for that, and how is it going work? Again, a test machine to experiment on is a good idea.

If you half-installed 12.0 then stay with that - don't jump to 12.2 or anything - just be a bigger mess.

But first step is to find something that will boot, then you can see if you can mount the drive(s), copy anything off, come up with a repair plan.

Good luck.
 
I just want a functional system again. If I have that, I have access to tape and can reconfigure anything because I'll have access to /etc and /usr/local/etc/ files.

I've been using a secondary machine to test boot media, but it's too dissimilar (already on 12.2) to do much else with.

Plan is to get installation media working, install 11.4 and then try to get back to where we were before. User data should be safe. The hardware appears to be OK.
 
Whew, booting from the USB stick worked. I'm not talking to a rock any more.

Not sure I could create a test machine with the appropriate environment if I tried, because I'd have to make it broken and I'm not sure why it broke in the first place.

I'm thinking the re-install is the best bet, given that I have no idea which files are good and which are bad. And I never got to the step where I update all the packages, so they are all primed for 11.4.
 
noodlefling said:
Not sure I could create a test machine with the appropriate environment if I tried, because I'd have to make it broken and I'm not sure why it broke in the first place.
Click to expand...
I meant an environment as close as possible - i.e. 11.4 upgraded to first 12.0 boot. See how to get that back to 11.4. At least if you botch that test machine up, you can try again. Botch it up on the server, and you'll just have made your life harder.
 
Hmm. Yeah, so, it's scary again.

I booted with USB to 11.4. I started the install and could see all my partitions, but I'm not totally clear which is which. Which one is /usr, /var, and such. I can probably work it out, but that leads me to my second point which is that I ONLY want to blast / and maybe /usr.

Not sure I can get away with that with the install. I don't really want to blast user data if it's not necessary, even if there is a tape backup.

Any suggestions for simplicity? If I could totally wipe / and /usr, I'd be happy to pick up the pieces later. I've always hated FreeBSD's install shell, but since I have to do it so infrequently, it never bothered me much.
 
Maybe the ideal scenario is to install / in a new place, although all the space is taken. The user partition is large, but I'd have to resize it.

Looks like the only way to do all this is with the manual partitioning and that's pretty harsh to do under pressure while panicked. Easy to break things in there.
 
I thought you would boot to a shell and mount the old drives and check their heatlh - see if you can copy anything off?

If you can see them then maybe you can copy the OS from the boot media to the partitions (if they are healthy).

But yes, I wouldn't try any of this on a live server - try on a test machine.

I wouldn't run the installer - because of what you are seeing - but if you can find the /usr partition on the drive and mount r/w then you can "untar" the usr install set.
 
e.g. to restore /usr

forums.freebsd.org

Solved - Deleted /usr folder

Hi everybody! I'm new in FreeBSD world (always used Linux) and I'm really enjoying it :) I was playing with some configurations when I deleted part of /usr folder. Now I don't have anymore commands like tar, xz, cut, uname..... How can I restore them? I tried to look in the packages but I wasn't...
forums.freebsd.org forums.freebsd.org

Caution - there be dragons here (have I mentioned trying on a test system first?)

Anyway, that's my 2c, good luck.
 
OK, I'm in the shell. Having a brain fart because of the panic, but trying to keep a lock on it.

I guess mount -a doesn't work because the /etc/fstab is not on the stick.

It's not so much about rescuing the data on the drives, I have the tape. I just want a functional server again so it can do the userland stuff in the morning.

If the necessary files exist on the stick to make the system bootable again (without every command responding with "invalid file format", I'm happy to nurse it back to health in some boring brute force manner. Stuck on a lump right now. Doing some remedial reading that I haven't needed for so long, it's making me feel stupid.
 
It's a horrible feeling - trying to read/understand when your heart has sunk through your stomach (filled with butterflies) and into your boots.

Again - deep breath - what's your goal? If you know the backups are good (really? Are you 100% sure?!) then just go straight to scrubbing the disks and do a fresh install - maybe of 11.4 to start with. If that goes well, try the upgrade to 12.2, then restore your data on there. If the upgrade fails, re-install 11.4 and restore your files there. If the upgrade fails then you've got an issue with this server and 12.2 but a few months until 11.x goes EOL, so time to plan. If the upgrade works then you've done the job. But understandbly might just want to get back to solid 11.4 ground.

Every time I've thought my backups are good and I've done a re-install I've missed SOMETHING - some config, special port/package, cron job etc. so ... maybe don't trust yourself on the backups bit yet?

I've manually mounted drives from boot media before but I do it so infrequently I can't give you a useful step-by-step. I'm not being obtuse or withholding helpful instructions - I just know they are there and work - but don't have them "front of mind".

If UFS and you didn't do any exotic disk set-up, then it won't be a lot to do. Just keep breathing and reading.
 
Here's gpt show from a took-the-defaults MBR/BIOS install:
Code: 
% gpart show
=>        40  7812939696  da0  GPT  (3.6T)
          40        1024    1  freebsd-boot  (512K)
        1064  7803501568    2  freebsd-ufs  (3.6T)
  7803502632     8388608    3  freebsd-swap  (4.0G)
  7811891240     1048496       - free -  (512M)
Do you know the device name e.g. da0 or ada0 or mfi0 or mrsas0 etc?
 
Had a possibly good idea in using the manual installation and only telling it about / and /usr as partitions. Hopefully it will not blast the others for fun. If that works, it should have a new / and /usr and everything else should be unmolested. Then I can recover /etc from tape and hopefully be close-ish to home.

Hope this works!

(Sorry to leave you hanging, it takes forever to reboot and I've chewed through my last metaphorical fingernail!)
 
Going down the rabbit hole here...

Now it says something to the effect of

gptboot no boot loader

This is not verbatim, but it's rebooting and I'm obviously not on that machine.

Am rebooting to USB stick again now (it takes a while!), so I will try the command above, although I'm not so sure it will be as useful as it might have been a few minutes ago.

Oh, and I guess I'm embarrassed to say that I don't know how to mount the corrupt system to /mnt. mount -a did no magic and I hit a learning wall, so I started down a different path.
 
OK, I ignored the gptboot thing, remembered that I always had a fight with the installer when getting partitions the way I wanted them, and now I'm up and running with a fresh / and a fresh /usr.

I added the user filesystem to /etc/fstab and it was right there, with all the data safe and sound.

So now I just need to pull the config data off the tape and hope the other filesystems survived as well, although since they are all backed up too, hopefully that's not a big deal.

Soon I'll be out of the server room and back in a comfortable office and won't be climbing the walls.

Thanks to everyone who helped out! Just having a human to talk to and bounce ideas off is very helpful, although I hope never to have to abuse this forum in quite this way again.

On the bright side, if I get things happy again, I will take SirDice's advice and go straight to 12.2, which will save me some time.
 
noodlefling said:
Going down the rabbit hole here...

Now it says something to the effect of

gptboot no boot loader

This is not verbatim, but it's rebooting and I'm obviously not on that machine.

Am rebooting to USB stick again now (it takes a while!), so I will try the command above, although I'm not so sure it will be as useful as it might have been a few minutes ago.

Oh, and I guess I'm embarrassed to say that I don't know how to mount the corrupt system to /mnt. mount -a did no magic and I hit a learning wall, so I started down a different path.
Click to expand...

You first need to know the disk device(s) of the corrupt system. They will show up with gpart show. Their exact nomenclature will depend on whether they're old bsdlabel, GPT etc.

Then once found, let's assume the boot disk is da0s1a, you mount /dev/da0s1a /mnt
 
You must log in or register to reply here.
Back
Top