UFS GPT: secondary header gets rewritten after reboot

Chris_H

Chris_H

OK I'm tracking CURRENT, where I have this issue. But I don't think it has anything to do with the fact that it's CURRENT. As writing the headers/tables on an 11 box, results in the same error after rebooting with the drive in the CURRENT box.
Specifics;
Only occurs on USB connected drives -- flash, or pata / sata (with adapter)
[man=8]gpart[/man] destroy -F da0
[man=8]gpart[/man] create -s GPT da0
[man=8]gpart[/man] add -t freebsd-ufs -l external da0
[man=8]newfs[/man] -U -o time /dev/gpt/external
reboot with drive still attached, but not mounted. Results in the following error during the final boot stage:
Code: 
 kernel: GEOM: da0: the secondary GPT table is corrupt or invalid.
 kernel: GEOM: da0: using the primary only -- recovery suggested.
 kernel: GEOM: new disk ada0
So I simply issue a [man=8]gpart[/man] recover da0. Which returns fine.
But the problem is, why or how is / does this happen? Or perhaps better; how can I make this stop?
I mean it's fine, as long as the primary header/table is good. But if it ever has a problem, I'm screwed.

Thank you for all your time, and consideration.

--Chris
 
Try pinging the freebsd-current mailing lists. It may just have been missed.
 
Good idea, SirDice . Thanks for the suggestion.
I did address this on the mailing list, and Andrey V. Elsukov suggested that it might well be the BIOS. Here's the dialog:
Code: 
On 01.04.2017 00:58, Chris H wrote:
> > So. I spin up an old 11 server I have sitting in the closet, with
> > this external drive attached to it. I do *NOT* get the corrupt GPT
> > message. So I blank/partition/newfs the external drive &&
> > mount the partitions individually to /mnt && restore again. When I
> > reboot to the external drive still connected to the old 11 server,
> > I do *NOT* receive the corrupt GPT message. WooHoo! I think.
> > So I re-attach the drive to the new 12 server. Reboot, and can't
> > boot to it && get the corrupt GPT message.
> >
> > GEOM seems to be broken in 12, maybe even (recent) 11. As the 11
> > server I used for testing is ~9 mos out.
> >
> > What can I do to (help?) fix this mess?
>
> Just a guess, BIOS on the system, where FreeBSD 12 is installed
> overwrites the last sector of your disks.
> I have seen such reports, and always this was the cause.
>
> You can do the following steps to make sure:
> * on the old 11 system with the sane GPT save the last sector to some file.
> * reboot, save the sector again to another file and compare both files.
> * attach the disk to your 12 system, GPT should become corrupted. Save
> the last sector and compare with previous file.
>
> You can look at the hexdump of this file, and probably it should be
> obviously what is extraneous in the data.
>
> To save the last sector you need to know its number, it can be found by
> this command:
>
>  # diskinfo da0 | awk '{print $4-1}'
>
> Then use dd to save it:
>
>  # dd if=/dev/da0 of=./sector skip='diskinfo da0 | awk '{print $4-1}''
>  # hexdump -C ./sector
>
> You should see something like this:
> 00000000  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI
> PART....\...|
    ...
> *
> 00000200
>
> The dump of correct GPT header should not have more lines.
>
Andrey, Thank you!

OK I'm having trouble with the concept. But *indeed* the
output indicates *always* good on the 11 server (confirmed
following your steps above).
Moving it to the new 12 server, returns corrupt secondary GPT
table message && hexdump output is:

00000000  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000010  65 12 5c 16 00 00 00 00  2f 60 38 3a 00 00 00 00  |e.\...../'8:....|
00000020  01 00 00 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
00000030  07 60 38 3a 00 00 00 00  91 e5 f5 c1 0d 16 e7 11  |.'8:............|
00000040  8d 49 00 24 81 ce ba 87  08 60 38 3a 00 00 00 00  |.I.$.....'8:....|
00000050  80 00 00 00 80 00 00 00  00 00 00 00 86 da fa 98  |................|
00000060  61 66 13 80 09 fe d0 54  35 59 db 8e 43 b8 7e 37  |af.....T5Y..C.~7|
00000070  c9 77 0e 9d 35 fd 45 04  de 9a d3 ff 30 83 8f b4  |.w..5.E.....0...|
00000080  b9 84 1d 41 59 44 ef fd  fd 89 3e 1e 9e c6 23 e1  |...AYD....>...#.|
00000090  83 17 a7 53 e1 e7 51 c8  5f 87 2b 76 f8 60 c4 ca  |...S..Q._.+v.'..|
000000a0  e2 3e 1e eb 12 69 12 32  33 c3 29 42 d6 aa 1a bc  |.>...i.23.)B....|
000000b0  90 af fc 4f d0 e1 58 c3  52 f5 5c 54 ca bd 05 8c  |...O..X.R.\T....|
000000c0  89 04 8d 7b 11 a3 b2 1e  07 6e fe 1b 79 00 c0 15  |...{.....n..y...|
000000d0  1a 39 79 28 91 a3 e8 24  93 1a 35 ef e9 f8 e5 17  |.9y(...$..5.....|
000000e0  e6 93 f1 a2 5d aa 3e 2f  40 dc b3 17 19 4c f6 05  |....].>/@....L..|
000000f0  cf 75 3e 88 ad a4 2a 68  8c 04 c4 99 a1 bb a2 1c  |.u>...*h........|
00000100  9c 8d fe c7 3e e4 cb 56  ce 3d 33 5b 28 a5 c9 45  |....>..V.=3[(..E|
00000110  c7 3f aa e2 1e 98 bc e2  6d 9d 91 12 84 24 d6 13  |.?......m....$..|
00000120  3d b5 14 bd 9a 44 e9 ee  3f b5 91 31 73 86 79 7e  |=....D..?..1s.y~|
00000130  09 bd 4e 01 cb 06 81 b4  41 11 cd cf 97 dd 97 a1  |..N.....A.......|
00000140  a7 73 e5 f7 c5 a4 75 c9  1f 6b 5e 88 fe 1a 92 d2  |.s....u..k^.....|
00000150  3a cc 70 21 1f b8 30 34  b9 0e 5c b2 d0 14 5e 82  |:.p!..04..\...^.|
00000160  56 60 04 35 77 c9 25 04  7a af ce e1 8d 24 37 53  |V'.5w.%.z....$7S|
00000170  a3 0c dd 63 3c 15 fe 9f  a4 46 00 97 c1 b0 27 be  |...c<....F....'.|
00000180  f5 c7 f9 b5 71 9e 1b 90  f7 9c ee 8a 8e 7b 77 61  |....q........{wa|
00000190  23 13 4a 93 0b e0 f0 9e  3f dc 8e 12 f9 19 d3 75  |#.J.....?......u|
000001a0  f2 52 6d bd 12 30 cd bf  0c 91 79 10 1a bd 5b d4  |.Rm..0....y...[.|
000001b0  0f 9c 1b ff 7b 60 74 79  d7 fa bb 02 6f 19 be e4  |....{'ty....o...|
000001c0  06 fd f4 7c cb 05 23 eb  89 2f 7f cc 9b 01 fa f7  |...|..#../......|
000001d0  4c 07 c4 72 55 9f 3d 39  f3 71 64 94 bf 7e 74 b0  |L..rU.=9.qd..~t.|
000001e0  49 80 c1 37 4f 49 91 e0  54 a7 e5 4d 83 8f b8 32  |I..7OI..T..M...2|
000001f0  62 f2 61 50 6f f2 16 05  a4 60 2f 06 be 45 a6 72  |b.aPo....'/..E.r|
00000200

gpart recover da0 && hexdump output from newly captured last
sector:

00000000  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000010  aa 9c 5f 36 00 00 00 00  2f 60 38 3a 00 00 00 00  |.._6..../'8:....|
00000020  01 00 00 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
00000030  07 60 38 3a 00 00 00 00  91 e5 f5 c1 0d 16 e7 11  |.'8:............|
00000040  8d 49 00 24 81 ce ba 87  09 60 38 3a 00 00 00 00  |.I.$.....'8:....|
00000050  80 00 00 00 80 00 00 00  65 12 5c 16 00 00 00 00  |........e.\.....|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

My problem with your theory; 3 reasons I question it
1) The internal SATA3 disk never experiences this
2) When I first plugged in the external drive, it had ghostbsd
on it, and the server attempted to boot it w/o error. It wasn't
until I blanked/partitioned/newfs'd it, that the trouble(s) manifest
3) This server is brand new hardware and the *only* media that has
ever touched this prior to the anomaly was the FreeBSD install DVD
(I'm trying to eliminate the possibility of VIRII here).

Can you please elaborate on how/why the BIOS should/would modify
boot media && only USB boot media?

Thank you again, Andrey, for all your input on this!
So. While it appears he may indeed be correct. What the h*ll is one to do -- rewrite the BIOS?
I've done that in the past. But why would the BIOS intentionally do this, in the first place? Does it simply come with a VIRUS pre-installed? Or is it the NSA dropping a backdoor. :rolleyes:

Thanks again for the reply, SirDice !

--Chris
 
Good call, SirDice !
Yes. Right after concluding that Andrey appeared to be on to something. I applied the only updates they offered. None of which addressed the issue. :(
I hope this isn't going to turn into an RMA. Any suggestions for a BIOS editor?

Thanks again, SirDice !

--Chris
 
There are many BIOSes, even from respected manufacturers, that think they have liberty to modify partition tables if they don't like the content. One example we experienced a few years ago, when UEFI and GPT was young: We had a BIOS that decided that if the master copy of the GPT (at the beginning of the disk) fails checksum verification, it would replace it with the backup copy (from the end of the disk), without first checking that the backup copy had valid GPT headers or a valid checksum. Unfortunately, this was for an embedded system that uses its own header format, so our headers were consistently being clobbered by whatever valid data was stored at the end of the disk. We ended up (a) screaming at the BIOS engineering team, (b) getting it fixed so GPTs are only overwritten under user control, (c) changing our software to put valid GPTs at both beginning and end of the disk, and shifting our data accordingly.

More recent example, from this week: Major Linux distro upgrade tool sees what it thinks is a GPT but that's stored at the beginning of a partition (rather than at the beginning of a disk), kindly decides that it is known problem with an older file system (not one that murders the wife, but of a similar age), and overwrites it with zeroes. Oops, our data is gone.

If GPTs vanish, I would first suspect either the (UEFI-) BIOS or an OS install tool.
 
Hello ralphbsz, and thanks for the reply.
I spent most of yesterday determining what, and when, in an effort to pinpoint the culprit. If I gpart(8), and newfs(8) any USB attached drive, and verify the status gpart status da0. Then (while unmounted) issue a halt -p. Disconnect the drive. Fire up the box, and plug the drive in. Everything is fine (gpart status OK). If I leave the drive plugged in and reboot, gpart(8) echoes the warning regarding the inconsistent secondary header. The same is true, if I plug in any GPT partitioned drive in prior to firing up the box -- even if the drive was partitioned on another box. SO either FreeBSD is wrting to the drive during post/bootstrap (when control is passed to the drive) or, more likely the BIOS sees the need to write to the drive, perhaps thinking that the end of the drive is a safe/unused place. Safe/Unused seems unlikely, as the Manufacturer indicates GPT as supported. Hell, even Windows versions post 8 use/understand GPT.
I spent the day (attempting) installation of every incarnation of BSD I could imagine compitable with 6mos. old hardware. As well as many incarnations of Linux. Linux seemed the most resilient to the problem. BSD won't boot. In fact, the BIOS won't even recognize the drive as bootable after a successful installation.
Many a BIOS manufacturer has offered the option to check/prevent a VIRUS attack on the drive (an option I have always disabled in the BIOS). Usually using maintaining a hash of the boot sector. Their methods of maintaining that data varies -- from keeping the info internally, to actually writing the data to the drive, itself. The latter being pretty foolhardy IMHO.
Anyway, today, I'm going to experiment with MBR partitioning. Which saves it's backup in a different location, and see what kind of results I get. Given that the MBR partitioning doesn't clobber the secondary GPT header. I'm going to need to figure out the math to dd(1) those last bytes off the disk. I don't suppose you have a method handy? :-)

Thanks for the reply, ralphbsz !

--Chris
 
So you wasted the whole day because of some BIOS? That's sad.

Here's my suggestion: Find a small hammer and a chisel. Open the computer, and find the BIOS on the motherboard. Using the chisel, cut out that part of the PC board (safer to cut a few extra inches around it too), and throw it in the trash. Close the computer again. From now on any disk connected to this motherboard will not be overwritten. Promise!

If you were a very large customer of this board vendor, you should contact them, and tell them to stop this nonsense. I've had good luck with this technique, but it only works if you can convince them that they will lose many millions of revenue. For individual BIOSes, this doesn't work.

Here is my only constructive suggestion: Why is the BIOS copying things around? Probably because it thinks that one of the two copies of the GPT has become invalid, while the other one is valid. Now in my opinion, BIOSes shouldn't do that without asking for permission first (because there may be good reasons for not having one of the GPTs), but my opinion is usually not relevant. Could it be that the BIOS is right? Here's my suggestion: The GPT format is really well documented; even the wiki page describing it is accurate enough to go by. Do the following: Using dd and hex dump, check the two copies of the GPT. If you feel like it, it would only take half hour to write a script in your favorite language that generates a nicely formatted dump and verifies the CRC.

You ask: The math to dd the last few bytes? Let's assume your disks are (S-)ATA. Start by verifying the capacity and logical block size of the disk, with camcontrol identify /dev/adaXX. Check that the logical block size (a.k.a. sector size) is really 512, and write down the capacity of the disk (in sectors), which is in the line that starts with "LBA48". Say it is 5860533168 sectors (that's what I get on the first disk I just checked).

Reading the first sector (the header) is easy: dd if=/dev/adaXX of=/tmp/GPT_BEGINNING bs=512 count=1 skip=1. Then you can hex dump that file and compare it to the documented GPT format. The last sector requires a tiny bit of math: dd if=/dev/adaXX of=/tmp/GPT_BEGINNING bs=512 count=1 skip=5860533167 (the number of sectors to skip is capacity-1, to get the last sector). Again, decode it by hand. The two copies should be mostly identical, except they have different checksums, and the locations of the current and backup header are switched. To then see the partition tables for the first two partitions: At the beginning of the disk, use "skip=2", and at the end of the disk, instead of subtracting 1 from the capacity, you subtract 32, and so on.

EDITed to add: Don't forget to step on the piece of the PC board with the broken BIOS, to make sure it's really broken. Old musician joke: What's the difference between a trampoline and an oboe? You take your shoes off before you step on a trampoline.
 
Thanks for the delightful reply, ralphbsz ! My sentiments exactly! :)
OK I did contact the vendor (motherboard) and indicated I didn't appreciate their version of the AMI BIOS dropping a trojan , or backdoor on all of my USB drives. But said that I would understand if the NSA required them to do it. But that in either case, I was on to it, and would like to know if they had a solution. Promising I'd keep my mouth shut, if they came through.

Good Idea regarding the chisel, and hammer. But at this point, if I get my hands on a hammer, the BIOS won't be the only thing that gets it!

About the BIOS checking both copies of the headers;
I'm not inclined to trust the BIOS to do so; history has revealed (to me, anyway) that some of them simply make a hash of the primary, and write it to disk. For later comparison. An act I am completely against. They can't presume to know what, and where the data I'm writing is, or where it belongs. So how can they possibly know where a "safe" place could be? Further; if the BIOS becomes infected, how easy can it be to write trojans, backdoors, loaders, and other goodies to the media. In my not-so humble opinion; the BIOS has no business touching anything; it can look, but not touch -- ever.

If I don't get an RMA for this, I'm simply going to inform them that I write for several major operating systems, and having squandered 2 full days getting to the bottom of this. That I would need to justify this to the bean counters, and that they would likely deduce (as they should) that their product was just too expensive to justify, at any cost. That should help get the ball rolling. :cool:

As to the math;
a bit of research seemed to indicate that I can glean, or destroy the secondary header with some fairly easy math. Grabbing the last 34 blocks/sectors gives me what I need, and fortunately diskinfo(8) gives me the exact number to work against;
fe;
Code: 
diskinfo -v da0
da0
        512             # sectorsize
        1000204883968   # mediasize in bytes (932G)
        1953525164      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        121601          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        20140512005338F # Disk ident.
So, given the results above; I simply subtract 34 from 1953525164 giving me 1953525130.
This works for the seek= value, using dd(1) to clobber it, or writing it, for later inspection. :-)
All of which pretty much corresponds with what you also offered -- thanks!
Applying that math, and comparing the results immediately after creating a GPT partition scheme. Reveals that they are both identical. Naturally, after rebooting, this is no longer the case. :mad:
So, I've found that until this situation is resolved that the only way I can get any *BSD onto a USB attached drive, is by using an MBR scheme.
Well, time to install a copy of FreeBSD onto a USB attached drive, and see what changes on the hard drive after reboot. :rolleyes:

Thanks again for the great reply, ralphbsz ! greatly appreciated!

--Chris
 
I fear your conclusion is right: Given what the hardware does, MBR is the only thing that will work.

OK, I have another COMPLETELY impractical idea. If your disk were a self-encrypting drive (many are today, the encryption capability is shipped by default from many vendors), you could tell the drive to segregate it's LBA address space into multiple bands, which are encrypted and protected separately. Create three bands, one tiny for the first copy of the GPT table (at the beginning of the drive), another tiny for the backup copy (at the end), and finally a big one for all the rest (the meat in this particular burger). Then you could set the encryption keys and permissions such that the two outer bands are readable but not writable. Fundamentally what this amounts to is write-protecting the area containing the GPTs.

The drawbacks are obvious: figuring out how to do this would take a serious expert a week or two. And once you have done that, any attempt of changing the partition layout would require changing the bands to be writeable (and changing bands usually destroys the data in the bands, so you have to copy the data out, change the band, write it back). And making the drive "operable" at all would require sending encryption keys to it, which you would have to do not just before mounting it, but before the OS can access it and get the partition tables. You'd have to store the encryption keys somewhere (and they tend to be long and complex enough, a yellow sticky note on the laptop won't be sufficient). Working with self-encrypted drives is way too much fun, because it's a good way of finding OS limitations and bugs.
 
You must log in or register to reply here.
Back
Top