Recovering deleted files from zfs pool.

GlitchyDot

GlitchyDot

Hi, curious - is there a way to recover deleted/disappeared files from zfs pool ?
I know its not FreeBSD related but its with other OS under zfs ( Proxmox )
Long story short.
I have proxmox .. have usb drive passed to samba for jellyfin ( cockpit container with shared folder/pool/data)
was watching jelly last night, all good. paused on one tv show and went to bed - woke up, tried to resume watching and nothing happens. Jumped to cockpit - i see only one directory and one file left so i thought my hard drive started to play games with me - rebooted containers - same, rebooted host - same , 1 directory and 1 file. Unmounted zpool - re-mounted and thought it might help ... nope , same 1 and 1.
Checked logs - cant understand but few places noted - could not mount directory from my zpool.
Checking proxmox - and my zpool shows 3.99TB - 3.99TB FREE and almost 100MB allocated.
P.s. I think my HDD pool originally was created in FreeBSD and then i used it as import/export between hosts ( prox, laptopt and my FreeBSD VM under prox )
Can it be that my host OS or one of the containers actually deleted my whole drive ?
Im checking cockpit and one file which is left - drwxrwxrwx permissions. Basically anyone could delete/write/execute anywhere on that drive.
No, i dont have ghosts in my apartment and i dont do sleep walking ( as far as i know )
 
In general: no, there is no way to recover deleted files. Deleted is gone.
(there are some exceptions - there exists programs that can scan "hard disk space" for specific file types like images (digital photos) and so on. But a generic "recover all deleted files" - no.)
And that is the reason why you should have backups - verified backups - of all data that you care for.
 
tingo said:
In general: no, there is no way to recover deleted files. Deleted is gone.
(there are some exceptions - there exists programs that can scan "hard disk space" for specific file types like images (digital photos) and so on. But a generic "recover all deleted files" - no.)
And that is the reason why you should have backups - verified backups - of all data that you care for.
Click to expand...
Well that hard drive was my back up drive with extension for music and movies.
Oh well... time to re-think life then ...
 
First: kudo's for being honest about all this (referring to Proxmox). Not gonna lie either: part of me is saying.. well, you've picked a free yet still commercial product. Go ask them? Now: I don't mean this as criticism (!) but you need to keep in mind that there might be options within Proxmox that people like me are not familiar with.

Of course I'm also convinced you had your own reasons for asking this community and I'm happy to try and help.

Unfortunately though...

Tingo already said as much, the main problem here is that ZFS is what I'd like to call an "active filesystem". Meaning that it usually does way more than just storing and retrieving data. If you delete something on, say, FAT32 or UFS then there's a good chance that the deletion process only consisted of marking the "dataslots" as 'usable' again. In other words: only the "index" of the filesystem got updated, and that was it. Meaning that data is retrievable.

ZFS otoh works way differently. For starters: it's build on a whole different concept: one pool => many filesystems. So the moment stuff gets deleted... the spare data gets used up again in no time. And not just for data, also "system data": ZFS constantly tries to keep up with data integrity as well and it uses plenty of temp data for this.

I'm really sorry to say this but... Tingo is right, when data gets trashed on ZFS....
 
Lesson learned !
Im still baffled on to why and how it happened but i think its the way i set up/mounted my drive and configured Jellyfin ... still trying to get down to it in reddit but im not sure ill get something out of it...
Now im trying to set up Jellyfin on FreeBSD and forget containers in Proxmox. Its only reason i went to Jelly/Cockpit so i could access my everything from multiple pc`s ( laptop with FreeBSD and FreeBSD VM within Proxmox ) ... and not to export/import every time i change hardware :).
 
If you've created snapshots, you can rollback to the snapshot using zfs rollback. See zfs-snapshot(8) and zfs-rollback(8) for details.

As you said that the affected pool is a backup. Are the original data in another ZFS pool? If so, the easiest way for backing up would be using zfs send and zfs recv. See zfs-send(8) and zfs-recv(8) for details.

And another way if you have snapshot is discussed here.

Note that snapshots are NOT destroyed automatically and all snapshots consumes the actual total size of added/modified files and metadata until next snapshot or running state and doesn't free up the size of deleted files.
This is why zfs rollback can restore the actual state of the dataset when the snapshot is created, including deleted files and the reason that already-unneeded snapshots should be destroyed manually (or using specific 3rd party tools to do so).
 
Ok, i think "Backup" was a bit differently understood.
I have config, work data on my pc and just in case of data loss or anything else - i save them to my 4TB Hard drive, this is how i use backup but also i use that drive for music/videos ( have 2TB of it ) and i wanted to watch some movies while working ...
but i dont do snapshot fot this type of files, im kinda forgetting and just copy directories to my drive
 
T-Aoki said:
Exactly. But snapshot eases backups by send | recv to pools in other computer or another physical media. This should be the large benefit of ZFS.
Click to expand...
Snapshots are essential for consistent backups without stopping the system. UFS has snapshots for that use, too. Dump was modified to make snapshots from file systems for that reason. Having write only snapshots should reduce the time for "restore" when your files got accidently encrypted. You know, when some user accidentally clicks on that email? Does any backup software contain metrics of change ratios and starts to bleat alarm when the ratio suddenly spikes, so you don't backup (maybe even overwriting old correct ones?) encrypted data? Having the older files in an immutable state on fast access is great then. For me as a home user, that seems a bit too much. But for a company I would require this from the file server.
 
Crivens said:
Dump was modified to make snapshots from file systems for that reason.
Click to expand...
In my opinion dump is the one main thing which UFS holds over ZFS, and I'm really missing out on it.

Simple reasoning: with restore you can browse through the whole archive and only restore parts of it. Can't do as much with ZFS: the backup has to become part of the pool itself. Also keep in mind that this UFS snapshot could easily be located on a remote SMB or NFS mount and thus not even part of the main system. Still no issue for restoration but ZFS doesn't like that.

Meh... sometimes I wonder... maybe I should try and do somethign drastic: like building zfsdump & -restore.
 
ShelLuser said:
Meh... sometimes I wonder... maybe I should try and do somethign
Click to expand...
Adding a TUI to the zfs receive would be enough, I think. Or the possibility to dump the fs tree and take a filename list for receiving (making the TUI a separate thing). I'd start looking at the format for the saved data, if there is any link to "inside" blocks or if it is completely block oriented.
 
Well, turns out i made a huge cock-up ! I had jellyfin`s cache directory set as my hard drive`s pool ( pool /hardas ) which means once Jelly finishes doing things it removes cache as cache was /hardas and thats how i lost all my data but i could have been protected if i was not playing around and set 777 permission for whole pool /hardas for time being as i needed to do some things ( still noob with permissions ) but thats how i learn.
Hard and very very painful lesson.
Still have some important files covered in my usb ( thank god for my extra paranoia if my hdd goes tits up ) bbut rest of it ...
Thanks to everyone for advice !
 
tingo said:
In general: no, there is no way to recover deleted files. Deleted is gone.
(there are some exceptions - there exists programs that can scan "hard disk space" for specific file types like images (digital photos) and so on. But a generic "recover all deleted files" - no.)
And that is the reason why you should have backups - verified backups - of all data that you care for.
Click to expand...
do you know name of these programs as i had photos and music there ... without my music i can live but photos - 15 years of it... i still have not done anything to my hdd still in original zpool and i didnt destroyed it just hopping maybe a miracle happens.
 
they work if the files are not fragmented. they just scan the header find the data size, read n blocks, save to another place, repeat.they have/need no knolledge about the underlying file system
 
magicrescue seem not to be compatible with freebsd without some patching, i wonder if the porter ever tested it
it tries to read sizes that are not multiple of the blocksize from a block device which is unsupported
 
ShelLuser said:
Simple reasoning: with restore you can browse through the whole archive and only restore parts of it. Can't do as much with ZFS
Click to expand...
Not quite so, if I'm understanding you correctly. In the simplest case, one can rsync the desired files or directories from the .zfs/snapshot/.../whatever/path/you/want to any destination, just like any standard rsync operation. If read-only access to a file system is sufficient, a snapshot can be mounted (either temporarily or routinely) and accessed like any other file system. And thirdly, if read-write access is required, a snapshot can be cloned into an entirely new filesystem without having to roll back or modify the original snapshotted filesystem at all.

ZFS offers at least those three modes of data restoration if you have viable snapshots of a damaged or compromised filesystem.
 
You must log in or register to reply here.
Back
Top