UFS FreeBSD 11.0 Release inumber .. on /var: filesystem full

M

Mealerz

Hi everyone,

I'm puzzled on an issue, which I've had before but this time I am really puzzled.
I have upgraded my FreeBSD 10.3 machine to 11 and everything seems to be working perfectly. However I am seeing lots of this error in /var/log/messages.
Code: 
inumber .. on /var: filesystem full
I have a bunch of processes reporting this.

I installed 10.3 using the download and used entire disk when installing just on /. I also added a secondary drive and mounted that to /var, and a 3rd drive and mounted that to /data. I am running quotas on /var and my fstab looks like this -
Code: 
# Device    Mountpoint    FStype    Options    Dump    Pass#
/dev/da0p2    /        ufs    rw    1    1
/dev/da0p3    none        swap    sw    0    0
/dev/da1p1    /var        ufs    rw,userquota    2    2
/dev/da2p1    /data        ufs    rw    2    2
fdesc        /dev/fd        fdescfs    rw    0    0
tmpfs        /tmpfs        tmpfs    rw,size=4G,uid=106,gid=106,mode=1755    0    0
tmpfs        /tmp        tmpfs    rw,nosuid,noexec,mode=01777    0    0
Please see df -h -
Code: 
Filesystem    Size    Used   Avail Capacity  Mounted on
/dev/da0p2     37G    7.3G     27G    22%    /
devfs         1.0K    1.0K      0B   100%    /dev
/dev/da1p1    291G    180G     87G    67%    /var
/dev/da2p1     19G    192M     18G     1%    /data
fdescfs       1.0K    1.0K      0B   100%    /dev/fd
tmpfs         4.0G    4.0K    4.0G     0%    /tmpfs
tmpfs         1.9G     42M    1.8G     2%    /tmp

I have tried booting the machine into single user mode option 2 and of course I see just the root partition. I went into /var and there really isn't much going on. I see nothing large in that directory. I thought there would be a huge log file but really it's fine. I also ran [/cmd]fsck -y /var[/cmd] and still nothing in that directory and no errors - certainly there are no large files and df -h reports the same space. I then mounted /dev/da1p1 to /var and ran fsck(8) again, but when I try this it says NO WRITE. df -h shows the same as above.

Since I've rebooted, I can't see how a process has some log file open or anything. Also the issue seems to be cosmetic, since the server is working perfectly fine.

I have this hosted on vmware, so no real tools installed, just the tools via ports.

Is this a bug with the new release? It's very new of course. I don't see anyone else posting this so I thought I might report it.

Any tips or advice would be welcome.

Thanks.
 
The file sytem could be full because of too many small files. Then no new space is available, even if the total size is not exceeded. Iam not sure about the relation to the quotas. At least I hope this is a useful pointer.
 
Hi,
Sorry you are correct, I should not have mounted the disk. However, this is still happening even after I try your suggestion. I have now rebooted into single user mode, I ran fsck -f /dev/dap1p and it found a few changes, but the space is the same.
I've pressed exit, I'm now back in and receiving the same error messages. Again my system seems to be running fine. It's processing email.

I feel like the problem is kernel related, since I see no problem with the machine other that it reports this error. It's like the kernel doesn't know what disk I'm using? I'm really not sure.

I have this set up as a POP mail server and I have qpopper and procmail both reporting this error, yet everything is working absolutely fine.
Code: 
 df -h
Filesystem    Size    Used   Avail Capacity  Mounted on
/dev/da0p2     37G    7.3G     27G    22%    /
devfs         1.0K    1.0K      0B   100%    /dev
/dev/da1p1    291G    180G     87G    67%    /var
/dev/da2p1     19G    192M     18G     1%    /data
fdescfs       1.0K    1.0K      0B   100%    /dev/fd
tmpfs         4.0G    4.0K    4.0G     0%    /tmpfs
tmpfs         4.1G     64K    4.1G     0%    /tmp
Code: 
cat - /var/log/messages

Oct 20 03:14:36 test-v1 kernel: pid 16888 (qpopper), uid 21880 inumber 15261650 on /var: filesystem full
Oct 20 03:14:41 test-v1 kernel: pid 17279 (procmail), uid 11690 inumber 15252133 on /var: filesystem full
Oct 20 03:16:06 test-v1 saslauthd[606]: do_request      : NULL password received
Oct 20 03:16:54 test-v1 kernel: pid 18229 (procmail), uid 11690 inumber 15252133 on /var: filesystem full
Oct 20 03:17:31 test-v1 saslauthd[606]: do_request      : NULL password received
Oct 20 03:19:39 test-v1 saslauthd[611]: do_request      : NULL password received
Oct 20 03:19:46 test-v1 kernel: pid 19198 (qpopper), uid 21880 inumber 15261650 on /var: filesystem full
Oct 20 03:19:56 test-v1 saslauthd[606]: do_request      : NULL password received

Do you have any other suggestions?

Thanks,
Rich
 
You ran out of inodes, you can check with df -i. The simple explanation is that each file uses an inode, when you have a large amount of small files you can run out of inodes long before you run out of disk space. The only way to get more inodes on a filesystem is with newfs(8) unfortunately. So, you need to save whatever you want saved from /var, unmount it and run newfs(8) on it.
 
Have I?

Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on
/dev/da0p2 38577212 7632440 27858596 22% 501580 4554546 10% /
devfs 1 1 0 100% 0 0 100% /dev
/dev/da1p1 304687800 188486472 91826304 67% 158115 39247579 0% /var
/dev/da2p1 20305148 196608 18484132 1% 372 2648074 0% /data
fdescfs 1 1 0 100% 4 234987 0% /dev/fd
tmpfs 4194304 4 4194300 0% 1 14680063 0% /tmpfs
tmpfs 1648568 64 1648504 0% 12 2147483635 0% /tmp

Should I check this in single user mode on the / partition?

I'm so confused, is this a bug?
 
Mealerz said:
Have I?

Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on
...
/dev/da1p1 304687800 188486472 91826304 67% 158115 39247579 0% /var
Click to expand...
No

Does a subsequent run of fsck -f /dev/da1p1 return no errors ?
 
Hi,

Last night at 3am I booted the machine to single user mode and ran that command. It didn't return any errors, just the normal stuff really. Some small fixes but I see the same on df -h.
I did just type 'exit' after I ran that command and I checked df -h in normal mode, rather than mounting that drive and checking it in single user mode.

I really don't believe there are any problems with the machine, apart from this error popping up a lot!

Thanks,
Rich
 
Mealerz said:
I really don't believe there are any problems with the machine, apart from this error popping up a lot!
Click to expand...
As a first step of an ongoing investigation I would ask to put aside any assumption.
"Assumptions" as opposed to "facts" lead to searches in the wrong direction. ;)

Mealerz said:
Some small fixes but I see the same on df -h.
Click to expand...
That's what we can know ... and it is not clear what "small fixes" means.

Mealerz said:
Oct 20 03:16:54 test-v1 kernel: pid 18229 (procmail), uid 11690 inumber 15252133 on /var: filesystem full
Click to expand...
So far all questions were pertinent ... based on the errors we can see, basically they all point to:
- a full /var filesystem
- possible inode table overflow
- a bug may exists ... but so far remain to be demonstrated.

Next:
what return quota -v and quota -u 11690 and quota -u 21880 ?
 
Understood. I was connected via a console to the screen when in single user mode, so there wasn't an easy way to copy / paste the output from my fsck.

quota -v
Disk quotas for user root (uid 0):
Filesystem usage quota limit grace files quota limit grace
/var 4771912 0 0 132180 0 0

quota -u 11690
Disk quotas for user iamspam (uid 11690):
Filesystem usage quota limit grace files quota limit grace
/var 1656684* 990000 1000000 none 3 0 0

quota -u 21880
Disk quotas for user fn3696n (uid 21880):
Filesystem usage quota limit grace files quota limit grace
/var 44500 1990000 2000000 4 0 0

So it does appear to be complaining about boxes that are over quota..

Thanks.
 
You are correct - it is only complaining about uids that are over quota. I didn't spot that before. Thank you so much everyone for your help!
 
You must log in or register to reply here.
Back
Top