UFS Slow real disk write results

40 IOs per second on an SSD? Something is very wrong.

I do not think it is the file system or buffer cache; that code works well on millions of other computers. To test what's going on, I would change to doing raw IO, and see whether the device itself and its OS driver are super slow. Unfortunately, that is destructive, and will overwrite the disk. One (relatively easy) option would be to make a complete image backup of the disk (while booting from another media, like a USB stick), then do the destructive test, then restore the image backup. There is risk of mistakes causing data loss.
 
weldlive said:
Thank you, but I keep repeating: I don't compare UFS to EXT3!
Click to expand...
But that is exactly what you are doing. You are comparing one FS on one OS to a different FS on a different OS -- installed on the same physical machine (with no control over where things are actually residing on that media).

Create a EXT3 FS on the FreeBSD box and repeat the test. Or, UFS on the Linux. Or, both FSs (different partitions/slices) on either OS. So you can see what consequences the FS has on performance.

E.g., NTFS and FAT32 perform differently on a thumb drive under Windows. Yet, the thumb drive hasn't changed! Nor the OS. Gee, what could be the cause for that difference??
 
Did you run fio under Linux to compare the result with o_direct?

You have very high rate of interrupts. Here's example of the server
Code: 
% vmstat -i
interrupt                          total       rate
irq9: acpi0                            1          0
irq18: ehci0 ehci1+             15829239          2
cpu0:timer                      74348892         10
cpu1:timer                      30693758          4
cpu2:timer                      35704491          5
cpu3:timer                      37338901          5
cpu4:timer                      36891724          5
cpu5:timer                      37345059          5
cpu6:timer                      36720060          5
cpu7:timer                      37988271          5
cpu8:timer                      35143815          5
cpu9:timer                      35692808          5
cpu10:timer                     35235973          5
cpu11:timer                     35730380          5
cpu12:timer                     35180004          5
cpu13:timer                     35755930          5
cpu14:timer                     36102639          5
cpu15:timer                     36155947          5
irq56: ciss0                    14543582          2
irq66: igb1:rxq0                42890055          6
irq67: igb1:rxq1                18347713          2
irq68: igb1:rxq2                19801076          3
irq69: igb1:rxq3                18517077          2
irq70: igb1:rxq4                19198544          3
irq71: igb1:rxq5                18085434          2
irq72: igb1:rxq6                19251245          3
irq73: igb1:rxq7                25038671          3
irq74: igb1:aq                         2          0
irq76: xhci0                     4772164          1
Total                          828303455        109

Can you check if the disk cache is enabled with
camcontrol identify ada0
 
cracauer@ said:
Because it hold all the results in RAM.

If you run my benchmark you get better numbers, after fsync.
Click to expand...
If it was in ram the transfer speed would be in GiB range. Most likely it uses the disk cache and the result is the cache speed on the disk over sata II (300mb/s) or sata 3 half-duplex
 
VladiBG said:
If it was in ram the transfer speed would be in GiB range.
Click to expand...

No, the VM cache is not that fast if you do through system calls.

Here is an example, everything except sequential write (frist number) is in the filesystem / VM cache in RAM:
Code: 
         ---Sequential Output----- ---Input---- ------Random-- -----Random----
         ---Block---- ---Rewrite-- ---Block---- ---ro Seeks--- -Seeks rewrite-
    MB    M/sec %CPU    M/sec %CPU   M/sec %CPU      /sec  %CPU     /sec  %CPU
  1024   127.3  13.9  1762.6  73.3   6980.8 100.1 875946.3 285.4 168490.4  71.1
 
You are using the wrong benchmark for the disk i/o. In your bonnie test it just show that your CPU is the limiting factor. Try with fio and in-memory fs
fio --name=write_throughput --directory=/mnt --numjobs=1 --size=500M --time_based --runtime=60s --ramp_time=2s --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write --group_reporting=1
Click to expand...

This is on laptop with DDR3 and old SATA SSD INTEL SSDSA1M160G2HP 2CV102M3
Code: 
# fio --name=write_throughput --directory=/root --numjobs=1 --size=500M --time_based --runtime=60s --ramp_time=2s --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write --group_reporting=1
  write: IOPS=100, BW=101MiB/s (106MB/s)(6057MiB/60021msec); 0 zone resets
Run status group 0 (all jobs):
  WRITE: bw=101MiB/s (106MB/s), 101MiB/s-101MiB/s (106MB/s-106MB/s), io=6057MiB (6351MB), run=60021-60021msec

# mount -t tmpfs tmpfs /mnt
# fio --name=write_throughput --directory=/mnt --numjobs=1 --size=500M --time_based --runtime=60s --ramp_time=2s --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write --group_reporting=1
  write: IOPS=3682, BW=3682MiB/s (3861MB/s)(216GiB/60001msec); 0 zone resets
Run status group 0 (all jobs):
  WRITE: bw=3682MiB/s (3861MB/s), 3682MiB/s-3682MiB/s (3861MB/s-3861MB/s), io=216GiB (232GB), run=60001-60001msec

# cd /root/
# bonnie -s 500
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          500 104086 31.9 97196 11.5 108928 10.5 344318 100.0 3432923 100.0 318928.4 236.9
# cd /mnt
# bonnie -s 500
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          500 319917 100.0 2486426 99.8 1867331 100.0 357113 100.0 3734119 102.6 394788.8 258.2

And the same test on another VM machine with DDR5

Code: 
# fio --name=write_throughput --directory=/mnt --numjobs=1 --size=500M --time_based --runtime=60s --ramp_time=2s --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write --group_reporting=1
  write: IOPS=12.6k, BW=12.3GiB/s (13.2GB/s)(736GiB/60001msec); 0 zone resets
Run status group 0 (all jobs):
  WRITE: bw=12.3GiB/s (13.2GB/s), 12.3GiB/s-12.3GiB/s (13.2GB/s-13.2GB/s), io=736GiB (791GB), run=60001-60001msec
 
You must log in or register to reply here.
Back
Top