I am experiencing generally slow system on my arm64 (rk3399) box - slow opening of new tmux pane, slow opening new zsh instance, slow startup of vim, slow opening files in vim. All config files are the same as on my 10 years old amd64 laptop which works as expected. Same arm64 hardware with OpenBSD also feels slow, but with Linux feels much more responsive.
Tests:
arm64 FreeBSD:
arm64 Linux:
For curiosity, here is my 10 years old laptop with 5 years old (used) 2.5" SSD:
More details about disk subsystem on slow configuration:
TL
R;
zsh startup:
arm64 FreeBSD: 1.0469 s
arm64 Linux: 0.5564 s
amd64 FreeBSD old laptop: 0.2428 s
How to debug it? How to make it more responsive?
Tests:
Code:
for i in $(seq 1 10) ; do {time zsh -i -c "sleep 0.1 ; exit"} 2>&1 ; done
for i in $(seq 1 10) ; do dd if=/dev/nvme0n1 of=/dev/null bs=1M count=4K conv=sync 2>&1 | tail -1 ; done
for i in $(seq 1 10) ; do fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1 | tail -1 ; done
for i in $(seq 1 10) ; do fio --randrepeat=1 --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4M --iodepth=256 --size=10G --readwrite=read --ramp_time=4 | tail -1 ; donearm64 FreeBSD:
Code:
running test - zsh
--------------------------------------------------------------------------------
zsh -i -c "sleep 0.1 ; exit" 0.74s user 0.31s system 91% cpu 1.144 total
zsh -i -c "sleep 0.1 ; exit" 0.75s user 0.25s system 90% cpu 1.102 total
zsh -i -c "sleep 0.1 ; exit" 0.67s user 0.32s system 90% cpu 1.098 total
zsh -i -c "sleep 0.1 ; exit" 0.48s user 0.28s system 87% cpu 0.861 total
zsh -i -c "sleep 0.1 ; exit" 0.69s user 0.31s system 90% cpu 1.099 total
zsh -i -c "sleep 0.1 ; exit" 0.65s user 0.36s system 91% cpu 1.107 total
zsh -i -c "sleep 0.1 ; exit" 0.64s user 0.24s system 89% cpu 0.973 total
zsh -i -c "sleep 0.1 ; exit" 0.64s user 0.32s system 90% cpu 1.064 total
zsh -i -c "sleep 0.1 ; exit" 0.75s user 0.25s system 90% cpu 1.107 total
zsh -i -c "sleep 0.1 ; exit" 0.55s user 0.27s system 88% cpu 0.914 total
running test - dd
--------------------------------------------------------------------------------
4294967296 bytes transferred in 10.447751 secs (411090110 bytes/sec)
4294967296 bytes transferred in 10.872883 secs (395016405 bytes/sec)
4294967296 bytes transferred in 10.884630 secs (394590117 bytes/sec)
4294967296 bytes transferred in 10.874099 secs (394972228 bytes/sec)
4294967296 bytes transferred in 10.893587 secs (394265665 bytes/sec)
4294967296 bytes transferred in 10.703562 secs (401265232 bytes/sec)
4294967296 bytes transferred in 10.881222 secs (394713684 bytes/sec)
4294967296 bytes transferred in 10.884584 secs (394591765 bytes/sec)
4294967296 bytes transferred in 10.693615 secs (401638464 bytes/sec)
4294967296 bytes transferred in 10.702620 secs (401300535 bytes/sec)
running test - fio1
--------------------------------------------------------------------------------
WRITE: bw=2931KiB/s (3001kB/s), 2931KiB/s-2931KiB/s (3001kB/s-3001kB/s), io=172MiB (180MB), run=60022-60022msec
WRITE: bw=2829KiB/s (2897kB/s), 2829KiB/s-2829KiB/s (2897kB/s-2897kB/s), io=166MiB (174MB), run=60084-60084msec
WRITE: bw=2902KiB/s (2971kB/s), 2902KiB/s-2902KiB/s (2971kB/s-2971kB/s), io=170MiB (178MB), run=60024-60024msec
WRITE: bw=2875KiB/s (2944kB/s), 2875KiB/s-2875KiB/s (2944kB/s-2944kB/s), io=168MiB (177MB), run=60009-60009msec
WRITE: bw=2915KiB/s (2985kB/s), 2915KiB/s-2915KiB/s (2985kB/s-2985kB/s), io=171MiB (179MB), run=60029-60029msec
WRITE: bw=2957KiB/s (3028kB/s), 2957KiB/s-2957KiB/s (3028kB/s-3028kB/s), io=173MiB (182MB), run=60010-60010msec
WRITE: bw=2966KiB/s (3037kB/s), 2966KiB/s-2966KiB/s (3037kB/s-3037kB/s), io=174MiB (182MB), run=60024-60024msec
WRITE: bw=2850KiB/s (2919kB/s), 2850KiB/s-2850KiB/s (2919kB/s-2919kB/s), io=167MiB (175MB), run=60020-60020msec
WRITE: bw=2857KiB/s (2925kB/s), 2857KiB/s-2857KiB/s (2925kB/s-2925kB/s), io=167MiB (176MB), run=60009-60009msec
WRITE: bw=2882KiB/s (2951kB/s), 2882KiB/s-2882KiB/s (2951kB/s-2951kB/s), io=169MiB (177MB), run=60017-60017msec
running test - fio2
--------------------------------------------------------------------------------
READ: bw=237MiB/s (249MB/s), 237MiB/s-237MiB/s (249MB/s-249MB/s), io=9564MiB (10.0GB), run=40339-40339msec
READ: bw=248MiB/s (260MB/s), 248MiB/s-248MiB/s (260MB/s-260MB/s), io=9124MiB (9567MB), run=36805-36805msec
READ: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=9480MiB (9941MB), run=38839-38839msec
READ: bw=248MiB/s (260MB/s), 248MiB/s-248MiB/s (260MB/s-260MB/s), io=9452MiB (9911MB), run=38069-38069msec
READ: bw=249MiB/s (261MB/s), 249MiB/s-249MiB/s (261MB/s-261MB/s), io=9100MiB (9542MB), run=36514-36514msec
READ: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=9444MiB (9903MB), run=38536-38536msec
READ: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=8900MiB (9332MB), run=35473-35473msec
READ: bw=243MiB/s (255MB/s), 243MiB/s-243MiB/s (255MB/s-255MB/s), io=9564MiB (10.0GB), run=39399-39399msec
READ: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=9520MiB (9982MB), run=38684-38684msec
READ: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=9536MiB (9999MB), run=39022-39022msecarm64 Linux:
Code:
running test - zsh
--------------------------------------------------------------------------------
zsh -i -c "sleep 0.1 ; exit" 0.26s user 0.28s system 92% cpu 0.584 total
zsh -i -c "sleep 0.1 ; exit" 0.24s user 0.25s system 92% cpu 0.532 total
zsh -i -c "sleep 0.1 ; exit" 0.24s user 0.22s system 92% cpu 0.502 total
zsh -i -c "sleep 0.1 ; exit" 0.28s user 0.24s system 92% cpu 0.558 total
zsh -i -c "sleep 0.1 ; exit" 0.26s user 0.30s system 93% cpu 0.598 total
zsh -i -c "sleep 0.1 ; exit" 0.24s user 0.24s system 91% cpu 0.529 total
zsh -i -c "sleep 0.1 ; exit" 0.29s user 0.27s system 92% cpu 0.604 total
zsh -i -c "sleep 0.1 ; exit" 0.26s user 0.21s system 87% cpu 0.536 total
zsh -i -c "sleep 0.1 ; exit" 0.24s user 0.27s system 92% cpu 0.544 total
zsh -i -c "sleep 0.1 ; exit" 0.27s user 0.26s system 91% cpu 0.577 total
running test - dd
--------------------------------------------------------------------------------
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.24124 s, 593 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 12.8347 s, 335 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 12.9368 s, 332 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 12.9022 s, 333 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 12.892 s, 333 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.0925 s, 328 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.0692 s, 329 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.0524 s, 329 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.0485 s, 329 MB/s
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.0796 s, 328 MB/s
running test - fio1
--------------------------------------------------------------------------------
WRITE: bw=17.0MiB/s (17.8MB/s), 17.0MiB/s-17.0MiB/s (17.8MB/s-17.8MB/s), io=1022MiB (1071MB), run=60040-60040msec
WRITE: bw=15.7MiB/s (16.5MB/s), 15.7MiB/s-15.7MiB/s (16.5MB/s-16.5MB/s), io=946MiB (992MB), run=60081-60081msec
WRITE: bw=15.7MiB/s (16.5MB/s), 15.7MiB/s-15.7MiB/s (16.5MB/s-16.5MB/s), io=942MiB (988MB), run=60040-60040msec
WRITE: bw=15.8MiB/s (16.6MB/s), 15.8MiB/s-15.8MiB/s (16.6MB/s-16.6MB/s), io=950MiB (996MB), run=60014-60014msec
WRITE: bw=15.8MiB/s (16.5MB/s), 15.8MiB/s-15.8MiB/s (16.5MB/s-16.5MB/s), io=947MiB (993MB), run=60037-60037msec
WRITE: bw=15.7MiB/s (16.4MB/s), 15.7MiB/s-15.7MiB/s (16.4MB/s-16.4MB/s), io=942MiB (988MB), run=60077-60077msec
WRITE: bw=15.8MiB/s (16.5MB/s), 15.8MiB/s-15.8MiB/s (16.5MB/s-16.5MB/s), io=946MiB (992MB), run=60048-60048msec
WRITE: bw=15.9MiB/s (16.6MB/s), 15.9MiB/s-15.9MiB/s (16.6MB/s-16.6MB/s), io=953MiB (999MB), run=60054-60054msec
WRITE: bw=15.8MiB/s (16.6MB/s), 15.8MiB/s-15.8MiB/s (16.6MB/s-16.6MB/s), io=950MiB (996MB), run=60011-60011msec
WRITE: bw=15.7MiB/s (16.5MB/s), 15.7MiB/s-15.7MiB/s (16.5MB/s-16.5MB/s), io=945MiB (990MB), run=60018-60018msec
running test - fio2
--------------------------------------------------------------------------------
READ: bw=365MiB/s (383MB/s), 365MiB/s-365MiB/s (383MB/s-383MB/s), io=8556MiB (8972MB), run=23430-23430msec
READ: bw=372MiB/s (390MB/s), 372MiB/s-372MiB/s (390MB/s-390MB/s), io=8544MiB (8959MB), run=22950-22950msec
READ: bw=373MiB/s (391MB/s), 373MiB/s-373MiB/s (391MB/s-391MB/s), io=8548MiB (8963MB), run=22897-22897msec
READ: bw=376MiB/s (395MB/s), 376MiB/s-376MiB/s (395MB/s-395MB/s), io=8540MiB (8955MB), run=22692-22692msec
READ: bw=385MiB/s (404MB/s), 385MiB/s-385MiB/s (404MB/s-404MB/s), io=8536MiB (8951MB), run=22166-22166msec
READ: bw=378MiB/s (397MB/s), 378MiB/s-378MiB/s (397MB/s-397MB/s), io=8508MiB (8921MB), run=22480-22480msec
READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=8508MiB (8921MB), run=21937-21937msec
READ: bw=385MiB/s (404MB/s), 385MiB/s-385MiB/s (404MB/s-404MB/s), io=8484MiB (8896MB), run=22026-22026msec
READ: bw=381MiB/s (400MB/s), 381MiB/s-381MiB/s (400MB/s-400MB/s), io=8496MiB (8909MB), run=22287-22287msec
READ: bw=384MiB/s (403MB/s), 384MiB/s-384MiB/s (403MB/s-403MB/s), io=8492MiB (8905MB), run=22115-22115msecFor curiosity, here is my 10 years old laptop with 5 years old (used) 2.5" SSD:
Code:
running test - zsh
--------------------------------------------------------------------------------
zsh -i -c "sleep 0.1 ; exit" 0.05s user 0.05s system 53% cpu 0.202 total
zsh -i -c "sleep 0.1 ; exit" 0.07s user 0.04s system 51% cpu 0.205 total
zsh -i -c "sleep 0.1 ; exit" 0.10s user 0.04s system 61% cpu 0.229 total
zsh -i -c "sleep 0.1 ; exit" 0.08s user 0.09s system 64% cpu 0.262 total
zsh -i -c "sleep 0.1 ; exit" 0.08s user 0.05s system 34% cpu 0.382 total
zsh -i -c "sleep 0.1 ; exit" 0.04s user 0.05s system 50% cpu 0.187 total
zsh -i -c "sleep 0.1 ; exit" 0.06s user 0.06s system 52% cpu 0.208 total
zsh -i -c "sleep 0.1 ; exit" 0.06s user 0.05s system 51% cpu 0.226 total
zsh -i -c "sleep 0.1 ; exit" 0.06s user 0.06s system 41% cpu 0.287 total
zsh -i -c "sleep 0.1 ; exit" 0.09s user 0.04s system 51% cpu 0.240 total
running test - dd
--------------------------------------------------------------------------------
4294967296 bytes transferred in 11.294465 secs (380271852 bytes/sec)
4294967296 bytes transferred in 10.897883 secs (394110251 bytes/sec)
4294967296 bytes transferred in 10.766730 secs (398911013 bytes/sec)
4294967296 bytes transferred in 11.137108 secs (385644747 bytes/sec)
4294967296 bytes transferred in 10.855323 secs (395655428 bytes/sec)
4294967296 bytes transferred in 11.843629 secs (362639471 bytes/sec)
4294967296 bytes transferred in 12.342351 secs (347986164 bytes/sec)
4294967296 bytes transferred in 13.853498 secs (310027650 bytes/sec)
4294967296 bytes transferred in 11.067325 secs (388076375 bytes/sec)
4294967296 bytes transferred in 12.458829 secs (344732823 bytes/sec)
running test - fio1
--------------------------------------------------------------------------------
WRITE: bw=13.3MiB/s (14.0MB/s), 13.3MiB/s-13.3MiB/s (14.0MB/s-14.0MB/s), io=802MiB (841MB), run=60164-60164msec
WRITE: bw=12.7MiB/s (13.3MB/s), 12.7MiB/s-12.7MiB/s (13.3MB/s-13.3MB/s), io=762MiB (799MB), run=60021-60021msec
WRITE: bw=8948KiB/s (9162kB/s), 8948KiB/s-8948KiB/s (9162kB/s-9162kB/s), io=527MiB (553MB), run=60347-60347msec
WRITE: bw=11.7MiB/s (12.2MB/s), 11.7MiB/s-11.7MiB/s (12.2MB/s-12.2MB/s), io=700MiB (734MB), run=60020-60020msec
WRITE: bw=13.2MiB/s (13.9MB/s), 13.2MiB/s-13.2MiB/s (13.9MB/s-13.9MB/s), io=793MiB (832MB), run=60021-60021msec
WRITE: bw=14.4MiB/s (15.1MB/s), 14.4MiB/s-14.4MiB/s (15.1MB/s-15.1MB/s), io=867MiB (909MB), run=60062-60062msec
WRITE: bw=11.2MiB/s (11.8MB/s), 11.2MiB/s-11.2MiB/s (11.8MB/s-11.8MB/s), io=687MiB (720MB), run=61089-61089msec
WRITE: bw=8843KiB/s (9055kB/s), 8843KiB/s-8843KiB/s (9055kB/s-9055kB/s), io=518MiB (543MB), run=60007-60007msec
WRITE: bw=10.1MiB/s (10.6MB/s), 10.1MiB/s-10.1MiB/s (10.6MB/s-10.6MB/s), io=608MiB (638MB), run=60006-60006msec
WRITE: bw=9.95MiB/s (10.4MB/s), 9.95MiB/s-9.95MiB/s (10.4MB/s-10.4MB/s), io=597MiB (626MB), run=60010-60010msec
running test - fio2
--------------------------------------------------------------------------------
READ: bw=295MiB/s (310MB/s), 295MiB/s-295MiB/s (310MB/s-310MB/s), io=9436MiB (9894MB), run=31967-31967msec
READ: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=9160MiB (9605MB), run=34305-34305msec
READ: bw=277MiB/s (290MB/s), 277MiB/s-277MiB/s (290MB/s-290MB/s), io=9528MiB (9991MB), run=34447-34447msec
READ: bw=317MiB/s (333MB/s), 317MiB/s-317MiB/s (333MB/s-333MB/s), io=9156MiB (9601MB), run=28851-28851msec
READ: bw=359MiB/s (377MB/s), 359MiB/s-359MiB/s (377MB/s-377MB/s), io=8992MiB (9429MB), run=25030-25030msec
READ: bw=337MiB/s (354MB/s), 337MiB/s-337MiB/s (354MB/s-354MB/s), io=9096MiB (9538MB), run=26979-26979msec
READ: bw=363MiB/s (381MB/s), 363MiB/s-363MiB/s (381MB/s-381MB/s), io=9084MiB (9525MB), run=25028-25028msec
READ: bw=356MiB/s (373MB/s), 356MiB/s-356MiB/s (373MB/s-373MB/s), io=9020MiB (9458MB), run=25372-25372msec
READ: bw=330MiB/s (346MB/s), 330MiB/s-330MiB/s (346MB/s-346MB/s), io=9056MiB (9496MB), run=27464-27464msec
READ: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=9660MiB (10.1GB), run=56943-56943msecMore details about disk subsystem on slow configuration:
Code:
pciconf -lcv
pcib1@pci0:0:0:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1d87 device=0x0100 subvendor=0x0000 subdevice=0x0000
vendor = 'Rockchip Electronics Co., Ltd'
device = 'RK3399 PCI Express Root Port'
class = bridge
subclass = PCI-PCI
cap 01[80] = powerspec 3 supports D0 D1 D3 current D0
cap 05[90] = MSI supports 1 message, 64 bit, vector masks
cap 11[b0] = MSI-X supports 1 message
Table in map 0x10[0x0], PBA in map 0x10[0x8]
cap 10[c0] = PCI-Express 2 root port max data 128(256) RO NS ARI disabled
max read 128
link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
slot 0 power limit 0 mW
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0017[274] = TPH Requester 1
nvme0@pci0:1:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa804 subvendor=0x144d subdevice=0xa801
vendor = 'Samsung Electronics Co Ltd'
device = 'NVMe SSD Controller SM961/PM961/SM963'
class = mass storage
subclass = NVM
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 32 messages, 64 bit
cap 10[70] = PCI-Express 2 endpoint max data 128(256) FLR RO NS
max read 512
link x4(x4) speed 5.0(8.0) ASPM disabled(L1) ClockPM disabled
cap 11[b0] = MSI-X supports 8 messages, enabled
Table in map 0x10[0x3000], PBA in map 0x10[0x2000]
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
ecap 0003[148] = Serial 1 0000000000000000
ecap 0004[158] = Power Budgeting 1
ecap 0019[168] = PCIe Sec 1 lane errors 0
ecap 0018[188] = LTR 1
ecap 001e[190] = L1 PM Substates 1
smartctl -a /dev/nvme0 | grep -i temperature
Temperature: 45 Celsius
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 45 Celsius
Temperature Sensor 2: 68 Celsius
nvmecontrol identify nvme0ns1
Size: 500118192 blocks
Capacity: 500118192 blocks
Utilization: 500118192 blocks
Thin Provisioning: Not Supported
Number of LBA Formats: 1
Current LBA Format: LBA Format #00
Metadata Capabilities
Extended: Not Supported
Separate: Not Supported
Data Protection Caps: Not Supported
Data Protection Settings: Not Enabled
Multi-Path I/O Capabilities: Not Supported
Reservation Capabilities: Not Supported
Format Progress Indicator: 0% remains
Deallocate Logical Block: Read Not Reported
Optimal I/O Boundary: 0 blocks
NVM Capacity: 256060514304 bytes
Globally Unique Identifier: 00000000000000000000000000000000
IEEE EUI64: 002538ca61006f20
LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: BestTL
R;zsh startup:
arm64 FreeBSD: 1.0469 s
arm64 Linux: 0.5564 s
amd64 FreeBSD old laptop: 0.2428 s
How to debug it? How to make it more responsive?
