Scientific Linux Forum.org



  Reply to this topicStart new topicStart Poll

> KVM + NFS Performance (ZFS/BTRFS/EXT4), KVM VM IO Performance
crash0veride
 Posted: Jul 3 2013, 06:26 PM
Quote Post


SLF Newbie


Group: Members
Posts: 9
Member No.: 1226
Joined: 24-January 12









I have done a fair bit of testing and experimenting to maximize KVM VM
performance on ZFS. Hopefully someone finds the below useful.

Guests/Benchmark used:
WinXP Pro + Crystal Disk Mark 1 pass at 1000MG
SL 6.4 + Gnome Disk Utility (palimpsest) R/W disk benchmark function

Each guest uses a file backed sparse raw disk located on an NFS mounted
directory ("/foo"). The NFS mounted directory is
interconnected via a dual 10Gbe Twin-ax copper from the targeted storage
server disk subsystem and filesystem. The connection is bonded using Linu
x
bonding (mode=4 miimon=250 xmit_hash_policy=layer2+3). NFS server e
xport
options were (rw,async,no_root_squash). The NFS client was left with defa
ult
mount options. The NFS version used was NFSv3 only. NFSv4 was disabled on

the storage server via "RPCNFSDARGS = -N 4".

*NOTES*
ZFS, EXT4, MD+LVM, or BTRFS disk subsystem configuration details are
specified in each below test configuration.
Low latency Intel SSD models were utilized in all cases wherein SSD are
being used.
When ext4 is used on top of a zvol with default mount options the
journal/metadata is synchronous and therefore offloaded to the ZIL.
Jumbo frames were enabled on all 10Gbe interfaces.
The following network tuning was applied for to optimize for 10Gbe
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 65536
net.core.rmem_default = 87380
net.core.netdev_max_backlog = 30000


WinXP qemu-kvm commandline:
/usr/libexec/qemu-kvm -name winxpfoo -M rhel6.4.0 -cpu qemu64,-svm
-enable-kvm -m 3072 -smp 2,sockets=1,cores=2,threads=1 -nodefconfig

-nodefaults -drive
file=/foo/winxp.img,if=none,id=drive-virtio-disk0,format=raw,cach
e=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-dis
k0,id=virtio-disk0
-spice port=5900,addr=3.57.109.210,disable-ticketing -k en-us -vga qx
l
-global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864
&

SL 6.4 qemu-kvm commandline:
/usr/libexec/qemu-kvm -name sl64 -M rhel6.4.0 -cpu qemu64,-svm -enable-kv
m
-m 3072 -smp 2,sockets=1,cores=2,threads=1 -nodefconfig -nodefaults
-drive
file=/vmstore/foo/SL64hd0.img,if=none,id=drive-virtio-disk0,format=
raw,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-dis
k0,id=virtio-disk0,bootindex=2
-drive
file=/vmstore/foo/SL64hd1.img,if=none,id=drive-virtio-disk1,format=
raw,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-dis
k1,id=virtio-disk1
-spice port=5900,addr=3.57.109.210,disable-ticketing -k en-us -vga qx
l
-global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864
&

-- Test Config #1 --

Storage Server: Sunfire x4140
RAM: 32GB
CPU: 2x AMD Opteron 2439SE
HBA: LSI 3801
Disks: 8 146GB 10K 2.5" SAS
Disk Config: LSI HW IME Raid + Linux LVM
Filesystem: LVM+ext4
Network: Intel x520DA2 dual port 10Gbe
OS: SL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Filesystem: LSI HW Raid+ ext4
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Benchmark Results (MB/s):
**** WinXP ****
seq: 295.4 189.7
512k: 298.8 188.9
4K: 14.75 12.99
4K QD32: 16.16 14.42

**** SL 6.4 (MB/s) ****
Min Read: 654.3
Max Read: 704.8
Avg Read: 682.5
Min Write: 233.6
Max Write: 551.1
Avg Write: 465.8
AVG Access: 0.5ms

-- End Test Config #1 --

-- Test Config #2 --
Storage Server: Sunfire x4270m2
RAM: 72GB
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 24 300GB 10K 2.5" SAS
Disks: 4 128GB Intel SATA SSD
Disk Config: ZFS Raid10 + Raid 0 ZIL + Raid 0 L2ARC
Filesystem: ZFS ZVOL+ext4
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ZFS Configuration Details/Commands
zpool create das0 mirror <dev> <dev> mirror <dev> <dev>... log <ssd> <ssd
>
cache <ssd> <ssd>
zfs create -s -V 100G -o volblocksize=64K das0/foo
mkfs.ext4 -L foo /dev/das0/foo
mount /dev/das0/foo /zvol/foo -o noatime

**** XP (MB/s) ****
seq: 315.4 273.4
512k: 302.4 263.0
4K: 12.94 8.688
4K QD32: 14.45 13.13

**** SL 6.4 (MB/s)****
Min Read: 630.0
Max Read: 698.2
Avg Read: 662.8
Min Write: 357.6
Max Write: 1.1 GB/s
Avg Write: 895.3
AVG Access: 0.5ms

-- End Test Config #2 --

-- Test Config #3 --

Storage Server: Sunfire x4270m2
RAM: 72GB
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 24 300GB 10K 2.5" SAS
Disks: 4 128GB Intel SATA SSD
Disk Config: ZFS Raid10 + Raid 0 ZIL + Raid 0 L2ARC
Filesystem: ZFS
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ZFS Configuration Details/Commands
zpool create das0 mirror <dev> <dev> mirror <dev> <dev>... log <ssd> <ssd
>
cache <ssd> <ssd>
zfs create das0/foo

**** XP (MB/s) ****
seq: 111.4 57.36
512k: 95.64 54.33
4K: 11.80 2.878
4K QD32: 11.31 3.095

-- End Test Config #3 --


-- Test Config #4 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: ZFS Raid10 + ZIL + L2ARC
Filesystem: ZFS ZVOL+ext4
Network: Intel x520DA2 dual port 10Gbe
OS: Fedora 18

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ZFS Configuration Details/Commands
zpool create das0 mirror <dev> <dev> mirror <dev> <dev> log <ssd> cache <
ssd>
zfs create -s -V 100G -o volblocksize=64K das0/foo
mkfs.ext4 -L foo /dev/das0/foo
mount /dev/das0/foo /zvol/foo -o noatime

**** XP (MB/s) ****
seq: 196.5 200.2
512k: 195.2 188.8
4K: 9.665 7.043
4K QD32: 12.02 8.140

**** SL 6.4 (MB/s) ****
Min Read: 423.7
Max Read: 575.3
Avg Read: 526.7
Min Write: 44.4
Max Write: 668.8
Avg Write: 540.5
AVG Access: 1.4ms


-- End Test Config #4 --

-- Test Config #5 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: ZFS Raid10 + ZIL + L2ARC
Filesystem: ZFS
Network: Intel x520DA2 dual port 10Gbe
OS: Fedora 18

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ZFS Configuration Details/Commands
zpool create das0 mirror <dev> <dev> mirror <dev> <dev> log <ssd> cache <
ssd>
zfs create das0/foo

**** XP (MB/s) ****
seq: 152.5 126.2
512k: 147.4 121.8
4K: 9.189 7.481
4K QD32: 11.90 8.003

**** SL 6.4 (MB/s) ****
Min Read: 409.3
Max Read: 564.4
Avg Read: 511.6
Min Write: 237.1
Max Write: 658.0
Avg Write: 555.3
AVG Access: 2.9ms

-- End Test Config #5 --


-- Test Config #6 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: Linux MD Raid10 + LVM + ext4
Filesystem: ext4
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ext4 Configuration Details/Commands
4 Disk SSD Linux MD Raid 10 Stripe Size 1MB
100GB LV(foo) from VG0 via PV MD device (ext4 data )

2 Disk SSD Linux MD Raid 0 Stripe Size 1MB
1GB LV(journaldev) from VG1 via PV MD device (ext4 journal/metadata)

ext4 filesystem lv-foo, external journal on lv-journaldev (external
journal_data_ordered)
mounted at /vg0/foo (mount options
rw,noatime,journal_checksum,journal_async_commit)

**** XP (MB/s) ****
seq: 251.3 203.9
512k: 249.0 217.7
4K: 10.12 9.012
4K QD32: 13.18 11.54

**** SL 6.4 (MB/s) ****
Min Read: 523.3
Max Read: 663.0
Avg Read: 606.4
Min Write: 239.9
Max Write: 656.0
Avg Write: 563.8
AVG Access: 0.8ms

-- End Test Config #6 --

-- Test Config #7 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: BTRFS Raid10 data + raid0 metadata
Filesystem: BTRFS
Network: Intel x520DA2 dual port 10Gbe
OS: Fedora 18

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/BTRFS Configuration Details/Commands
mkfs.btrfs -L foo -m raid0 <ssd> <ssd> -d raid10 <ssd> <ssd> <ssd> <ssd>
btrfs subvolume create /das0/foo

**** XP (MB/s) ****
seq: 195.4 180.0
512k: 180.8 174.5
4K: 9.898 7.183
4K QD32: 11.87 8.248

**** SL 6.4 (MB/s) ****
Min Read: 465.4
Max Read: 554.6
Avg Read: 514.1
Min Write: 450.5
Max Write: 670.3
Avg Write: 525.3
AVG Access: 0.8ms

-- End Test Config #7--


-- Test Config #8 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: BTRFS Raid10 data + raid0 metadata
Filesystem: BTRFS
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/BTRFS Configuration Details/Commands
mkfs.btrfs -L foo -m raid0 <ssd> <ssd> -d raid10 <ssd> <ssd> <ssd> <ssd>
btrfs subvolume create /das0/foo

**** XP (MB/s) ****
seq: 232.1 123.4
512k: 243.0 213.6
4K: 9.393 9.170
4K QD32: 12.04 10.74

**** SL 6.4 (MB/s) ****
Min Read: 508.0
Max Read: 652.4
Avg Read: 594.4
Min Write: 409.6
Max Write: 670.4
Avg Write: 525.8
AVG Access: 0.5

-- End Test Config #8 --
PM
^
AndrewSerk
 Posted: Jul 6 2013, 01:12 AM
Quote Post


SLF Moderator
*****

Group: Moderators
Posts: 457
Member No.: 54
Joined: 14-April 11









Thanks for the info/data crash0veride, I am sure many will find it useful.
PM
^
helikaon
 Posted: Jul 9 2013, 09:23 AM
Quote Post


SLF Administrator
*******

Group: Admins
Posts: 836
Member No.: 4
Joined: 8-April 11









QUOTE (AndrewSerk @ Jul 6 2013, 01:12 AM)
Thanks for the info/data crash0veride, I am sure many will find it useful.


Interesting indeed.
Ext4 and BTRFS beat the ZFS by quite a bit in the tests. It is not performed on same type of storage and with different hw raid configs, so it's not that accurate.

Question: do you use ZFS with linux in production? Or just testing ... Whats your own opinion on it?

As for me, i stay with Ext4 so far or XFS (xfs is actually very nice).


Thanks for this :]


--------------------
PMEmail Poster
^
crash0veride
 Posted: Sep 4 2013, 02:02 AM
Quote Post


SLF Newbie


Group: Members
Posts: 9
Member No.: 1226
Joined: 24-January 12









QUOTE (helikaon @ Jul 9 2013, 03:23 AM)
Question: do you use ZFS with linux in production? Or just testing ... Whats your own opinion on it?


I do have several ZOL based servers up and running 24x7 for a few years running.
If I was to keep things high level experience wise I would say ZOL is for you if:
1) If you want 100% guarantee on data integrity
2) Flexibility in file-system and disk provisioning (EG: what only ZFS and BTRFS can provide, except ZFS data vdevs cannot shrink or go away permanently)
3) Snapshotting and data-deduplication
4) Ability to use SSD as cache
5) Performance ATM is not your main requirement (it will get there, well on it's way)

I also have several BTRFS based servers humming along 24x7 as well (They run OEL). However the use case for them is slightly different in that performance is more a key requirement along with some off the above. I use SSD for metadata disks and spinning disks for data. I have achieved performance equal to or better then ext4/XFS+LVM+HW/SW Raid setting things up in this manner. What I gain is the other goodies that ext4/XFS+ LVM+HW/SW RAID cannot provide.

Ultimately I would say use what works for you, based on the use case it needs to fulfill. Better known as "right tool for the job cool.gif"

Forward looking in coming years BTRFS along with ZOL are on the right track and will definitely become staple file systems in Linux.
PM
^
0 User(s) are reading this topic (0 Guests and 0 Anonymous Users)
0 Members:

Topic Options Reply to this topicStart new topicStart Poll