ZFS xattr tuning on Linux

It took me a while to figure out why my Linux ZFS disk was so slow, but there's an easy fix.

I moved disks over from OpenSolaris b134 to ZFSOnLinux directly. The new NAS had just awful performance over Samba and rsync, especially with large folders. I did a bunch tracing and watched the xattr request for posix ACLs use tons of time.

After reading up on it, what I found out is this: ZFS on Linux stores xattrs in a hidden folder, as regular files! This is very slow, requiring multiple seeks per xattr, and it doesn't appear to cache very well either.

The fix is to store the data in the inodes. And given the performance impact, I can't tell at all why this isn't the default:
zfs set xattr=sa (pool)
If you use ZfsOnLinux, you should probably go do this now. It's that big a difference.

(Also, some bugs were reported early on with "xattr=sa" but they appear to be fixed as of 0.6.2.)

Performance is quite amazing after this change, and I recommend it. I hope it's the default someday.

Also, I installed samba4, and this new version can store xattr data in a "tdb" (Samba's "temporary" database.) I didn't need to do this after the above fix.

4 comments:

  1. Very useful, thanks. Filesystem browsing improved a lot!

    ReplyDelete
  2. Anonymous1:51 PM

    Can you talk a little bit (or have you already; just found your blog in a search) about ZonL, and (specifcally) whether the Z features make that a good target for Xen VDI storage, even at the cost of putitng a filesystem between the guest and the disks?

    (PS: what a random -- and somewhat limited -- choice of comment auth providers. :-})

    ReplyDelete
  3. IMO, ZFS under Solaris is pretty amazing at caching.

    Compared to that, ZoL is significantly worse at metadata ops (handling lots of files), but it's probably closer to Solaris speed for a single file / .VDI-style storage. I've even had problems with an SSD l2arc on Linux (took it out of the array because it was slower). But overall, it's solid and production-ready aside from the performance questions.

    (Re: comments - Yeah it's Blogger, so comment providers are whatever Google allows!)

    ReplyDelete
  4. See https://github.com/zfsonlinux/zfs/issues/443 for an explanation why this isn't default :)

    ReplyDelete