I moved disks over from OpenSolaris b134 to ZFSOnLinux directly. The new NAS had just awful performance over Samba and rsync, especially with large folders. I did a bunch tracing and watched the xattr request for posix ACLs use tons of time.
After reading up on it, what I found out is this: ZFS on Linux stores xattrs in a hidden folder, as regular files! This is very slow, requiring multiple seeks per xattr, and it doesn't appear to cache very well either.
The fix is to store the data in the inodes. And given the performance impact, I can't tell at all why this isn't the default:
zfs set xattr=sa (pool)If you use ZfsOnLinux, you should probably go do this now. It's that big a difference.
(Also, some bugs were reported early on with "xattr=sa" but they appear to be fixed as of 0.6.2.)
Performance is quite amazing after this change, and I recommend it. I hope it's the default someday.
Also, I installed samba4, and this new version can store xattr data in a "tdb" (Samba's "temporary" database.) I didn't need to do this after the above fix.