If you go too wide you end up with a situation where you are writing extremely small amounts of data across a large number of disks, and this hurts performance. Also when you are trying to write small amounts of data you might write 4K of data, but then have to write several times that in parity data. Kind of inefficient when your checksums are several times what your actual data uses. Throughput still increase for linear writes and linear reads (which really aren't linear in ZFS but hopefully the thought is still conveyed) but when you need more than the smallest amount of iops your pool performance is basically crap. As pools get bigger you end up needing more iops just because your file system metadata needs it. So directory listing start getting slower and slower. It's just not a pleasant experience in the long term. But, early on it looks like it'll be fine. I did tests with 10 disk RAIDZ2, 11 disk RAIDZ3 and 18 disk RAIDZ3 and I was convinced that the recommendat
I did an 18 disk raidz3 because I wanted to see if even as a home user would the recommendations matter. I'm a single user and live alone and yet I was disgusted with the performance. My new pool (which is about to start getting filled with data) is a 10 disk RAIDZ2. I learned my lesson and I'll never go wider than the "recommendations" again. That was just too much of a pain and once you've realized what you've done, undoing it requires you to destroy the pool. Obviously that isn't something that most people can "easily" do.
Yes, this does put some limitations on how you expand. Sorry, but that basically "life" with ZFS. ZFS isn't designed for home users. ZFS is for enterprise class users that are willing to spend money to get ZFS' great features. The "limitations" you have are mostly money problems. It's expensive to get it all right. For companies going with ZFS, the data is worth far more than the money spent on the server. I've joked that
BTRFS is "poor man's ZFS" because it has the potential to be a good alternative for ZFS when dealing with home users (and even to challenge some enterprises that want ZFS-like quality). It's supposed to let you do things like add single disks, take single disks out, mix and match disk sizes, and all sorts of other stuff I'm sure I don't even know of. Only time will tell though as BTRFS isn't anywhere near as good as ZFS currently. BTRFS may grow and compete with ZFS or it may die before it gets anywhere.