Okay,
here was the path I used:
1. shutdown freenas, replace 1 x 2TB Drive with a 2.5TB Drive, start up box.
2. Go to WebUI/Storage/Volumes/View All Volumes/View Disks.
The missing 2TB drive is marked "unknown", click on the "Replace" button and choose the available 2.5TB.
The missing disk will still be shown, but with a wierd numeric name like 3232423454552234 - choose "remove" or "delete" on this defunct drive.
Doing all this through the GUI is better, since the correct Swap space is created before anything else and keeps it neat and tidy.
3. Allow the disk to resilver, you can check progress with "zpool status TANKNAME" through ssh. (The WebUI Zpool status just shows everything as "HEALTHY" but don't trust that!)
[root@freenas] ~# zpool status RAID5Z
pool: RAID5Z
state: ONLINE
scrub: resilver in progress for 0h10m, 2.93% done, 8h40m to go
config:
NAME STATE READ WRITE CKSUM
RAID5Z ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
ada2p2 ONLINE 0 0 0
ada3p2 ONLINE 0 0 0
ada4p2 ONLINE 0 0 0 5.9G resilvered
errors: No known data errors
4. Once the disk is completed resilvering, powerdown and go back to (1.) and do the next disk.
For 5 disks took me a few days, since I was resilvering about 1TB of data per drive. Once the last disk is resilvered, just reboot and the extra space will be available:
before:
[root@freenas] ~# df -h
RAID5Z 7.1T 3.8T 3.3T 54% /mnt/RAID5Z
after:
[root@freenas] ~# df -h
RAID5Z 8.9T 3.8T 5.1T 43% /mnt/RAID5Z
During this entire process of resilvering disks, I was still able to watch 1080p movies, download stuff with sabnzbd/sickbeard/couchpotato and generally work as if nothing was going on. The ability to upgrade the disks without downtime is the reason I wanted FreeNAS in the first place.
Pk