Hanging on boot / import

[Hexland]

I've almost got my test rig to the point where I'm comfortable moving into a production environment...

HP N40L Microserver
8Gb ECC RAM
6 x 2Tb HDD
IBM 1015M SAS Controller (IT Mode)
Intel GT Gigabit Ethernet
8Gb Flash Drive

Night before last-- I set up the drives in the following configuration
ZPOOL01 -- 6 x 2Tb HDD (Force 4K Sectors), Raid-Z2
This reported 7.1Tb of free space on the pool

I set up a ZFS Volume of 7TB and exported it as a device extent through iSCSI

I set up my WHS (v1) machine with the Microsoft iSCSI Initiator, and mounted the ZFSVolume as an NTFS iSCSI volume through the dedicated Intel GT Gigabit adapter (with a single 2ft crossover cable)

I proceeded to 'robocopy D:\Shares I:\ /mir /w:5 /r:3' from WHS and push a copy of my 5TB of data from my old WHS server to the new FreeNAS machine.

I left it copying for about a day and a half (averaging about 50Mb/s write speed)

During the copy operation, there WHS machine started reporting 'I/O errors' and 'Device not ready' on the iSCSI device. FreeNAS was also logging lots of errors on its end too. Looking at the ZPOOL said that I was down to 384K of free space (which I found odd, because I hadn't been writing to the pool, only the volume, and its size should have been fixed at 7Tb (leaving 0.1Tb free)

On reboot, the FreeNAS machine would get as far as mounting the ZFS Pool, and just hang.
I booted into safe mode, and fsck'd the main OS drive, and tried to manually import the ZFS Pool -- again just hanging.
I swapped out the flash drive hosting FreeNAS and put in a fresh one, booted into the clean FreeNAS and tried to import the ZFS Pool, with no success.

After googling a bit, I saw some suggestions that I should boot with an Ubuntu Live CD and install the experimental ZFS filesystem, and try importing and fixing it that way, but I didn't get a chance.

Anyway -- this has got me a bit worried now... I'm almost ready to move all my data over to the new server, but if a simple power failure is enough to trash something that I thought was super resiliant, I'm a bit concerned.

Was there something else I missed? Should I have tried another step?

 

[Hexland]

Son of a ****... it did it again.

I just copied 3.86Tb of info from my WHS machine to FreeNAS via iSCSI to a ZFS Volume which should be 6Tb in size (from a ZPool of 7.1Tb (made up of 6x2Tb drives in RaidZ2 format))

WHS has just stalled with Device errors, and FreeNAS is churning out iSCSI errors -- and the ZPool, which should have 1.1Tb of free space after the ZVolume creation is reporting 0 remaining space.


I logged in with the shell, and there are no files on /mnt/ZPOOL01

Repeated over and over (/var/log/messages is completely full)

Code:

Jun 13 19:00:01 freenas istgt[3140]: istgt_lu_disk.c:6201:istgt_lu_disk_execute: ***ERROR*** lu_disk_lbwrite() failed
Jun 13 19:00:01 freenas istgt[3140]: istgt_lu_disk.c:4026:istgt_lu_disk_lbwrite: ***ERROR*** lu_disk_write() failed

Here's the head of /var/log/messages

Code:

Jun 12 00:52:19 freenas newsyslog[1453]: logfile first created
Jun 12 00:52:19 freenas syslogd: kernel boot file is /boot/kernel/kernel
Jun 12 00:52:19 freenas kernel: Copyright (c) 1992-2011 The FreeBSD Project.
Jun 12 00:52:19 freenas kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jun 12 00:52:19 freenas kernel: The Regents of the University of California. All rights reserved.
Jun 12 00:52:19 freenas kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jun 12 00:52:19 freenas kernel: FreeBSD 8.2-RELEASE-p6 #0: Wed Apr 18 16:16:31 PDT 2012
Jun 12 00:52:19 freenas kernel: root@build.ixsystems.com:/build/home/jpaetzel/freenas/e2e-bld.OfE4KSqZ/os-base/amd64/build/home/jpaetzel/freenas/e2e-bld.OfE4KSqZ/FreeBSD/src/sys/FREENAS.amd64 amd64
Jun 12 00:52:19 freenas kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Jun 12 00:52:19 freenas kernel: CPU: AMD Turion(tm) II Neo N40L Dual-Core Processor (1497.51-MHz K8-class CPU)
Jun 12 00:52:19 freenas kernel: Origin = "AuthenticAMD"  Id = 0x100f63  Family = 10  Model = 6  Stepping = 3
Jun 12 00:52:19 freenas kernel: Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Jun 12 00:52:19 freenas kernel: Features2=0x802009<SSE3,MON,CX16,POPCNT>
Jun 12 00:52:19 freenas kernel: AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
Jun 12 00:52:19 freenas kernel: AMD Features2=0x837ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,<b19>>
Jun 12 00:52:19 freenas kernel: TSC: P-state invariant
Jun 12 00:52:19 freenas kernel: real memory  = 8589934592 (8192 MB)
Jun 12 00:52:19 freenas kernel: avail memory = 8126205952 (7749 MB)
Jun 12 00:52:19 freenas kernel: ACPI APIC Table: <HP     ProLiant>
Jun 12 00:52:19 freenas kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Jun 12 00:52:19 freenas kernel: FreeBSD/SMP: 1 package(s) x 2 core(s)
Jun 12 00:52:19 freenas kernel: cpu0 (BSP): APIC ID:  0
Jun 12 00:52:19 freenas kernel: cpu1 (AP): APIC ID:  1
Jun 12 00:52:19 freenas kernel: ioapic0 <Version 2.1> irqs 0-23 on motherboard
Jun 12 00:52:19 freenas kernel: kbd1 at kbdmux0
Jun 12 00:52:19 freenas kernel: cryptosoft0: <software crypto> on motherboard
Jun 12 00:52:19 freenas kernel: acpi0: <HP ProLiant> on motherboard
Jun 12 00:52:19 freenas kernel: acpi0: [ITHREAD]
Jun 12 00:52:19 freenas kernel: acpi0: Power Button (fixed)
Jun 12 00:52:19 freenas kernel: acpi0: reservation of fee00000, 1000 (3) failed
Jun 12 00:52:19 freenas kernel: acpi0: reservation of ffb80000, 80000 (3) failed
Jun 12 00:52:19 freenas kernel: acpi0: reservation of fec10000, 20 (3) failed
Jun 12 00:52:19 freenas kernel: acpi0: reservation of fed80000, 1000 (3) failed
Jun 12 00:52:19 freenas kernel: acpi0: reservation of 0, a0000 (3) failed
Jun 12 00:52:19 freenas kernel: acpi0: reservation of 100000, d7f00000 (3) failed
Jun 12 00:52:19 freenas kernel: Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Jun 12 00:52:19 freenas kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
Jun 12 00:52:19 freenas kernel: cpu0: <ACPI CPU> on acpi0
Jun 12 00:52:19 freenas kernel: cpu1: <ACPI CPU> on acpi0
Jun 12 00:52:19 freenas kernel: acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Jun 12 00:52:19 freenas kernel: Timecounter "HPET" frequency 14318180 Hz quality 900
Jun 12 00:52:19 freenas kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Jun 12 00:52:19 freenas kernel: pci0: <ACPI PCI bus> on pcib0
Jun 12 00:52:19 freenas kernel: pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
Jun 12 00:52:19 freenas kernel: pci1: <ACPI PCI bus> on pcib1
Jun 12 00:52:19 freenas kernel: vgapci0: <VGA-compatible display> port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe6f0000-0xfe6fffff,0xfe500000-0xfe5fffff irq 18 at device 5.0 on pci1
Jun 12 00:52:19 freenas kernel: pcib2: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci0
Jun 12 00:52:19 freenas kernel: pci2: <ACPI PCI bus> on pcib2
Jun 12 00:52:19 freenas kernel: mps0: <LSI SAS2008> port 0xd000-0xd0ff mem 0xfe7fc000-0xfe7fffff,0xfe780000-0xfe7bffff irq 18 at device 0.0 on pci2
Jun 12 00:52:19 freenas kernel: mps0: Firmware: 11.00.00.00
Jun 12 00:52:19 freenas kernel: mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
Jun 12 00:52:19 freenas kernel: mps0: [ITHREAD]
Jun 12 00:52:19 freenas kernel: pcib3: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
Jun 12 00:52:19 freenas kernel: pci3: <ACPI PCI bus> on pcib3
Jun 12 00:52:19 freenas kernel: em0: <Intel(R) PRO/1000 Network Connection 7.1.9> port 0xe800-0xe81f mem 0xfe8e0000-0xfe8fffff,0xfe800000-0xfe87ffff,0xfe8dc000-0xfe8dffff irq 16 at device 0.0 on pci3
Jun 12 00:52:19 freenas kernel: em0: Using MSIX interrupts with 3 vectors
Jun 12 00:52:19 freenas kernel: em0: [ITHREAD]
Jun 12 00:52:19 freenas last message repeated 2 times
Jun 12 00:52:19 freenas kernel: em0: Ethernet address: 68:05:ca:02:eb:23
Jun 12 00:52:19 freenas kernel: pcib4: <ACPI PCI-PCI bridge> irq 18 at device 6.0 on pci0
Jun 12 00:52:19 freenas kernel: pci4: <ACPI PCI bus> on pcib4
Jun 12 00:52:19 freenas kernel: bge0: <HP NC107i PCIe Gigabit Server Adapter, ASIC rev. 0x5784100> mem 0xfe9f0000-0xfe9fffff irq 18 at device 0.0 on pci4
Jun 12 00:52:19 freenas kernel: bge0: CHIP ID 0x05784100; ASIC REV 0x5784; CHIP REV 0x57841; PCI-E
Jun 12 00:52:19 freenas kernel: miibus0: <MII bus> on bge0
Jun 12 00:52:19 freenas kernel: brgphy0: <BCM5784 10/100/1000baseTX PHY> PHY 1 on miibus0
Jun 12 00:52:19 freenas kernel: brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
Jun 12 00:52:19 freenas kernel: bge0: Ethernet address: 2c:76:8a:ab:dc:92
Jun 12 00:52:19 freenas kernel: bge0: [FILTER]
Jun 12 00:52:19 freenas kernel: ahci0: <ATI IXP700 AHCI SATA controller> port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe4ffc00-0xfe4fffff irq 19 at device 17.0 on pci0
Jun 12 00:52:19 freenas kernel: ahci0: [ITHREAD]
Jun 12 00:52:19 freenas kernel: ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported
Jun 12 00:52:19 freenas kernel: ahcich0: <AHCI channel> at channel 0 on ahci0
Jun 12 00:52:19 freenas kernel: ahcich0: [ITHREAD]
Jun 12 00:52:19 freenas kernel: ahcich1: <AHCI channel> at channel 1 on ahci0
Jun 12 00:52:19 freenas kernel: ahcich1: [ITHREAD]
Jun 12 00:52:19 freenas kernel: ahcich2: <AHCI channel> at channel 2 on ahci0
Jun 12 00:52:19 freenas kernel: ahcich2: [ITHREAD]
Jun 12 00:52:19 freenas kernel: ahcich3: <AHCI channel> at channel 3 on ahci0
Jun 12 00:52:19 freenas kernel: ahcich3: [ITHREAD]
Jun 12 00:52:19 freenas kernel: ahcich4: <AHCI channel> at channel 4 on ahci0
Jun 12 00:52:19 freenas kernel: ahcich4: [ITHREAD]
Jun 12 00:52:19 freenas kernel: ahcich5: <AHCI channel> at channel 5 on ahci0
Jun 12 00:52:19 freenas kernel: ahcich5: [ITHREAD]
Jun 12 00:52:19 freenas kernel: ohci0: <OHCI (generic) USB controller> mem 0xfe4fe000-0xfe4fefff irq 18 at device 18.0 on pci0
Jun 12 00:52:19 freenas kernel: ohci0: [ITHREAD]
Jun 12 00:52:19 freenas kernel: usbus0: <OHCI (generic) USB controller> on ohci0
Jun 12 00:52:19 freenas kernel: ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe4ff800-0xfe4ff8ff irq 17 at device 18.2 on pci0
Jun 12 00:52:19 freenas kernel: ehci0: [ITHREAD]
Jun 12 00:52:19 freenas kernel: usbus1: EHCI version 1.0
Jun 12 00:52:19 freenas kernel: usbus1: <EHCI (generic) USB 2.0 controller> on ehci0
Jun 12 00:52:19 freenas kernel: ohci1: <OHCI (generic) USB controller> mem 0xfe4fd000-0xfe4fdfff irq 18 at device 19.0 on pci0
Jun 12 00:52:19 freenas kernel: ohci1: [ITHREAD]
Jun 12 00:52:19 freenas kernel: usbus2: <OHCI (generic) USB controller> on ohci1
Jun 12 00:52:19 freenas kernel: ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe4ff400-0xfe4ff4ff irq 17 at device 19.2 on pci0
Jun 12 00:52:19 freenas kernel: ehci1: [ITHREAD]
Jun 12 00:52:19 freenas kernel: usbus3: EHCI version 1.0
Jun 12 00:52:19 freenas kernel: usbus3: <EHCI (generic) USB 2.0 controller> on ehci1
Jun 12 00:52:19 freenas kernel: pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
Jun 12 00:52:19 freenas kernel: isab0: <PCI-ISA bridge> at device 20.3 on pci0
Jun 12 00:52:19 freenas kernel: isa0: <ISA bus> on isab0
Jun 12 00:52:19 freenas kernel: pcib5: <ACPI PCI-PCI bridge> at device 20.4 on pci0
Jun 12 00:52:19 freenas kernel: pci5: <ACPI PCI bus> on pcib5
Jun 12 00:52:19 freenas kernel: ohci2: <OHCI (generic) USB controller> mem 0xfe4fc000-0xfe4fcfff irq 18 at device 22.0 on pci0
Jun 12 00:52:19 freenas kernel: ohci2: [ITHREAD]
Jun 12 00:52:19 freenas kernel: usbus4: <OHCI (generic) USB controller> on ohci2
Jun 12 00:52:19 freenas kernel: ehci2: <EHCI (generic) USB 2.0 controller> mem 0xfe4ff000-0xfe4ff0ff irq 17 at device 22.2 on pci0
Jun 12 00:52:19 freenas kernel: ehci2: [ITHREAD]
Jun 12 00:52:19 freenas kernel: usbus5: EHCI version 1.0
Jun 12 00:52:19 freenas kernel: usbus5: <EHCI (generic) USB 2.0 controller> on ehci2
Jun 12 00:52:19 freenas kernel: amdtemp0: <AMD K8 Thermal Sensors> on hostb4
Jun 12 00:52:19 freenas kernel: acpi_button0: <Power Button> on acpi0
Jun 12 00:52:19 freenas kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Jun 12 00:52:19 freenas kernel: acpi_hpet1: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Jun 12 00:52:19 freenas kernel: device_attach: acpi_hpet1 attach returned 12
Jun 12 00:52:19 freenas kernel: orm0: <ISA Option ROM> at iomem 0xcf000-0xcffff on isa0
Jun 12 00:52:19 freenas kernel: atkbd: unable to set the command byte.
Jun 12 00:52:19 freenas kernel: sc0: <System console> at flags 0x100 on isa0
Jun 12 00:52:19 freenas kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Jun 12 00:52:19 freenas kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Jun 12 00:52:19 freenas kernel: atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
Jun 12 00:52:19 freenas kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Jun 12 00:52:19 freenas kernel: kbd0 at atkbd0
Jun 12 00:52:19 freenas kernel: atkbd0: [GIANT-LOCKED]
Jun 12 00:52:19 freenas kernel: atkbd0: [ITHREAD]
Jun 12 00:52:19 freenas kernel: psm0: unable to set the command byte.
Jun 12 00:52:19 freenas kernel: ppc0: cannot reserve I/O port range
Jun 12 00:52:19 freenas kernel: acpi_throttle0: <ACPI CPU Throttling> on cpu0
Jun 12 00:52:19 freenas kernel: hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Jun 12 00:52:19 freenas kernel: fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
Jun 12 00:52:19 freenas kernel: Timecounters tick every 1.000 msec
Jun 12 00:52:19 freenas kernel: usbus0: 12Mbps Full Speed USB v1.0
Jun 12 00:52:19 freenas kernel: usbus1: 480Mbps High Speed USB v2.0
Jun 12 00:52:19 freenas kernel: usbus2: 12Mbps Full Speed USB v1.0
Jun 12 00:52:19 freenas kernel: usbus3: 480Mbps High Speed USB v2.0
Jun 12 00:52:19 freenas kernel: usbus4: 12Mbps Full Speed USB v1.0
Jun 12 00:52:19 freenas kernel: usbus5: 480Mbps High Speed USB v2.0
Jun 12 00:52:19 freenas kernel: ugen0.1: <ATI> at usbus0
Jun 12 00:52:19 freenas kernel: uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
Jun 12 00:52:19 freenas kernel: ugen1.1: <ATI> at usbus1
Jun 12 00:52:19 freenas kernel: uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
Jun 12 00:52:19 freenas kernel: ugen2.1: <ATI> at usbus2
Jun 12 00:52:19 freenas kernel: uhub2: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
Jun 12 00:52:19 freenas kernel: ugen3.1: <ATI> at usbus3
Jun 12 00:52:19 freenas kernel: uhub3: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
Jun 12 00:52:19 freenas kernel: ugen4.1: <ATI> at usbus4
Jun 12 00:52:19 freenas kernel: uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
Jun 12 00:52:19 freenas kernel: ugen5.1: <ATI> at usbus5
Jun 12 00:52:19 freenas kernel: uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
Jun 12 00:52:19 freenas kernel: uhub4: 4 ports with 4 removable, self powered
Jun 12 00:52:19 freenas kernel: uhub0: 5 ports with 5 removable, self powered
Jun 12 00:52:19 freenas kernel: uhub2: 5 ports with 5 removable, self powered
Jun 12 00:52:19 freenas kernel: uhub5: 4 ports with 4 removable, self powered
Jun 12 00:52:19 freenas kernel: uhub1: 5 ports with 5 removable, self powered
Jun 12 00:52:19 freenas kernel: uhub3: 5 ports with 5 removable, self powered
Jun 12 00:52:19 freenas kernel: ugen3.2: <PNY Technologies> at usbus3
Jun 12 00:52:19 freenas kernel: umass0: <PNY Technologies USB 2.0 FD, class 0/0, rev 2.00/81.92, addr 2> on usbus3
Jun 12 00:52:19 freenas kernel: ugen0.2: <Microsoft> at usbus0
Jun 12 00:52:19 freenas kernel: ums0: <Microsoft Microsoft 3-Button Mouse with IntelliEye(TM), class 0/0, rev 1.10/3.00, addr 2> on usbus0
Jun 12 00:52:19 freenas kernel: ums0: 3 buttons and [XYZ] coordinates ID=0
Jun 12 00:52:19 freenas kernel: ugen0.3: <Microsoft> at usbus0
Jun 12 00:52:19 freenas kernel: ukbd0: <Microsoft Natural Ergonomic Keyboard 4000, class 0/0, rev 2.00/1.73, addr 3> on usbus0
Jun 12 00:52:19 freenas kernel: kbd2 at ukbd0
Jun 12 00:52:19 freenas kernel: uhid0: <Microsoft Natural Ergonomic Keyboard 4000, class 0/0, rev 2.00/1.73, addr 3> on usbus0
Jun 12 00:52:19 freenas kernel: da0 at mps0 bus 0 scbus0 target 0 lun 0
Jun 12 00:52:19 freenas kernel: da0: <ATA ST2000DL003-9VT1 CC3C> Fixed Direct Access SCSI-6 device 
Jun 12 00:52:19 freenas kernel: da0: 600.000MB/s transfers
Jun 12 00:52:19 freenas kernel: da0: Command Queueing enabled
Jun 12 00:52:19 freenas kernel: da0: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Jun 12 00:52:19 freenas kernel: da1 at mps0 bus 0 scbus0 target 1 lun 0
Jun 12 00:52:19 freenas kernel: da1: <ATA WDC WD20EARS-00M AB51> Fixed Direct Access SCSI-6 device 
Jun 12 00:52:19 freenas kernel: da1: 300.000MB/s transfers
Jun 12 00:52:19 freenas kernel: da1: Command Queueing enabled
Jun 12 00:52:19 freenas kernel: da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Jun 12 00:52:19 freenas kernel: da2 at mps0 bus 0 scbus0 target 4 lun 0
Jun 12 00:52:19 freenas kernel: da2: <ATA ST32000542AS CC34> Fixed Direct Access SCSI-6 device 
Jun 12 00:52:19 freenas kernel: da2: 300.000MB/s transfers
Jun 12 00:52:19 freenas kernel: da2: Command Queueing enabled
Jun 12 00:52:19 freenas kernel: da2: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Jun 12 00:52:19 freenas kernel: da3 at mps0 bus 0 scbus0 target 5 lun 0
Jun 12 00:52:19 freenas kernel: da3: <ATA ST32000542AS CC34> Fixed Direct Access SCSI-6 device 
Jun 12 00:52:19 freenas kernel: da3: 300.000MB/s transfers
Jun 12 00:52:19 freenas kernel: da3: Command Queueing enabled
Jun 12 00:52:19 freenas kernel: da3: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Jun 12 00:52:19 freenas kernel: da4 at mps0 bus 0 scbus0 target 6 lun 0
Jun 12 00:52:19 freenas kernel: da4: <ATA Hitachi HDS72302 A5C0> Fixed Direct Access SCSI-6 device 
Jun 12 00:52:19 freenas kernel: da4: 600.000MB/s transfers
Jun 12 00:52:19 freenas kernel: da4: Command Queueing enabled
Jun 12 00:52:19 freenas kernel: da4: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Jun 12 00:52:19 freenas kernel: da5 at mps0 bus 0 scbus0 target 7 lun 0
Jun 12 00:52:19 freenas kernel: da5: <ATA Hitachi HDS72302 A5C0> Fixed Direct Access SCSI-6 device 
Jun 12 00:52:19 freenas kernel: da5: 600.000MB/s transfers
Jun 12 00:52:19 freenas kernel: da5: Command Queueing enabled
Jun 12 00:52:19 freenas kernel: da5: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Jun 12 00:52:19 freenas kernel: da6 at umass-sim0 bus 0 scbus7 target 0 lun 0
Jun 12 00:52:19 freenas kernel: da6: <PNY USB 2.0 FD 8192> Removable Direct Access SCSI-4 device 
Jun 12 00:52:19 freenas kernel: da6: 40.000MB/s transfers
Jun 12 00:52:19 freenas kernel: da6: 7728MB (15826944 512 byte sectors: 255H 63S/T 985C)
Jun 12 00:52:19 freenas kernel: SMP: AP CPU #1 Launched!
Jun 12 00:52:19 freenas kernel: GEOM: da6s1: geometry does not match label (16h,63s != 255h,63s).
Jun 12 00:52:19 freenas kernel: Trying to mount root from ufs:/dev/ufs/FreeNASs1a
Jun 12 00:52:19 freenas kernel: ZFS filesystem version 4
Jun 12 00:52:19 freenas kernel: ZFS storage pool version 15
Jun 12 00:52:19 freenas ntpd[1619]: ntpd 4.2.4p5-a (1)
Jun 12 00:52:20 freenas root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Jun 12 00:52:28 freenas ntpd[1620]: time reset +0.219650 s
Jun 12 00:53:22 freenas notifier: istgt not running? (check /var/run/istgt.pid).
Jun 12 00:53:22 freenas notifier: istgt not running? (check /var/run/istgt.pid).
Jun 12 00:53:22 freenas notifier: Starting istgt.
Jun 12 00:53:22 freenas notifier: istgt version 0.4 (20111008)
Jun 12 00:53:22 freenas istgt[3139]: istgt version 0.4 (20111008)
Jun 12 00:53:22 freenas notifier: normal mode
Jun 12 00:53:22 freenas istgt[3139]: normal mode
Jun 12 00:53:22 freenas notifier: LU1 HDD UNIT
Jun 12 00:53:22 freenas notifier: LU1: LUN0 file=/dev/zvol/ZPOOL01/ZVOL01, size=6597069766656
Jun 12 00:53:22 freenas notifier: LU1: LUN0 12884901888 blocks, 512 bytes/block
Jun 12 00:53:22 freenas notifier: LU1: LUN0 6144.0GB storage for iqn.2011-03.example.org.istgt:t
Jun 12 00:53:22 freenas notifier: LU1: LUN0 serial 6805ca02eb23000
Jun 12 00:53:22 freenas notifier: LU1: LUN0 read cache enabled, write cache enabled
Jun 12 00:53:22 freenas notifier: LU1: LUN0 command queuing enabled, depth 32
Jun 12 00:53:37 freenas istgt[3140]: Login from iqn.1991-05.com.hexland:media (192.168.1.4) on iqn.2011-03.example.org.istgt:t LU1 (192.168.1.5:3260,1), ISID=400001370000, TSIH=1, CID=1, HeaderDigest=off, DataDigest=off
Jun 12 01:07:57 freenas ntpd[1620]: kernel time sync status change 2001
Jun 12 23:30:01 freenas ntpd[1620]: kernel time sync status change 6001
Jun 12 23:31:07 freenas ntpd[1620]: kernel time sync status change 2001
Jun 12 23:58:58 freenas ntpd[1620]: time reset -0.465579 s
Jun 13 00:19:32 freenas ntpd[1620]: time reset -0.339976 s
Jun 13 00:40:14 freenas ntpd[1620]: time reset -0.322454 s
Jun 13 01:01:36 freenas ntpd[1620]: time reset -0.332818 s
Jun 13 01:18:38 freenas ntpd[1620]: time reset -0.266723 s
Jun 13 01:40:29 freenas ntpd[1620]: time reset -0.357423 s
Jun 13 02:02:35 freenas ntpd[1620]: time reset -0.347785 s
Jun 13 02:23:31 freenas ntpd[1620]: time reset -0.330155 s
Jun 13 02:43:11 freenas ntpd[1620]: time reset -0.324558 s
Jun 13 03:01:43 freenas ntpd[1620]: time reset -0.300611 s
Jun 13 03:26:16 freenas ntpd[1620]: time reset -0.369692 s
Jun 13 03:47:52 freenas ntpd[1620]: time reset -0.322085 s
Jun 13 04:10:13 freenas ntpd[1620]: time reset -0.334176 s
Jun 13 04:29:59 freenas ntpd[1620]: time reset -0.318691 s
Jun 13 04:52:49 freenas ntpd[1620]: time reset -0.349234 s
Jun 13 05:11:50 freenas ntpd[1620]: time reset -0.282268 s
Jun 13 05:34:03 freenas ntpd[1620]: time reset -0.317147 s
Jun 13 05:51:40 freenas ntpd[1620]: time reset -0.271044 s
Jun 13 06:10:24 freenas ntpd[1620]: time reset -0.286192 s
Jun 13 06:34:41 freenas ntpd[1620]: time reset -0.354641 s
Jun 13 06:55:50 freenas ntpd[1620]: time reset -0.296277 s
Jun 13 07:16:39 freenas ntpd[1620]: time reset -0.294856 s
Jun 13 07:41:18 freenas ntpd[1620]: time reset -0.345716 s
Jun 13 08:04:36 freenas ntpd[1620]: time reset -0.296575 s
Jun 13 08:28:25 freenas ntpd[1620]: time reset -0.331621 s
Jun 13 08:54:28 freenas ntpd[1620]: time reset -0.319070 s
Jun 13 09:13:22 freenas ntpd[1620]: time reset -0.277664 s
Jun 13 09:32:16 freenas ntpd[1620]: time reset -0.265619 s
Jun 13 09:58:39 freenas ntpd[1620]: time reset -0.330703 s
Jun 13 10:22:03 freenas ntpd[1620]: time reset -0.298091 s
Jun 13 10:47:33 freenas ntpd[1620]: time reset -0.305414 s
Jun 13 11:08:26 freenas ntpd[1620]: time reset -0.286215 s
Jun 13 11:33:18 freenas ntpd[1620]: time reset -0.308672 s
Jun 13 11:56:34 freenas ntpd[1620]: time reset -0.291557 s
Jun 13 12:19:45 freenas ntpd[1620]: time reset -0.276678 s
Jun 13 12:48:42 freenas ntpd[1620]: time reset -0.318643 s
Jun 13 13:15:04 freenas ntpd[1620]: time reset -0.299472 s
Jun 13 13:37:56 freenas ntpd[1620]: time reset -0.276879 s
Jun 13 14:04:33 freenas ntpd[1620]: time reset -0.294856 s
Jun 13 14:26:07 freenas ntpd[1620]: time reset -0.271667 s
Jun 13 14:48:55 freenas ntpd[1620]: time reset -0.255790 s
Jun 13 15:00:16 freenas istgt[3140]: istgt_lu_disk.c:4026:istgt_lu_disk_lbwrite: ***ERROR*** lu_disk_write() failed
Jun 13 15:00:16 freenas istgt[3140]: istgt_lu_disk.c:6201:istgt_lu_disk_execute: ***ERROR*** lu_disk_lbwrite() failed

 

[paleoN]

I have a suspicion this is from metadata/checksums. There's this post over on NexentaStor forums and this older thread on [post=6775]FreeNAS forum[/post]. I haven't played with iSCSI much but have you tried with a file based extent? Is the performance even less vs a device extent. I know the doc states a device extent is faster, but have you tried it?

Another thing to keep in mind:

At 90% ZFS switches from performance to space based optimization, which has massive performance implications.

See bullet 4 in ZFS Best Practices#General Storage Pool Performance Considerations. A mostly full pool is only recommended if it's read-only and even then scrub/resilver performance suffers.

Any special reason you are using iSCSI? I get better performance from CIFs than iSCSI, but my client is XP. Also, I haven't looked at any tunings for iSCSI.

 

[Hexland]

I have no real reason for iSCSI, other than the 'theoretical' speed advantage that iSCSI is supposed to provide (with a much lower protocol overhead on network traffic). I also decided to go with iSCSI because I wouldn't need to dick around with the built in server shares under WHS too much... If I connect FreeNAS to WHS via CIFS/SMB -- I'm not sure if WHS will let me use those network shares as sources for the server shares.

My final setup was to be...

1 x FreeNAS server (HP N40 Microserver) providing RAID-Z2 over 6 x 2Tb drives
1 x Windows Home Server 2011 (HP N40 Microserver) with 1 x 250Gb stock drive

The WHS machine was to be connected directly (via a dedicated Gb NIC) to the FreeNAS server, and the RAID-Z2 pool available to WHS via a single iSCSI volume.

The WHS machine would provide all of the media sharing and PC client backups, and FreeNAS would be free to be purely a NAS.



So... I just finished a 15hr scrub of the ZPOOL and it found no errors -- all is well apparently
However, I'm a little puzzled by the results of 'zfs list'

Code:

NAME             USED  AVAIL  REFER  MOUNTPOINT
ZPOOL01         7.13T      0   224K  /mnt/ZPOOL01
ZPOOL01/ZVOL01  7.13T      0  7.13T  -

The ZVOL01 was created as a 6Gb volume... but 'zfs list' now reports it as 7.13Tb. Why would a volume (with a set size) grow?

I deliberately left 1.1Tb of space on the ZPOOL01 (because I was going to set up AFP for TimeMachine backups)


Hmmm... could someone help me out here, interpreting this?

Code:

$ zpool get "all" ZPOOL01

NAME     PROPERTY       VALUE                  SOURCE
ZPOOL01  size           10.9T                  -
ZPOOL01  used           10.7T                  -
ZPOOL01  available      173G                   -
ZPOOL01  capacity       98%                    -
ZPOOL01  altroot        /mnt                   local
ZPOOL01  health         ONLINE                 -
ZPOOL01  guid           16012583193434220967   default
ZPOOL01  version        15                     default
ZPOOL01  bootfs         -                      default
ZPOOL01  delegation     on                     default
ZPOOL01  autoreplace    off                    default
ZPOOL01  cachefile      /data/zfs/zpool.cache  local
ZPOOL01  failmode       wait                   default
ZPOOL01  listsnapshots  off                    default

$ zfs get all

NAME            PROPERTY              VALUE                  SOURCE
ZPOOL01         type                  filesystem             -
ZPOOL01         creation              Tue Jun 12  0:22 2012  -
ZPOOL01         used                  7.13T                  -
ZPOOL01         available             0                      -
ZPOOL01         referenced            224K                   -
ZPOOL01         compressratio         1.00x                  -
ZPOOL01         mounted               yes                    -
ZPOOL01         quota                 none                   default
ZPOOL01         reservation           none                   default
ZPOOL01         recordsize            128K                   default
ZPOOL01         mountpoint            /mnt/ZPOOL01           default
ZPOOL01         sharenfs              off                    default
ZPOOL01         checksum              on                     default
ZPOOL01         compression           off                    default
ZPOOL01         atime                 on                     default
ZPOOL01         devices               on                     default
ZPOOL01         exec                  on                     default
ZPOOL01         setuid                on                     default
ZPOOL01         readonly              off                    default
ZPOOL01         jailed                off                    default
ZPOOL01         snapdir               hidden                 default
ZPOOL01         aclmode               passthrough            local
ZPOOL01         aclinherit            passthrough            local
ZPOOL01         canmount              on                     default
ZPOOL01         shareiscsi            off                    default
ZPOOL01         xattr                 off                    temporary
ZPOOL01         copies                1                      default
ZPOOL01         version               4                      -
ZPOOL01         utf8only              off                    -
ZPOOL01         normalization         none                   -
ZPOOL01         casesensitivity       sensitive              -
ZPOOL01         vscan                 off                    default
ZPOOL01         nbmand                off                    default
ZPOOL01         sharesmb              off                    default
ZPOOL01         refquota              none                   default
ZPOOL01         refreservation        none                   default
ZPOOL01         primarycache          all                    default
ZPOOL01         secondarycache        all                    default
ZPOOL01         usedbysnapshots       0                      -
ZPOOL01         usedbydataset         224K                   -
ZPOOL01         usedbychildren        7.13T                  -
ZPOOL01         usedbyrefreservation  0                      -
ZPOOL01/ZVOL01  type                  volume                 -
ZPOOL01/ZVOL01  creation              Tue Jun 12  0:46 2012  -
ZPOOL01/ZVOL01  used                  7.13T                  -
ZPOOL01/ZVOL01  available             0                      -
ZPOOL01/ZVOL01  referenced            7.13T                  -
ZPOOL01/ZVOL01  compressratio         1.00x                  -
ZPOOL01/ZVOL01  reservation           none                   default
ZPOOL01/ZVOL01  volsize               6T                     -
ZPOOL01/ZVOL01  volblocksize          8K                     -
ZPOOL01/ZVOL01  checksum              on                     default
ZPOOL01/ZVOL01  compression           off                    default
ZPOOL01/ZVOL01  readonly              off                    default
ZPOOL01/ZVOL01  shareiscsi            off                    default
ZPOOL01/ZVOL01  copies                1                      default
ZPOOL01/ZVOL01  refreservation        6T                     local
ZPOOL01/ZVOL01  primarycache          all                    default
ZPOOL01/ZVOL01  secondarycache        all                    default
ZPOOL01/ZVOL01  usedbysnapshots       0                      -
ZPOOL01/ZVOL01  usedbydataset         7.13T                  -
ZPOOL01/ZVOL01  usedbychildren        0                      -
ZPOOL01/ZVOL01  usedbyrefreservation  0                      -