SOLVED Best way to expand pool with larger HDDs

[NasKar]

I have 6 4TB drive in Z2. My motherboard has no available sata connectors but does have a PCI Express 3.0 x16 slot open.
#1 Add the larger drives to the pool one at a time and let it resilver till all 6 drives are replaced. Downside is it degrades the pool.
#2 Add a SATA card to the existing motherboard and resilver each replacement drive. Advantage - no degradation of pool . Docs
#3 Use an old computer with non ECC memory to create a new 6 drive z2 pool and transfer the data with replication from the old to the new then swap out the drives. Advantage - no additional purchase needed but more complicated.

Which would be the best way?
If option #2 and recommendation for an inexpensive Sata card?

 

[joeschmuck]

You know, I like option #3 however I would run MemTest86 on the old computer and if it passes, then build up the server. You don't need to use replication, you can just copy data over if you like but since this option will take the longest period of time to transfer your data, replication is the best option. Use it to learn something new.

Option #2 is the best option in my opinion given this scenario however this option means you need to spend some money, or borrow a SATA card from a friend. Look in my signature for the IOCREST SATA card if you want to buy a good compatible card. You can buy a lesser model and it may work fine however I'll promote what I have personally tested.

 

[danb35]

RAIDZ2? Option 1.

 

[Inxsible]

The whole point of RAIDZ2 is that it gives you 2 drive redundancy, so the pool being in DEGRADED state is something that RAIDZ2 can easily live with. Just don't pull more than 1 drive at the same time. I'd just go with Option 1 and keep replacing 1 drive at a time.

 

[farmerpling2]

I use #1, though #2 with replace is safer.

If I had a spare HBA/PCI port, I'd go with #2.

 

[danb35]

If I had a spare HBA/PCI port, I'd go with #2.

Likewise, but I wouldn't think it's worth buying an add-in card just to do that.

 

[NasKar]

@danb35 and @Inxsible I appreciate the input. I'm leaning toward option #1. I could get the proceess started as I currently have only 5/6 8TB replacement drives. I'm waiting for the next sale <G>. @joeschmuck I would like to learn about replication, maybe I can use the old 4TB drives in a backup system using replication. Would Z1 be the best setup for a backup? My last thought is to get the sata controller and an SSD to it for the FreeNAS OS. Would need a 4 pin molex to SATA power adapter as my powersupply doesn't have any sata power ports left. Not sure it would all fit in my Fractal Node 304 case.

 

[farmerpling2]

Playing the bad guys advocate and going paranoid mode.

You run Z2 to minimize the possibility of the RAID failure. Taking a drive out of service reduces redundancy.

For a company, #1 is not an option. Downtime/recovery costs too much. Do #2.

#2 Is a shutdown, install card, reboot (if Murphy stays away). After that just do a replace and away it goes! Do one disk at a time.

 

[farmerpling2]

I would like to learn about replication, maybe I can use the old 4TB drives in a backup system using replication. Would Z1 be the best setup for a backup?

You should do the second system with replication for a backup BEFORE you do any of this. Purely as a backup. You can learn about replication, do it a few times and not hurt anything.

 

[Inxsible]

Would need a 4 pin molex to SATA power adapter as my powersupply doesn't have any sata power ports left. Not sure it would all fit in my Fractal Node 304 case.

I recently bought 2 of these to connect 2 of my drives as I too had a similar issue with my Node 304. My PSU (Corsair CS450M) came with only 2 SATA power cables and I could connect only 4 drives. I connected the other 2 using the Molex 4 pin cable (that came with the PSU) and added these adapters on them

 

[siconic]

I run Z3 and just did an upgrade. I offlined two disks, and replaced. The time it took was 24 hours per occurence, so I did not think it likely 2 disks would fail in that period of time (Raid Z3 = Tripple disk parity, so offline 2, would require two more disk failures to lose data). But bear that in mind with your decision making. If you have a lot of data, and slow disks, and it will take more than 24 hours to resilver, it may be a bit of a risk. In my opinion, if it were to estimate the time to resilver to be longer than 72 hours and having 11 disks, I would not risk degrading my RAID. 72+ hours is a long time to ask for a disk failure.

In your case with 6 disks, multi disk failure becomes statistically less likely, but is still a factor. So, my two.

 

[NasKar]

Update: I have just installed the last 8 TB HHD to enlarge my pool and it's resilvering. The storage space in my pool show the increase space now. These drives seem to run 2-4 degrees hotter which makes sense 30-34 C. The only glitch I ran into, is the Fractal Design Node 304 HDD holders don't fit the new drives as the bottom holes have moved and I can only use 2/4 mounting screws. Fractal Design will send new ones if I pay the shipping but they are out of stock till next month. Very nice considering it's out of warranty.

 

[NasKar]

@joeschmuck I purchased the IOCREST SI-PEX40062 4 port SATA PCI-E card to run an SSD to boot FreeNAS and possible extra port to replace drives if needed in future. Your system specs mention IOCREST(in pass-thru for NAS Drives). What is "in pass-thru for NAS Drives" mean? My system is working but I get errors in the GRUB loading.... Welcome to GRUB! menu.
Could these errors happen if I didn't prepare the SSD properly? I just installed FreeNAS 11U4 from a USB install stick on the new SSD.

 

[joeschmuck]

@NasKar What do you mean by

My system is working but I get errors in the GRUB loading.... Welcome to GRUB! menu.

Is FreeNAS running fine or do you mean that your system powers up okay?

Off the top of my head I would just say that FreeNAS does not recognize some of the hardware you have installed. It could be a sound card, Serial port, etc... Please post the output of dmesg and maybe that will show something, but if FreeNAS is running then all should be fine.

Your system specs mention IOCREST(in pass-thru for NAS Drives). What is "in pass-thru for NAS Drives" mean?

I run ESXi on my computer hardware and then FreeNAS run in a virtual machine. In order to provide the virtual machine direct access to the hard drives I need to pass the controller cards through to the virtual machine and this is called "pass-thru". In this situation the main PC nor other virtual machines can see or access the controllers and hard drives, only FreeNAS can. However I did test these out running FreeNAS on the bare metal machine and they worked perfectly.

Could these errors happen if I didn't prepare the SSD properly?

If FreeNAS is running then you didn't install it wrong. It either works or it doesn't. You could have a hardware issue or like I said, it could just be some hardware that FreeNAS doesn't recognize or doesn't have drivers for.

 

[NasKar]

Is FreeNAS running fine or do you mean that your system powers up okay?

FreeNAS is running fine. Here is dmesg.today

Code:

Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-STABLE #0 r321665+25fe8ba8d06(freenas/11.0-stable): Mon Sep 25 06:24:11 UTC 2017
	root@gauntlet:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
CPU: Intel(R) Core(TM) i3-4130T CPU @ 2.90GHz (2893.60-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fdafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17985175552 (17152 MB)
avail memory = 16521682944 (15756 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff80fa9d40, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
padlock0: No ACE support.
acpi0: <ALASKA A M I> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: _OSC returned error 0x10
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
ahci0: <Marvell 88SE9235 AHCI SATA controller> port 0xe050-0xe057,0xe040-0xe043,0xe030-0xe037,0xe020-0xe023,0xe000-0xe01f mem 0xf7340000-0xf73407ff irq 16 at device 0.0 on pci1
ahci0: AHCI v1.00 with 4 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
xhci0: <Intel Lynx Point USB 3.0 controller> mem 0xf7400000-0xf740ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.1 (no driver attached)
ehci0: <Intel Lynx Point USB 2.0 controller USB-B> mem 0xf7414000-0xf74143ff irq 16 at device 26.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pcib2: [GIANT-LOCKED]
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci2: <ACPI PCI bus> on pcib3
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xd000-0xd01f mem 0xf7200000-0xf727ffff,0xf7280000-0xf7283fff irq 18 at device 0.0 on pci2
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: d0:50:99:5b:da:e4
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci3: <ACPI PCI bus> on pcib4
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xc000-0xc01f mem 0xf7100000-0xf717ffff,0xf7180000-0xf7183fff irq 19 at device 0.0 on pci3
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: d0:50:99:5b:da:e5
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci4: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 19 at device 0.0 on pci4
pci5: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> port 0xb000-0xb07f mem 0xf6000000-0xf6ffffff,0xf7000000-0xf701ffff irq 19 at device 0.0 on pci5
vgapci0: Boot video device
ehci1: <Intel Lynx Point USB 2.0 controller USB-A> mem 0xf7413000-0xf74133ff irq 23 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci1: <Intel Lynx Point AHCI SATA controller> port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf020-0xf03f mem 0xf7412000-0xf74127ff irq 19 at device 31.2 on pci0
ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
ahcich6: <AHCI channel> at channel 2 on ahci1
ahcich7: <AHCI channel> at channel 3 on ahci1
ahcich8: <AHCI channel> at channel 4 on ahci1
ahcich9: <AHCI channel> at channel 5 on ahci1
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
ichwd0: <Intel Lynx Point watchdog timer> on isa0
wbwd0: <Nuvoton NCT6776 (0xc3/0x33) Watchdog Timer> at port 0x2e-0x2f on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcafff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
freenas_sysctl: adding account.
freenas_sysctl: adding directoryservice.
freenas_sysctl: adding middlewared.
freenas_sysctl: adding network.
freenas_sysctl: adding services.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
nvme cam probe device init
usbus0: 5.0Gbps Super Speed USB v3.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 480Mbps High Speed USB v2.0
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen1.1: <Intel EHCI root HUB> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <Intel EHCI root HUB> at usbus2
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub2: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub0: 21 ports with 21 removable, self powered
ugen0.2: <American Megatrends Inc. Virtual Hub> at usbus0
uhub3 on uhub0
uhub3: <7-port Hub> on usbus0
ugen2.2: <vendor 0x8087 product 0x8000> at usbus2
uhub4 on uhub2
uhub4: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.05, addr 2> on usbus2
ugen1.2: <vendor 0x8087 product 0x8008> at usbus1
uhub5 on uhub1
uhub5: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.05, addr 2> on usbus1
uhub5: 6 ports with 6 removable, self powered
uhub4: 8 ports with 8 removable, self powered
uhub3: 5 ports with 5 removable, self powered
ugen0.3: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus0
ukbd0 on uhub3
ukbd0: <Keyboard Interface> on usbus0
kbd2 at ukbd0
ugen0.4: <American Power Conversion Back-UPS BR1000G FW868.L1 .D USB FWL1> at usbus0
ugen0.5: <Microsoft Comfort Curve Keyboard 2000> at usbus0
ukbd1 on uhub0
ukbd1: <Microsoft Comfort Curve Keyboard 2000, class 0/0, rev 2.00/1.73, addr 4> on usbus0
kbd3 at ukbd1
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <ADATA SU700 B170206b> ACS-2 ATA SATA 3.x device
ada0: Serial Number 2H19200xxxxx
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 114473MB (234441648 512 byte sectors)
ada1 at ahcich4 bus 0 scbus4 target 0 lun 0
ada1: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
ada1: Serial Number VK1Axxxx
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 7630885MB (15628053168 512 byte sectors)
ada2 at ahcich5 bus 0 scbus5 target 0 lun 0
ada2: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
ada2: Serial Number VK1Axxxx
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 7630885MB (15628053168 512 byte sectors)
ada3 at ahcich6 bus 0 scbus6 target 0 lun 0
ada3: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
ada3: Serial Number VK19xxxx
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 7630885MB (15628053168 512 byte sectors)
ada4 at ahcich7 bus 0 scbus7 target 0 lun 0
ada4: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
ada4: Serial Number VK19xxxx
ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 7630885MB (15628053168 512 byte sectors)
ada5 at ahcich8 bus 0 scbus8 target 0 lun 0
ada5: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
ada5: Serial Number VK1Axxxx
ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 7630885MB (15628053168 512 byte sectors)
ada6 at ahcich9 bus 0 scbus9 target 0 lun 0
ada6: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
ada6: Serial Number VJH8xxxx
ada6: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 7630885MB (15628053168 512 byte sectors)
random: unblocking device.
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC-low" frequency 1446797802 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/default []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
ipmi0: IPMI device rev. 1, firmware rev. 0.22, version 2.0
ipmi0: Number of channels 2
ipmi0: Attached watchdog
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI> UCP/8/48/0x3f8<EDG,THR,REA,WRI,INV,QUA,PRC> UCF/1/48/0x60<REA,WRI>
ums0 on uhub3
ums0: <Mouse Interface> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
uhid0 on uhub0
uhid0: <Microsoft Comfort Curve Keyboard 2000, class 0/0, rev 2.00/1.73, addr 4> on usbus0
igb0: link state changed to UP
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
bridge0: Ethernet address: 02:b6:00:93:5d:00
bridge0: link state changed to UP
igb0: promiscuous mode enabled
epair0a: Ethernet address: 02:ff:e0:00:05:0a
epair0b: Ethernet address: 02:ff:30:00:06:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
igb0: link state changed to DOWN
epair0a: promiscuous mode enabled
epair1a: Ethernet address: 02:ff:e0:00:06:0a
epair1b: Ethernet address: 02:ff:30:00:07:0b
epair1a: link state changed to UP
epair1b: link state changed to UP
epair1a: promiscuous mode enabled
epair2a: Ethernet address: 02:ff:e0:00:07:0a
epair2b: Ethernet address: 02:ff:30:00:08:0b
epair2a: link state changed to UP
epair2b: link state changed to UP
epair2a: promiscuous mode enabled
epair3a: Ethernet address: 02:ff:e0:00:08:0a
epair3b: Ethernet address: 02:ff:30:00:09:0b
epair3a: link state changed to UP
epair3b: link state changed to UP
epair3a: promiscuous mode enabled
igb0: link state changed to UP
epair4a: Ethernet address: 02:ff:e0:00:09:0a
epair4b: Ethernet address: 02:ff:30:00:0a:0b
epair4a: link state changed to UP
epair4b: link state changed to UP
epair4a: promiscuous mode enabled
epair5a: Ethernet address: 02:ff:e0:00:0a:0a
epair5b: Ethernet address: 02:ff:30:00:0b:0b
epair5a: link state changed to UP
epair5b: link state changed to UP
epair5a: promiscuous mode enabled
epair6a: Ethernet address: 02:ff:e0:00:0b:0a
epair6b: Ethernet address: 02:ff:30:00:0c:0b
epair6a: link state changed to UP
epair6b: link state changed to UP
epair6a: promiscuous mode enabled
epair7a: Ethernet address: 02:ff:e0:00:0c:0a
epair7b: Ethernet address: 02:ff:30:00:0d:0b
epair7a: link state changed to UP
epair7b: link state changed to UP
epair7a: promiscuous mode enabled
epair8a: Ethernet address: 02:ff:e0:00:0d:0a
epair8b: Ethernet address: 02:ff:30:00:0e:0b
epair8a: link state changed to UP
epair8b: link state changed to UP
epair8a: promiscuous mode enabled

If FreeNAS is running then you didn't install it wrong. It either works or it doesn't. You could have a hardware issue or like I said, it could just be some hardware that FreeNAS doesn't recognize or doesn't have drivers for.

The only hardware change is the SATA card, no errors in GRUB with USB stick boot.

 

[joeschmuck]

The only hardware change is the SATA card, no errors in GRUB with USB stick boot.

And again I want to be very clear here... (maybe you could use a few more decriptive words so i understand and do not make any assumptions) are you saying that with the SATA card installed and when booting off of the USB flash drive (using the exact same FreeNAS version) that you do not have any errors?

If you are saying this then I would ask you to move the SSD to one of the motherboard SATA ports (typically where a boot device would be anyway) and plug the spinning rust drive into the controller card. Reboot it, see if the problem clears. If it fails to clear then I would reinstall FreeNAS. If that fails too then I'd try a different boot drive, even a spinnign rust one just for testing. I'm grasping at straws but it could be the SSD too. I'm not saying it's a bad thing, just a possible cause. Sometimes you need to think outside the box.

"dmesg" didn't indicate any issues at all, in fact it looks good. I can see all the SATA controllers and the ports, no other errors. I don't see any reason that you can't live with the error messages if all is working, but again, I'd move the SATA connections around first. Myself, I would have the boot device plugged into SATA0 port. It's the first device the computer sees when it trys to boot.

Time for bed for this old man, I get up at 4AM to head in to work then work for 10 to 12 hours. Let me know what you find out.

 

[NasKar]

And again I want to be very clear here... (maybe you could use a few more decriptive words so i understand and do not make any assumptions) are you saying that with the SATA card installed and when booting off of the USB flash drive (using the exact same FreeNAS version) that you do not have any errors?

With the SSD plugged into the sata card and no USB Stick I get the errors in Grub menu
With the SSD plugged into the sata card and the USB Stick inserted I don't get errors if I boot to the USB stick in the BIOS
With the SSD plugged into the sata card and the USB Stick inserted booting to the SSD in the bios it boots the USB Stick instead and I get the Grub menu errors
I switch the HDD sata cable with the SSD it turned out to be ada5 and adjusted the bios to boot off the SSD connected to the motherboard sata port and I no longer get Grub Errors.

In conclusion it must be that Grub gives errors if your not booting off a sata port on the motherboard. Thanks for your help as usual.

 

[joeschmuck]

Many motherboards do not like to boot from add-on cards, or boot very well, GRUB shouldn't have had an issue with it but then again what do I know. I figured that moving the SSD to the motherboard might solve the issue.

When you installed FreeNAS, did you use UEFI or Legacy/BIOS?
If you used UEFI, reinstall using Legacy next time and maybe the problem will go away.

 

[NasKar]

Many motherboards do not like to boot from add-on cards, or boot very well, GRUB shouldn't have had an issue with it but then again what do I know. I figured that moving the SSD to the motherboard might solve the issue.

When you installed FreeNAS, did you use UEFI or Legacy/BIOS?
If you used UEFI, reinstall using Legacy next time and maybe the problem will go away.

I used Legacy/Bios.