SOLVED Suspected Raid Card Failing

[rycodge]

I'm rebuilding an older 12 drive SAN and I've ran into some interesting issues. I've replaced all the drives, all smart tests passed, memtestx86 passed, etc, etc. I've configured the drives in two 6 disk raidz2 arrays. after copying some data over, I ran a scrub on both volumes. Every drive reports a checksum error:

Code:

status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 3.69M in 2h21m with 0 errors on Mon Mar 24 19:04:02 2014
config:
 
        NAME                                            STATE    READ WRITE CKSUM
        bertha0                                        ONLINE      0    0    0
          raidz2-0                                      ONLINE      0    0    0
            gptid/9adae491-b3ac-11e3-a6aa-003048d600c2  ONLINE      0    0    37
            gptid/9c2e2f01-b3ac-11e3-a6aa-003048d600c2  ONLINE      0    0    40
            gptid/9ca9359b-b3ac-11e3-a6aa-003048d600c2  ONLINE      0    0    25
            gptid/9d08c78a-b3ac-11e3-a6aa-003048d600c2  ONLINE      0    0    55
            gptid/9d6b27d2-b3ac-11e3-a6aa-003048d600c2  ONLINE      0    0    46
            gptid/9dcb8308-b3ac-11e3-a6aa-003048d600c2  ONLINE      0    0    51
 
errors: No known data errors
 
  pool: bertha1
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub canceled on Tue Mar 25 16:01:43 2014
config:
 
        NAME                                            STATE    READ WRITE CKSUM
        bertha1                                        ONLINE      0    0    0
          raidz2-0                                      ONLINE      0    0    0
            gptid/2912b0d1-b37a-11e3-a6aa-003048d600c2  ONLINE      0    0    16
            gptid/2a41bc19-b37a-11e3-a6aa-003048d600c2  ONLINE      0    0    22
            gptid/2a9439e5-b37a-11e3-a6aa-003048d600c2  ONLINE      0    0    5
            gptid/2b0bda59-b37a-11e3-a6aa-003048d600c2  ONLINE      0    0    13
            gptid/2b4c8c68-b37a-11e3-a6aa-003048d600c2  ONLINE      0    0    11
            gptid/2baf41fe-b37a-11e3-a6aa-003048d600c2  ONLINE      0    0    18
 
errors: No known data errors

Also, the UDMA_CRC_Error_Count smart value keep rising over time on every drive. Has anyone seen anything like this before? Any insight is much appreciated. I'm going to swap the cable when it arrives later this week and I'm tempted to order a new LSI raid controller.

I've been using freenas on two SANs for two years now, rock solid besides my sata drives failing and my other SAN's raid card failing completely.

 

[rycodge]

I have resolved the problem. Long story short, my older 3ware 9690SA-4I raid card was not compatible with the new Western Digital drives I ordered. I created a raidz2 array out of some of the older drives and everything functions as expected.

As an odd side note, the scrubs took FOREVER to finish on the incompatible drives. The scrub speed was between 3-4 MB's/sec.