Mystery: "CAM status: CCB request completed with an error" on new build disappearing after testing

[mauorrizze]

Update:
my problem has not disappeared! I've installed FreeNAS on two SanDisk Cruzer Fit 16GB USB2 drives which leads sooner (or later :( ) to the following errors:

...
(da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 00 09 00 a2 00 00 28 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 34 3a 47 00 00 01 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command
...
(da0:umass-sim0:0:0:0): Error 5, Retries exhausted
(da0:umass-sim0:0:0:0): got CAM status 0x44
(da0:umass-sim0:0:0:0): fatal error, failed to attach to device
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
da0: <SanDisk Cruzer Fit 1.00> s/n 4C530001070921123024 detached
(da0:umass-sim0:0:0:0): Periph destroyed

and

NAME STATE READ WRITE CKSUM
freenas-boot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
da1p2 ONLINE 0 0 0
da0p2 FAULTED 2 214 0 too many errors

errors: No known data errors


The thumb drives themselves are alright, booting the same machine to a live linux and testing several hours (badblocks read/write) reveals no errors at all. Whereas in FreeNAS both drives are affected.
I've replaced one drive with an older Intenso stick, which, until now, hasn't have these problems. I don't have more thumb drives available right now, but I've ordered some. Until then I have no idea if this strange incompatibility is between FreeBSD and the USB controller or the SanDisk Cruzer drives.


For further infos my original post:
Hi,
I've got a problem (or not), that a reproducible error with reading or writing to the FreeNAS boot drives has been disappearing after some tests to identify the problem.

After a fresh install (current stable) on a used-bought X9SCM-F and two SanDisk Cruzer Fit 16GB USB2 drives right on the first boot during the initialization I got the CCB-Errors with read-commands on the one thumb drive and shortly after write-commands on the other drive. Once the error appeared, disk access was impossible and the errors were flooding the screen. After a reboot freenas-boot was degraded and the second stick did not show on camcontrol. So I tested the thumb drives on a linux machine (read/write-test with badblocks), no problems. I also ran memtest86+ on the X9SCM over night.

Next installation (same sticks): FreeNAS installation and first-boot-initialization went through but errors appeared shortly after. After a reboot the pool wasn't degraded (perhaps only read errors this time), but the error came back. This time I used a live linux on that machine and ran badblocks on both sticks, after a while I added a mprime/prime95 instance. After 3 hours, 0 errors.

Next installation: Initialization went through again and this time I had to start several scrubs on freenas-boot, about 4, until the error started spawning again. After that I tried a cheap USB3-PCIe-card and placed the thumb drives there, with the result of instant CAM/CCB-Error on the first access of the installation. I was not at all surprised, but I don't know if this is to be expected, even with a noname USB3 card? Anyway...

After the last installation I thought I'd clone one of the thumb drives to another one of another brand for easier identification. But during the whole dd-process: no error. A reboot and several scrubs and dd-reads later, still no errors. I'm going to test the original combination again in case I just removed the "incompatible" one, but as on the earlier try both drives were affected, my guts tell me I won't see this error again today. Maybe later...

But that's the point: how should I proceed? Be happy and start using the system (I mean it's only the boot-, not the data pool)? Or can you think of other tests or parameters I could check? I'd be happy to read your opinions, but thanks for reading!

 

[scrappy]

I seem to be having similar issues. After several failed installs due to these CCB errors, I was finally able to complete the install to one of my SanDisk Cruzer fits (have three of them and all have similar write errors). Once I booted that machine and played around for maybe 10 minutes it locked up. After restart it locked up again after a short while.

 

[mauorrizze]

Hi, could you test these drives on another PC or OS? In this thread https://forums.freenas.org/index.php?threads/strange-boot-device-problems.49757/ a third person might have issues with the Cruzer drives on FreeNAS. In my opinion it's a strange incompatibility, perhaps between FreeBSD, the USB controller, these USB drives and whatnot. In my case my two drives are working fine everywhere else, while in FreeNAS the CCB errors may still appear during a lot of reads and writes (like the installation or scrubs on freenas-boot), but not consistently at all.
I'd recommend to combine only one Cruzer with one or two drives of another brand and use them in raid 1 mode. My three different drives (Cruzer Fit, Ultra Fit and Kingston DataTraveler 3.0) are kind of stable, I had only a degraded state once after a reboot during the last two months which fixed itself over time. Unfortunately I didn't research if the Cruzer was offline.
On the prospective update to Corral I definitely gonna replace the drives with a SSD.

We should definitely convince some guide writers not to recommend the Cruzers anymore.

 

[scrappy]

Hi, could you test these drives on another PC or OS? In this thread https://forums.freenas.org/index.php?threads/strange-boot-device-problems.49757/ a third person might have issues with the Cruzer drives on FreeNAS. In my opinion it's a strange incompatibility, perhaps between FreeBSD, the USB controller, these USB drives and whatnot. In my case my two drives are working fine everywhere else, while in FreeNAS the CCB errors may still appear during a lot of reads and writes (like the installation or scrubs on freenas-boot), but not consistently at all.
I'd recommend to combine only one Cruzer with one or two drives of another brand and use them in raid 1 mode. My three different drives (Cruzer Fit, Ultra Fit and Kingston DataTraveler 3.0) are kind of stable, I had only a degraded state once after a reboot during the last two months which fixed itself over time. Unfortunately I didn't research if the Cruzer was offline.
On the prospective update to Corral I definitely gonna replace the drives with a SSD.

We should definitely convince some guide writers not to recommend the Cruzers anymore.

Yeah, I will test a few machines with these SanDisk Cruzers and see if I get the same errors. Stupid me bought three of them to keep one as a spare and the other two as mirrored boot disks. All that I have been able to successfully achieve was to burn the FreeNAS ISO to one of them.

For the sake of my sanity I have installed FreeNAS Corral onto a 2.5" spinning drive connected via eSata. There were zero installation errors and so far everything seems stable.

 

[scrappy]

I tested the same SanDisk Cruzer Fits as mirrored boot drives on my Lenovo laptop and installation went smoothly. After first boot, running an update, rebooting, and playing around with this setup for a while I can say everything seems to be working absolutely fine. This leads me to question if I either have something wrong with my MOBO (Supermicro X8DTU-F) which I find somewhat unlikely, or if there could be a setting I have enabled or disabled in BIOS causing these USB headaches.

 

[TiZu]

I've got the same errors aswell. Tried 4 different USB sticks on a SuperMicro X8SIL-F. All the same error. I was also questioning if my hardware is okay.
I get the errors instantly when booting the installer. When i'm able to install the system freezes after a while. After rebooting the usb stick cannot be found by FreeNAS.

Any solution yet?

I've also posted a thread on reddit:https://www.reddit.com/r/freenas/comments/62ge7f/cannot_install_freenas_to_usb/

 

[yottabit]

I'm having the same problem trying to add a second Cruzer Fit 16 GB into a boot mirror. Eventually during the resilvering I'll start getting these CCB errors. I've tested the second drive on a Debian system with dd:

Code:

dd if=/dev/zero of=/dev/sdb bs=2048k status=progress oflag=sync

Hums along writing at 5.4 MB/s without any problems.

I'm going to fiddle with some USB host settings in the BIOS, but I'm getting pretty annoyed that one of these drives has been working fine in the newly installed NAS (using the freshly released FreeNAS 11), while trying to add this second drive (which is the drive I actually installed from, btw!) starts erroring out.

If anyone could suggest any tuning parameter tweaks to make USB flash drives more compatible, etc., that would be great. I don't care if they're slow.

 

[Jailer]

If anyone could suggest any tuning parameter tweaks to make USB flash drives more compatible, etc., that would be great.

Tune them into the circular file and get an SSD to boot from. Even the cheapest SSD is monumentally better than USB flash drives.

 

[joeschmuck]

Just curious, exactly which model number is the SanDisk Cruzer? Are you using it in a USB 2.0 port or 3.0 port?

Also, Why run a second USB Flash drive? One works just fine. If you backup your configuration file then you will be able to restore it if you ever have a problem.

And as @Jailer said, if you have a free SATA port, just add a single small (cheap) SSD and you will be happy. Of course make a backup of your configuration file if you change the system, that should be done regardless of the boot pool medium.

 

[yottabit]

These are both plugged into USB 2.0 ports, and I've tried putting this troublesome drive into various ports to see if it made a difference. I've avoided the USB 3.0 ports, but saving any other miraculous resolution, I'm going to try one of those next.

This is a tiny NAS, built on a mini-ITX mobo with only 4x SATA ports, and all 4 are being used. So it's either spend a bunch of money to put in a PCI-E HBA and SATADOM/SSD, or get these USB boot drives to work reliably. I have used 8 GB Cruzer in my older, large NAS for years and never had any problems. I've now switched to Corsair Voyager GT 32 GB drives on that server since they're much faster. I also run those in a mirror without any problems.

The boot mirror is because I'm building this for a non-tech-savvy friend, and I don't want to have to depend on him making config backups. I want it to be easy and just work, without my having to get involved much.

The strange thing is that the activity light on the Cruzer remains active like it's writing even many minutes after FreeNAS has given up and dropped the drive from the pool. I wonder if there's a way to turn off caching and force sync writes to the boot pool.

 

[joeschmuck]

If you wanted to go the SATA route you could look into the add-on card I use to add 4 ports, move your drives to the add-on card, add a SSD to the MB port and be done with it. You can also buy a 2 port version but don't just by any add-on card, ensure compatibility.

Good luck with the USB Flash drives.

 

[yottabit]

Well I set to sync=always and I'll see if it makes a difference. If I can't get it to be reliable, I'll go with the add-in card at a later date. For now, to keep the cost low, I'd rather just use one boot drive and keep my own backup of his configuration. He won't be creating users, shares, datasets, or anything that really changes the config.

Code:

zfs get sync freenas-boot
zfs set sync=always freenas-boot
zfs get sync freenas-boot

4 more GB left to zero on the drive that's under the dd operation now, and then I'll try again.

 

[yottabit]

It finished mirroring about 85% and then I started getting the errors. But by the time I noticed them, they were actually on the source drive this time, not the destination drive. Strange.

This time, the destination drive was on USB3. I didn't bother fiddling with the BIOS yet.

So I just left it for a while, at least a half hour. The system finally crashed and rebooted. After reboot, the mirror is online with both drives just fine. ?!?!

I did a scrub just to make sure, and it completed successfully with no errors.

It seems like these drives just don't work well under FreeBSD and a high write-load. But once you manage to get the mirror established, seems fine.

 

[joeschmuck]

Sounds unstable to me. Looks like it's time for the add-on card and a SSD, if it were me of course.

 

[yottabit]

If it were me, I would. But the data integrity isn't affected. Good enough.

Sent from my Pixel XL using Tapatalk

 

[styno]

FYI I am booting 3 completely different FreeNAS systems (see sig) from pairs of mirrored SanDisk Cruzer Fit 16GB's for some time now without 'any' issues.

 

[pab49162]

I too am having strange problems with SanDisk Cruzer 16GB flash drives - mine are SanDisk Cruzer Glides. I was starting to doubt my sanity until I ran across this posting ...

I built a new FreeNAS server about 10 months ago using a Dell PowerEdge T20 Tower Server. When I did the original install, I had problem with one of the two Cruzer flash drives:


Aug 19 16:19:45 HomeNAS (da1:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 00 2c 3c 2c 00 00 80 00
Aug 19 16:19:45 HomeNAS (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Aug 19 16:19:45 HomeNAS (da1:umass-sim1:1:0:0): Retrying command
Aug 19 16:19:51 HomeNAS (da1:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 00 2c 3c 2c 00 00 80 00
Aug 19 16:19:51 HomeNAS (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Aug 19 16:19:51 HomeNAS (da1:umass-sim1:1:0:0): Retrying command
Aug 19 16:19:56 HomeNAS (da1:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 00 2c 3c 2c 00 00 80 00
Aug 19 16:19:56 HomeNAS (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Aug 19 16:19:56 HomeNAS (da1:umass-sim1:1:0:0): Error 5, Retries exhausted
...
Aug 19 16:23:01 HomeNAS (da1:umass-sim1:1:0:0): READ(10). CDB: 28 00 01 dc fc 28 00 00 80 00
Aug 19 16:23:01 HomeNAS (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Aug 19 16:23:01 HomeNAS (da1:umass-sim1:1:0:0): Retrying command
Aug 19 16:23:07 HomeNAS (da1:umass-sim1:1:0:0): READ(10). CDB: 28 00 01 dc fc 28 00 00 80 00
Aug 19 16:23:07 HomeNAS (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Aug 19 16:23:07 HomeNAS (da1:umass-sim1:1:0:0): Retrying command
Aug 19 16:23:13 HomeNAS (da1:umass-sim1:1:0:0): READ(10). CDB: 28 00 01 dc fc 28 00 00 80 00
Aug 19 16:23:13 HomeNAS (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Aug 19 16:23:13 HomeNAS (da1:umass-sim1:1:0:0): Error 5, Retries exhausted
...
Aug 19 16:23:40 HomeNAS (da1:umass-sim1:1:0:0): got CAM status 0x44
Aug 19 16:23:40 HomeNAS (da1:umass-sim1:1:0:0): fatal error, failed to attach to device
Aug 19 16:23:40 HomeNAS da1 at umass-sim1 bus 1 scbus6 target 0 lun 0
Aug 19 16:23:40 HomeNAS da1: <SanDisk Cruzer Glide 1.00> s/n 20044317530EE582FB49 detached
Aug 19 16:23:46 HomeNAS (da1:umass-sim1:1:0:0): Periph destroyed


Not sure what I exactly did, but somewhere along with way, the problems disappeared and both flash drives (mirrored) have been working perfectly. Well, until yesterday when I get an email from the server saying


The boot volume state is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state.


Looking at the log file, I saw errors very similar to above but now on the other Cruzer flash drive:


Jun 24 10:59:44 HomeNAS (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 3c 29 dc 00 00 0d 00
Jun 24 10:59:44 HomeNAS (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Jun 24 10:59:44 HomeNAS (da0:umass-sim0:0:0:0): Retrying command
Jun 24 10:59:50 HomeNAS (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 3c 29 dc 00 00 0d 00
Jun 24 10:59:50 HomeNAS (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Jun 24 10:59:50 HomeNAS (da0:umass-sim0:0:0:0): Retrying command
Jun 24 10:59:56 HomeNAS (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 3c 29 dc 00 00 0d 00
Jun 24 10:59:56 HomeNAS (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Jun 24 10:59:56 HomeNAS (da0:umass-sim0:0:0:0): Error 5, Retries exhausted
...
Jun 24 11:01:08 HomeNAS (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Jun 24 11:01:08 HomeNAS (da0:umass-sim0:0:0:0): Retrying command
Jun 24 11:01:14 HomeNAS (da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 00 00 06 38 00 00 10 00
Jun 24 11:01:14 HomeNAS (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Jun 24 11:01:14 HomeNAS (da0:umass-sim0:0:0:0): Retrying command
Jun 24 11:01:19 HomeNAS (da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 00 00 06 38 00 00 10 00
Jun 24 11:01:19 HomeNAS (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Jun 24 11:01:19 HomeNAS (da0:umass-sim0:0:0:0): Error 5, Retries exhausted
...
Jun 24 11:01:47 HomeNAS (da0:umass-sim0:0:0:0): got CAM status 0x44
Jun 24 11:01:47 HomeNAS (da0:umass-sim0:0:0:0): fatal error, failed to attach to device
Jun 24 11:01:47 HomeNAS da0 at umass-sim0 bus 0 scbus5 target 0 lun 0
Jun 24 11:01:47 HomeNAS da0: <SanDisk Cruzer Glide 1.00> s/n 20044320300EE582FAFC detached
Jun 24 11:01:52 HomeNAS (da0:umass-sim0:0:0:0): Periph destroyed


Thinking the other flash drive had a problem, I bought another identical Cruzer flash drive and did a Replace function this morning. The replace function started out fine but when I went to install GRUB on it, I started getting very similar errors. :(

In parallel to this effort, I used Flash Drive/Card Tester v1.14 to check the "failed" flash drive. Of course, the program did not find any issues with that flash drive.

Not having anything to lose, I tried a second replace command using the "failed" flash drive that tested good and again had the same type of problems as shown above. I then tried a couple more times using the USB2 ports instead of the USB3 ports but again no success.

At this point, I decided to try a different brand of flash drives, I found an old 16GB Maxell flash drive and as you can probably guess, it worked right away (replace, scrub and GRUB install). So far, after about 4 hours, things are still working fine.

While it is possible that the new SanDisk Cruzer 16GB drive and the"failed" one that tested good are both bad, it seems like a bit of stretch. This is especially true since other people seem to be having similar strange problems with SanDisk Cruzer 16GB flash drives.

Hopefully, this post might help someone else starting to doubt their sanity when strange things start happening.

Any thoughts or advice on these strange issues would be greatly appreciated. Thanks.

 

[andstein85]

Was having these same problems using 2x SanDisk 32GB Cruzer Blades that I picked them up for $4 each from Walmart... As soon as I moved the System Dataset from freenas-boot to one of my other zpools, the problems went away. Just to be sure though, I added a 3rd cruzer blade to the boot mirror and will see how long this lasts. The weird part is I ran this nas with only one of these sticks(no mirror) for nearly a year with no issues and it really only started becoming a problem after freenas 11u2.

Food for thought.

Will update if these 3 sticks show signs of error with the system dataset running on a spinning disk mirror.

 

[Michael Sierchio]

sysctl kern.cam.ada.write_cache=0

 

[PnoT]

I'm also running a Sandisk Cruzer on an X8SIL-F and have had nothing but trouble out of these things as other members have attested to. I'm going to dump the thing and run from another brand to see if that works for me. The upgrade to 11.x is causing me headaches right now but the 9.x runs perfectly fine so I'm not sure if it's the upgrade or just writing addition data to the USB is causing the problems.