Photo files corrupted randomly

[shnurov]

I am trying to pin-point why some of my images that I copy from an external HDD or SD card become corrupted at random.

I have two backups for this - thankfully the data is safe :)

This is a screenshot of what I see once I render the previews in Lightroom CC 2015.1
https://www.evernote.com/shard/s28/...1023fc-ad20-4d62-a990-45a498edc7d6/skitch.png

I have AFP file-sharing enabled as I have 4 Mac's that need to access the server. Running 9.3 Stable version.

I am attaching the debug file.

Anybody has any ideas?

Thanks!

 

[Nick2253]

I'm getting an access denied error. Can you upload to Imgur or another image sharing service?

Usually, if you are getting data corruption, it is because something in the pipe is corrupting the data. That's usually the HDD, the cable, the controller, the system memory/CPU, the network adapter, the network cable, the switch, the receiving network cable, the receiving network adapter, etc.

My strong guess is that it is due to the client side, since ZFS is fairly robust at detecting this kind of corruption in the pool.

I've taken a quick cursory glance at your debug, and the following pop out to me:

• Your drives are getting way too hot. Three have a peak lifetime temperature is above 50C. Not good. :(
  
• Your drives have a high load cycle count, which indicates they are aggressively parking.
• SMART data isn't indicating any major problems with bad sectors, so it's probably not the HDDs corrupting the data.
• ZFS isn't indicating any checksum errors.

 

[shnurov]

Thanks for checking in.

Here's a dropbox link:
https://www.dropbox.com/s/6mlbb4xohxhp0i7/corrupt photos.png?dl=0

Basically it could be linked to anything.
All the parts are new - including the switch and the router and seems to work OK.
I have 3x TS259 QNAP as well and I've never had those issues there - so I don't think it would be linked to something with the network or client itself

I changed the fans in my box to high performance Noctua's, but slowed down the 3000rpm ones as it was kind of engine take-off sound - I will increase the speed. Is there any quick way to check the HDD temp?

How can I fix the aggressive parking?

 

[gpsguy]

For HDD temps, etc. check out the scripts written by @Bidule0hm See - https://forums.freenas.org/index.ph...s-hdd-cpu-temps-and-hdd-identification.27365/

 

[Robert Trevellyan]

How can I fix the aggressive parking?

You can burn a copy of the Ultimate Boot CD and use wdidle3.

 

[SweetAndLow]

If you run a scrub do you see data getting fixed?

Code:

Aug 23 23:09:59 freenas afpd[9415]: dsi_stream_read: len:-1, Operation timed out
Aug 23 23:09:59 freenas afpd[9415]: dsi_stream_read: len:0, unexpected EOF
Aug 24 00:09:09 freenas afpd[9415]: read: Operation timed out

I have a feeling this is a problem with how you are using afp. Afp has been deprecated by apple in favor of cifs so I suggest using that.

 

[rogerh]

It does look more like incomplete file transfers rather than random data corruption.

 

[shnurov]

Thanks for all the feedback - I will look into the scrip at a later time as it I am currently doing a scrub, last one was on August 9th.

Running it now...

Code:

 91.3G scanned out of 3.62T at 216M/s, 4h45m to go                       
        0 repaired, 2.46% done   

It's not the first time I read about Apple going away from AFP, but CIFS can only give my 65-75MB/s in terms of speed! :( Given that when I complete wedding jobs I have to move ~60-100gb's from the SD cards to the server the 115MB/s speeds that AFP was giving me was great! I guess I will move back to Samba and see how it goes.
I will create a dataset of 300GB for Time Machine backups with AFP & create another one for SMB sharing for the rest of my files.

Regarding HDD parking - I thought it only affected the WD Green's.

 

[rogerh]

A quick look finds:

https://bugs.freenas.org/issues/10284

which seems to be the same thing. If so, CIFS should work alright!

PS, I haven't read your debug file, but the head parking thing also affects some rogue, early issue, WD reds.

 

[rogerh]

A quick look finds:

https://bugs.freenas.org/issues/10284

which seems to be the same thing. If so, CIFS should work alright!

PS, I haven't read your debug file, but the head parking thing also affects some rogue, early issue, WD reds.

Edit: if I read the bug report correctly, what is happening is that the files have been correctly recorded in FreeNAS, but opening them again over AFP introduces errors. Is it possible that this is what you are seeing?

Edit2: In case it is the same bug, try copying one of these files from FreeNAS to your local Mac in a finder folder, and then opening it. Apparently this avoided the corruption in the case in the bug report.

 

[Nick2253]

Your problem with temperature might be your DS380. I have the DS380, and I was having a bear of a time trying to get my temps down. My solution was to drill a number of holes in the back of the drive cage to improve airflow over the drive. That worked wonders! Drive temps dropped 10C (but some of that's probably due to the rest of my fixes, which only made a difference with the improved airflow).

If you're willing, it's worth it, because the DS380 has terrible drive airflow without some more exhaust in the back of the cage.

 

[Nick2253]

It's not the first time I read about Apple going away from AFP, but CIFS can only give my 65-75MB/s in terms of speed! :( Given that when I complete wedding jobs I have to move ~60-100gb's from the SD cards to the server the 115MB/s speeds that AFP was giving me was great! I guess I will move back to Samba and see how it goes.

You could always use an NFS mount.

My guess is that you weren't really getting 115MB/s. That's seriously fast, both in terms of reading the data from an SD card and being within 10MB/s of saturating gig-e. That may be part of your problem with corrupted data: the protocol says everything delivered correctly and super fast, but it really didn't.

I'm able to saturate a Gig-e connection in both read and write over CIFS, but only if I'm writing/reading from a fast enough client. Only my network, the only device I have that can do that is my main computer's SSD. All the other HDDs are too slow. (I suppose a RAM drive would do it too :p). I spent a while when I first set up FreeNAS trying to figure out why my speeds were so "slow" (in the 60-70 MB/s range) when my limiting factor was the client, not FreeNAS.

 

[shnurov]

Whoa, that sounds like exactly my problem. I will change to open everything under CIFS now & delete the AFP share.

The DS380 - I don't mind drilling holes in it - but I already have the most expensive Noctua fans in it - even got a 140mm fit in the exhaust :D
I will try to increase the speed a little first then get back to drilling & modding.

The 117MB/s was the fastest I could do from my retina 15" MBP with a thunderbolt/ethernet adapter & reading/writing from local SSD. With the SD cards I'm looking at about 70MB/s. Doing the same transfer from a similar specced SSD on a X79 platform with the same GBit connection I was only getting 65-70MB/s via Samba.

So list for today :
- break up into two shares, one 300GB for AFP (TimeMachine backups) & the rest of 7TB for photo storage
- increase fan speeds to 55-60% (of 3000rpm) to get slightly lower temps
- figure out how to install scripts & get the HDD/CPU temps
- figure out why I can't finish configuring OpenVPN :(

 

[Darren Myers]

next step would be doing some 10Gb if you have an old Mac pro (2009 or so) or a diff PC and your FreeNAS server :P

 

[shnurov]

So the 300GB AFP share for TimeMachine is set up.
The SMB one on the other hand is having some permission issues - I think I have to go back to see how to configure guest accesses.

next step would be doing some 10Gb if you have an old Mac pro (2009 or so) or a diff PC and your FreeNAS server :p

Not sure if you're hinting at making a 10GBit switch or doing another test?

 

[Darren Myers]

So the 300GB AFP share for TimeMachine is set up.
The SMB one on the other hand is having some permission issues - I think I have to go back to see how to configure guest accesses.

Not sure if you're hinting at making a 10GBit switch or doing another test?[/QUOTE]
if you keep running into speed issues over 1Gb LAN :p but its very simple to do direct connect 10Gb if you really want to try and saturate the drives, but then youre still limitd by the speed of spinning rust

So the 300GB AFP share for TimeMachine is set up.
The SMB one on the other hand is having some permission issues - I think I have to go back to see how to configure guest accesses.

Why SMB, why not use CIFS shares for both Mac and Windows? As for the AFP share, in order for timemachine to backup it needs AFP

 

[shnurov]

Not sure if you're hinting at making a 10GBit switch or doing another test?

if you keep running into speed issues over 1Gb LAN :p but its very simple to do direct connect 10Gb if you really want to try and saturate the drives, but then youre still limitd by the speed of spinning rust

Why SMB, why not use CIFS shares for both Mac and Windows? As for the AFP share, in order for timemachine to backup it needs AFP[/QUOTE]

My drives tested at ~220-240MB/s read/write. 10GBit is for later!

I think I just had a small revelation... CIFS != SMB...
Hence why I'm in 'Connect to Server' and type "smb://freenas/PhotoSet" it gives me permission errors.

 

[Darren Myers]

My drives tested at ~220-240MB/s read/write. 10GBit is for later!

I think I just had a small revelation... CIFS != SMB...
Hence why I'm in 'Connect to Server' and type "smb://freenas/PhotoSet" it gives me permission errors.

lol, on my Macs i connect to server CIFS://<IP ADDRESS> and BAM!

 

[SweetAndLow]

You can use the smb prefix just like you use the cifs prefix. Both sound work just fine.

 

[Ericloewe]

AFP has had some mysterious problems with big-name software. Adobe's various tools and Microsoft Office among them.

Nobody knows what the hell is going on, but it seems related to OS X's file manipulation API (the one the software uses to show open/save dialog boxes and stuff).

You can use the smb prefix just like you use the cifs prefix. Both sound work just fine.

One is SMB1, the other is SMB2/3. I don't remember which one goes where.