SOLVED smbd Crashing (Signal 4) on File Delete

Status
Not open for further replies.
JR Gonzalez

JR Gonzalez

Explorer
Joined
Aug 29, 2014
Messages
64
Using the nounix option for mount.cifs seems to fix it. Seems like there is a unix extensions issue somewhere. I'm guessing on the FreeNAS side. I have my FreeNAS box working as an AD DC so I'm not really sure if that has anything to do with it because I'm not familiar with how the others (above) are running their FreeNAS box. I'm guessing some here are using basic user authentication. So if you're looking for a workaround use mount.cifs with -o nounix. I've given up on using the users home dirs. on the FreeNAS box on a Linux box automounting with pam_mount .... I'll just go ahead and mount their FreeNAS home dir in /mnt/DOMAIN/user and symlink it with force 0700 permissions.
 
JR Gonzalez

JR Gonzalez

Explorer
Joined
Aug 29, 2014
Messages
64
Well... seems like vfs_streams_xattr is enabled by default for all newly created shares. There's your reason. Go to advanced mode and remove it from the vfs modules.
 
ewhac

ewhac

Contributor
Joined
Aug 20, 2013
Messages
177
teH gOoglE sez:
Name
vfs_streams_xattr — Store alternate data streams in posix xattrs

Synopsis
vfs objects = streams_xattr

DESCRIPTION
This VFS module is part of the samba(7) suite.

The vfs_streams_xattr enables storing of NTFS alternate data streams in the file system. As a normal posix file system does not support the concept of multiple data streams per file, the streams_xattr module stores the data in posix extended attributes (xattrs). The name of these attributes is user.DosStream."ADS-NAME".

The file system that is shared with this module enabled must support xattrs.

Please note that most file systems have severe limitations on the size of xattrs. So this module might work for applications like IE that stores small zone information in streams but will fail for applications that store serious amounts of data in ADSs.
Click to expand...

Some very brief Googling on the subject of Samba and xattrs suggests changing how xattrs are handled/stored could lead to other problems...
 
ewhac

ewhac

Contributor
Joined
Aug 20, 2013
Messages
177
Some further Googling this evening suggests that vfs_streams_xattr and kernel oplocks do not interact at all well. This bug report is from 2010, but the commentary suggests that a fix would be complicated, and is unlikely to be addressed unless someone with dire need makes it happen:

https://bugzilla.samba.org/show_bug.cgi?id=7537

Oplocks are enabled in smb4.conf.

So. Should one disable oplocks on a per-share basis? Should one remove streams_xattr from the share's vfs modules? Should one replace it with streams_depot, and then establish a depot directory for the alternate streams?
 
JR Gonzalez

JR Gonzalez

Explorer
Joined
Aug 29, 2014
Messages
64
Tried disabling oplocks earlier thinking that may be the reason. So far without streams_xattr disabled I haven't ran into any issues. I will try your other suggestions tho. I think I tried xattr_tdb but that had a bad effect.
 
B

barbierimc

Dabbler
Joined
Jun 25, 2016
Messages
22
So, I've been having the same problem on a fresh build.

When I try to delete some files on a debian (jessie) system on a CIFS share with windows permissions set, the rm command hangs. I also see the same signal 4 crash.

Permissions all seem to be fine, and it was only 'certain' files.

Eventually confirmed that this is only with files originating from the internet which contain alternate data streams (added by IE, others? to identify 'files downloaded from the internet')

Running strace on the rm command confirms that the command 'unlinkat(AT_FDCWD, "/shares/Temp/.....' hangs during removal.

Checking the log shows signal 4 crash messages about every 30 seconds until the rm command is killed

Manually removing the alternate data stream from the file resolves this issue.

Removing streams_xattr from the shares also resolves this problem.

I've now removed streams_xattr from all my CIFS shares.

If only I'd found this thread before I wasted several hours tracking down this issue!

Happy to raise a bug report but not sure if this is a FreeNAS or debian bug at this stage. I can provide more info (system, logs, etc) if required (Actually I'll link to this bug report I just found, sounds the same https://bugs.pcbsd.org/issues/15808)

I hope this saves someone a few wasted hours of their life.
 
Last edited:
anodos

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
barbierimc said:
So, I've been having the same problem on a fresh build.

When I try to delete some files on a debian (jessie) system on a CIFS share with windows permissions set, the rm command hangs. I also see the same signal 4 crash.

Permissions all seem to be fine, and it was only 'certain' files.

Eventually confirmed that this is only with files originating from the internet which contain alternate data streams (added by IE, others? to identify 'files downloaded from the internet')

Running strace on the rm command confirms that the command 'unlinkat(AT_FDCWD, "/shares/Temp/.....' hangs during removal.

Checking the log shows signal 4 crash messages about every 30 seconds until the rm command is killed

Manually removing the alternate data stream from the file resolves this issue.

Removing streams_xattr from the shares also resolves this problem.

I've now removed streams_xattr from all my CIFS shares.

If only I'd found this thread before I wasted several hours tracking down this issue!

Happy to raise a bug report but not sure if this is a FreeNAS or debian bug at this stage. I can provide more info (system, logs, etc) if required (Actually I'll link to this bug report I just found, sounds the same https://bugs.pcbsd.org/issues/15808)

I hope this saves someone a few wasted hours of their life.
Click to expand...

streams_xattr allows for storing "zone identifier" metadata as well NTFS alternate datastreams and DOS attributes in filesystem extended attributes. Unfortunately, the amount of information you can store in an xattr is fairly limited (64KB?). streams_xattr works fairly well most of the time... until you have files with enormously large amounts of data stored in an ADS (usually movies with subtitles and/or malware). So if you want your malware to work properly, you should look into using a different method of storing ADS
 
B

barbierimc

Dabbler
Joined
Jun 25, 2016
Messages
22
anodos said:
streams_xattr allows for storing "zone identifier" metadata as well NTFS alternate datastreams and DOS attributes in filesystem extended attributes. Unfortunately, the amount of information you can store in an xattr is fairly limited (64KB?). streams_xattr works fairly well most of the time... until you have files with enormously large amounts of data stored in an ADS (usually movies with subtitles and/or malware). So if you want your malware to work properly, you should look into using a different method of storing ADS
Click to expand...

Thanks for the feedback. I fully appreciate the intent of the ADS and the size limitations. In this case, the only information in the alternate stream is the "zone identifier" metadata. As the size of this alternate stream is only 26 bytes (see attached, << 64KB) then I still maintain that there is a problem with the use of streams_attr on CIFS shares on FreeNAS and the rm command on debian 8.5. I'm not sure which component has the actual problem.
 

Attachments

  • ADS.PNG
    ADS.PNG
    3.4 KB · Views: 372
anodos

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
barbierimc said:
Thanks for the feedback. I fully appreciate the intent of the ADS and the size limitations. In this case, the only information in the alternate stream is the "zone identifier" metadata. As the size of this alternate stream is only 26 bytes (see attached, << 64KB) then I still maintain that there is a problem with the use of streams_attr on CIFS shares on FreeNAS and the rm command on debian 8.5. I'm not sure which component has the actual problem.
Click to expand...
It is an interesting problem. Do note that the zone identifier is not the only information being stored as an extended attribute on your files. If "store dos attributes" is set to "yes", which is the default, then DOS attributes will also be stored as FS extended attributes. You can view them locally on the freenas server by typing
Code:
getextattr user DOSATTRIB <file>


These are not transparent to SMB clients.

But... I agree this may not be where your problem is. I'd probably try increasing logging verbosity for samba (full debug) and reproduce the problem. My guess is that it is a permissions error. Perhaps delete is failing because the user lacks appropriate privileges via ACLs in a way that is not immediately transparent to the kernel cifs client in Debian 8.5.

I use Centos 6 clients without problem, but I also set "store dos attributes = no".
 
Last edited:
B

barbierimc

Dabbler
Joined
Jun 25, 2016
Messages
22
anodos said:
It is an interesting problem. Do note that the zone identifier is not the only information being stored as an extended attribute on your files. If "store dos attributes" is set to "yes", which is the default, then DOS attributes will also be stored as FS extended attributes. You can view them locally on the freenas server by typing
Code:
getextattr user DOSATTRIB <file>


These are not transparent to SMB clients.

But... I agree this may not be where your problem is. I'd probably try increasing logging verbosity for samba (full debug) and reproduce the problem. My guess is that it is a permissions error. Perhaps delete is failing because the user lacks appropriate privileges via ACLs in a way that is not immediately transparent to the kernel cifs client in Debian 8.5.

I use Centos 6 clients without problem, but I also set "store dos attributes = no".
Click to expand...

In this case, I manually removed the single zone identifier AD stream from the file, then I was able to successfully delete the same file. No permission, changes on either end. For me this is highly reproducible.

I think I will setup a fresh test installation (so I can play and break it as much as I need!), enable debugging, etc and see what I can share to resolve this issue. I can also try a testing version of debian.

This will take a few days to get setup. Thanks,
 
anodos

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
barbierimc said:
In this case, I manually removed the single zone identifier AD stream from the file, then I was able to successfully delete the same file. No permission, changes on either end. For me this is highly reproducible.

I think I will setup a fresh test installation (so I can play and break it as much as I need!), enable debugging, etc and see what I can share to resolve this issue. I can also try a testing version of debian.

This will take a few days to get setup. Thanks,
Click to expand...

Post your results once you are done with the testing.
 
B

barbierimc

Dabbler
Joined
Jun 25, 2016
Messages
22
anodos said:
Post your results once you are done with the testing.
Click to expand...

Fresh build of FreeNAS current stable release. I created a single CIFS share with windows permissions. Set log-level to debug.

Added a single file with an ADS from the internet. Mounted the share on a debian 8.5 build. Performed 'rm' on the file. The 'rm' command never completes.

strace on the rm command shows that it hangs on the unlink():

Code:
newfstatat(AT_FDCWD, "openvpn.zip", {st_mode=S_IFREG|0775, st_size=12984, ...}, AT_SYMLINK_NOFOLLOW) = 0
geteuid()                               = 1000
newfstatat(AT_FDCWD, "openvpn.zip", {st_mode=S_IFREG|0775, st_size=12984, ...}, AT_SYMLINK_NOFOLLOW) = 0
faccessat(AT_FDCWD, "openvpn.zip", W_OK) = 0
unlinkat(AT_FDCWD, "openvpn.zip", 0^C
Process 1281 detached
<detached ...>



The log.smbd file rapidly fills with the following block of text over and over and over again until I control-c the rm command.

Code:
[2016/07/13 04:54:21.821077, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/open.c:3866(open_streams_for_delete)
  open_streams_for_delete found 2 streams
[2016/07/13 04:54:21.821091, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/open.c:3897(open_streams_for_delete)
  Unable to stat stream: openvpn.zip:Zone.Identifier:$DATA
[2016/07/13 04:54:21.821103, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/open.c:4973(create_file_default)
  create_file: access_mask = 0x10000 file_attributes = 0x80, share_access = 0x7, create_disposition = 0x1 create_options = 0x0 oplock_request = 0x0 private_flags = 0x4 root_dir_fid = 0x0, ea_list = 0x0x0, sd = 0x0x0, fname = openvpn.zip:Zone.Identifier:$DATA
[2016/07/13 04:54:21.821117, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/open.c:4450(create_file_unixpath)
  create_file_unixpath: access_mask = 0x10000 file_attributes = 0x80, share_access = 0x7, create_disposition = 0x1 create_options = 0x0 oplock_request = 0x0 private_flags = 0x4 ea_list = 0x0x0, sd = 0x0x0, fname = openvpn.zip:Zone.Identifier:$DATA
[2016/07/13 04:54:21.821157, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/trans2.c:301(get_ea_names_from_file)
  get_ea_names_from_file: ea_namelist size = 52
[2016/07/13 04:54:21.821184, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/trans2.c:210(get_ea_value)
  get_ea_value: EA user.DosStream.Zone.Identifier:$DATA is of length 27
[2016/07/13 04:54:21.821197, 10, pid=11240, effective(0, 0), real(0, 0)] ../lib/util/util.c:559(dump_data)
  [0000] 5B 5A 6F 6E 65 54 72 61   6E 73 66 65 72 5D 0D 0A   [ZoneTra nsfer]..
  [0010] 5A 6F 6E 65 49 64 3D 33   0D 0A 00                  ZoneId=3 ...


I've managed to catch a log file from service start until I killed it (without the log being overwritten in the process - it's quick)

You can see the posix_unlink and deletion starts at time-stamp 2016/07/13 05:21:03.532018 in the attached

Interested to hear your thoughts. Output seems to be exactly the same as this previous post https://forums.freenas.org/index.ph...ng-signal-4-on-file-delete.34865/#post-212783
 

Attachments

  • log.smbd.txt
    5.8 MB · Views: 346
Last edited:
anodos

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
barbierimc said:
Fresh build of FreeNAS current stable release. I created a single CIFS share with windows permissions. Set log-level to debug.

Added a single file with an ADS from the internet. Mounted the share on a debian 8.5 build. Performed 'rm' on the file. The 'rm' command never completes.

strace on the rm command shows that it hangs on the unlink():

Code:
newfstatat(AT_FDCWD, "openvpn.zip", {st_mode=S_IFREG|0775, st_size=12984, ...}, AT_SYMLINK_NOFOLLOW) = 0
geteuid()                               = 1000
newfstatat(AT_FDCWD, "openvpn.zip", {st_mode=S_IFREG|0775, st_size=12984, ...}, AT_SYMLINK_NOFOLLOW) = 0
faccessat(AT_FDCWD, "openvpn.zip", W_OK) = 0
unlinkat(AT_FDCWD, "openvpn.zip", 0^C
Process 1281 detached
<detached ...>



The log.smbd file rapidly fills with the following block of text over and over and over again until I control-c the rm command.

Code:
[2016/07/13 04:54:21.821077, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/open.c:3866(open_streams_for_delete)
  open_streams_for_delete found 2 streams
[2016/07/13 04:54:21.821091, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/open.c:3897(open_streams_for_delete)
  Unable to stat stream: openvpn.zip:Zone.Identifier:$DATA
[2016/07/13 04:54:21.821103, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/open.c:4973(create_file_default)
  create_file: access_mask = 0x10000 file_attributes = 0x80, share_access = 0x7, create_disposition = 0x1 create_options = 0x0 oplock_request = 0x0 private_flags = 0x4 root_dir_fid = 0x0, ea_list = 0x0x0, sd = 0x0x0, fname = openvpn.zip:Zone.Identifier:$DATA
[2016/07/13 04:54:21.821117, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/open.c:4450(create_file_unixpath)
  create_file_unixpath: access_mask = 0x10000 file_attributes = 0x80, share_access = 0x7, create_disposition = 0x1 create_options = 0x0 oplock_request = 0x0 private_flags = 0x4 ea_list = 0x0x0, sd = 0x0x0, fname = openvpn.zip:Zone.Identifier:$DATA
[2016/07/13 04:54:21.821157, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/trans2.c:301(get_ea_names_from_file)
  get_ea_names_from_file: ea_namelist size = 52
[2016/07/13 04:54:21.821184, 10, pid=11240, effective(0, 0), real(0, 0)] ../source3/smbd/trans2.c:210(get_ea_value)
  get_ea_value: EA user.DosStream.Zone.Identifier:$DATA is of length 27
[2016/07/13 04:54:21.821197, 10, pid=11240, effective(0, 0), real(0, 0)] ../lib/util/util.c:559(dump_data)
  [0000] 5B 5A 6F 6E 65 54 72 61   6E 73 66 65 72 5D 0D 0A   [ZoneTra nsfer]..
  [0010] 5A 6F 6E 65 49 64 3D 33   0D 0A 00                  ZoneId=3 ...


I've managed to catch a log file from service start until I killed it (without the log being overwritten in the process - it's quick)

You can see the posix_unlink and deletion starts at time-stamp 2016/07/13 05:21:03.532018 in the attached

Interested to hear your thoughts. Output seems to be exactly the same as this previous post https://forums.freenas.org/index.ph...ng-signal-4-on-file-delete.34865/#post-212783
Click to expand...

Please post "getfacl" output for openvpn.zip from your example.
 
Last edited:
B

barbierimc

Dabbler
Joined
Jun 25, 2016
Messages
22
anodos said:
Please post "getfacl" output for openvpn.zip from your example.
It is interesting that samba is trying to stat Zone.Identifier:$DATA when the extended attribute (when viewed from lsextattr user <FILE>) is DosStream.Zone.Identifier:$DATA. This might indicate why samba's unable to stat the Zone Identifier data. Another option is that your user lacks permissions to view xattrs.
Click to expand...

Code:
[root@freenas-test] /mnt/tank/Test# getfacl -v openvpn.zip
# file: openvpn.zip
# owner: root
# group: wheel
            owner@:read_data/write_data/execute/append_data/delete_child/delete/read_attributes/write_attributes/read_xattr/write_xattr/read_acl/write_acl/write_owner/synchronize:inherited:allow
            group@:read_data/write_data/execute/append_data/delete_child/delete/read_attributes/write_attributes/read_xattr/write_xattr/read_acl/write_acl/write_owner/synchronize:inherited:allow
         everyone@:read_data/execute/read_attributes/read_xattr/read_acl:inherited:allow
 
anodos

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
barbierimc said:
Code:
[root@freenas-test] /mnt/tank/Test# getfacl -v openvpn.zip
# file: openvpn.zip
# owner: root
# group: wheel
            owner@:read_data/write_data/execute/append_data/delete_child/delete/read_attributes/write_attributes/read_xattr/write_xattr/read_acl/write_acl/write_owner/synchronize:inherited:allow
            group@:read_data/write_data/execute/append_data/delete_child/delete/read_attributes/write_attributes/read_xattr/write_xattr/read_acl/write_acl/write_owner/synchronize:inherited:allow
         everyone@:read_data/execute/read_attributes/read_xattr/read_acl:inherited:allow
Click to expand...

I assume you authenticated as "root" when you tried to delete the file. Looks like it might be a bug. I'm not sure what to do to work around it from an admin perspective (other than turning off streams_xattr). It might also be worth posting this information in the samba mailing list. Someone there might be more helpful.
 
B

barbierimc

Dabbler
Joined
Jun 25, 2016
Messages
22
anodos said:
I assume you authenticated as "root" when you tried to delete the file. Looks like it might be a bug. I'm not sure what to do to work around it from an admin perspective (other than turning off streams_xattr). It might also be worth posting this information in the samba mailing list. Someone there might be more helpful.
Click to expand...

Yes, authenticated as root. I'll raise a ticket on the samba tracker. Thanks for looking at it.

Is it possible for me to pull in the latest samba release to test it, or will I need to install a nightly FreeNAS build (if indeed the latest release is in there?)
 
ewhac

ewhac

Contributor
Joined
Aug 20, 2013
Messages
177
The latest version of FreeNAS 9.10 I pulled down a few days ago contains an update of Samba that has a fix for this (or, going by the Samba bug commentary, a kluge workaround). I can now delete Windows-created files from the Linux client. So, marking as solved.
 
B

barbierimc

Dabbler
Joined
Jun 25, 2016
Messages
22
ewhac said:
The latest version of FreeNAS 9.10 I pulled down a few days ago contains an update of Samba that has a fix for this (or, going by the Samba bug commentary, a kluge workaround). I can now delete Windows-created files from the Linux client. So, marking as solved.
Click to expand...

Yes, this fix was specifically to address this bug. I tested it pre-release with the samba team and it appears to work as expected.
 
Status
Not open for further replies.

Similar threads

K
  • Locked
SOLVED Intermittently can't connect to SMB shares
Replies
1
Views
1K
kjp4756
K
B
  • Locked
Problem with CIFS homes
Replies
8
Views
2K
BlazeStar
B
D
  • Locked
SMB sharing issue on fresh install
Replies
3
Views
1K
anodos
anodos
K
  • Locked
FreeNAS 7 SMB/CIFS Active Directory
Replies
1
Views
4K
Al562
Al562
S
  • Locked
usergroup not resolving in ownership of files with SID lookup error
Replies
2
Views
1K
semaj
S
Top