Using Robocopy - the specified network name is no longer available

dvc9

Explorer
Joined
May 2, 2012
Messages
72
Hey, long time since I've been on the Forum now,
mostly anything I do with FreeNAS is working perfect :D

However, I have did now hit a strange bug.

--------------------------------------------------------------------------------------------------------------
The setup is an temp production server.
We needed some speed, and quick storage before January.

The FreeNAS is part of an Active Directory, and the join is ok.
DNS is setup right, NTP is verified, and domain users can login to shares, and use the server.

However, when running Robocopy from the "older" server, to the new one, it works perfectly for 8, 12 hours,
then suddenly I get the Error 64, "the specified network name is no longer available" as shown in the image bellow.
--------------------------------------------------------------------------------------------------------------

"
robocopy "Source" "\\10.0.0.1\**" /copyall /mir /fft /r:1 /w:1 /mt:16 /zb /np /ndl /xjd
robocopy "Source" "\\DNS.Domain**" /copyall /mir /fft /r:1 /w:1 /mt:16 /zb /np /ndl /xjd
"
Wont use Copy:DT, as I want to get the correct permissions.
And as stated, it does work, but after a while, on larger files, then it creates the error.

When the error starts, then I can't "browse" the FreeNAS, I can Ping it, but I cant access by //servername/
in file explorer, or in powershell. They will get a time out, as if SMB is crashing, and not responding.

Im coping from a Windows 2016 Server, to the FreeNAS
so all commands are run from the Windows side.

* I have tested several network settings, DHCP, Fixed IP and more.
* I have also tested the NICs as standalone, and as a 4x team, LACP.
* I have removed the machine, from DNS, added it again, and also tested to only run IP on the robocopy commands and more.


No effect on the error.

SMB won't respond until a SMB Restart in Terminal, or a reboot of the server.
Sometimes, it can respond, if the whole connection between Win/FreeNAS, is rested by "net use /delete *"
and then connecting again, and change from DNS to IP, or vie versa.

No enabled Tunables, and no Jumbo Frames enabled.
SMB Conf have one extra config : server multi channel support = yes

As an "Backup" im now pushing FileZilla, and enabled FTP...
I get speed around 500 + MB´s Write, thats ok I guess, theoretically I should get more..

--------------------------------------------------------------------------------------------------------------

The Setup, FreeNAS 11.2

Supermicro 2029U-E1CR4 2U Ultra Server
2x Intel Xeon SP Silver 4110 - 8-Core 2.10GHz
96GB RAM ( I will get more after proven the system )
Supermicro 2-Port SFP+ Intel 10GbE LAN card PCI-e LP X710 DA
--------------------------------------------------------------------------------------------------------------
Avago 9305-16i 12Gbps SAS/SATA PCIe 3.0 HBA
18x Samsung 860 EVO 4TB SATA3
- 5x RaidZ containing 4Disks
- 2x HotSpares
- 1x Intel SSD S4500 L2ARC store metadata exclusively

--------------------------------------------------------------------------------------------------------------
DD Speed Test

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 36.210101 secs (2965310225 bytes/sec)

dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 13.250111 secs (8103644021 bytes/sec)
--------------------------------------------------------------------------------------------------------------
Image of Robocopy error
27451

--------------------------------------------------------------------------------------------------------------
/usr/local/etc/smb4.conf

Code:
root@server[~]# cat /usr/local/etc/smb4.conf
[global]
    server min protocol = SMB2_02
    server max protocol = SMB3
    encrypt passwords = yes
    dns proxy = no
    strict locking = no
    oplocks = yes
    deadtime = 15
    max log size = 51200
    max open files = 2815517
    logging = file
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    getwd cache = yes
    guest account = nobody
    map to guest = Bad User
    obey pam restrictions = yes
    ntlm auth = no
    directory name cache size = 0
    kernel change notify = no
    panic action = /usr/local/libexec/samba/samba-backtrace
    nsupdate command = /usr/local/bin/samba-nsupdate -g
    server string = server - FreeNAS
    ea support = yes
    store dos attributes = yes
    lm announce = yes
    hostname lookups = yes
    unix extensions = no
    acl allow execute always = true
    dos filemode = yes
    multicast dns register = yes
    domain logons = no
    idmap config *: backend = tdb
    idmap config *: range = 90000001-100000000
    server role = member server
    workgroup = DOMAIN
    realm = DOMAIN.NO
    security = ADS
    client use spnego = yes
    local master = no
    domain master = no
    preferred master = no
    ads dns update = yes
    winbind cache time = 7200
    winbind offline logon = yes
    winbind enum users = yes
    winbind enum groups = yes
    winbind nested groups = yes
    winbind use default domain = no
    winbind refresh tickets = yes
    idmap config DOMAIN: backend = rid
    idmap config DOMAIN: range = 20000-90000000
    allow trusted domains = no
    client ldap sasl wrapping = plain
    template shell = /bin/sh
    template homedir = /home/%D/%U
    netbios name = server
    create mask = 0666
    directory mask = 0777
    client ntlmv2 auth = yes
    dos charset = CP437
    unix charset = UTF-8
    log level = 2
    server multi channel support = yes
   

[production]
    path = "/mnt/***"
    printable = no
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = yes
    access based share enum = no
    vfs objects = zfs_space zfsacl streams_xattr
    hide dot files = yes
    guest ok = no
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare


--------------------------------------------------------------------------------------------------------------
/var/log/samba4/log.smbd

Code:
  DOMAIN\nasadmin closed file **/xgen/collections/skalken/skalken_eyebrows/Noise1/p_body.xpd (numopen=1) NT_STATUS_OK
[2019/01/01 18:17:45.430917,  2] ../source3/smbd/open.c:1404(open_file)
  DOMAIN\nasadmin opened file **/xgen/collections/skalken/skalken_eyebrows/Noise1/p_body.xpd read=Yes write=Yes (numopen=3)
[2019/01/01 18:17:45.743948,  1] ../source3/modules/nfs4_acls.c:742(smbacl4_fill_ace4)
  nfs4_acls.c: file [**/xgen/collections/**/**_HeadHair_wavy/AnimWires2/Points]: could not convert S-1-5-21-19305936-478707698-815609150-500 to uid or gid
[2019/01/01 18:17:45.761140,  1] ../source3/modules/nfs4_acls.c:742(smbacl4_fill_ace4)
  nfs4_acls.c: file [**/xgen/collections/**/**_HeadHair_wavy/AnimWires3/Points]: could not convert S-1-5-21-19305936-478707698-815609150-500 to uid or gid
[2019/01/01 18:17:45.796226,  1] ../source3/modules/nfs4_acls.c:742(smbacl4_fill_ace4)
  nfs4_acls.c: file [**/xgen/collections/**/**_HeadHair_wavy/Clumping2/Points]: could not convert S-1-5-21-19305936-478707698-815609150-500 to uid or gid
[2019/01/01 18:17:45.960574,  2] ../source3/smbd/open.c:1404(open_file)
  DOMAIN\nasadmin opened file **/xgen/collections/sunnyPatchless/sunnyPatchless_jumper_tear/Noise1/p_sweather_thick.xpd read=No write=No (numopen=2)
[2019/01/01 18:17:45.961802,  2] ../source3/smbd/close.c:789(close_normal_file)
  DOMAIN\nasadmin closed file **/xgen/collections/sunnyPatchless/sunnyPatchless_jumper_tear/Noise1/p_sweather_thick.xpd (numopen=1) NT_STATUS_OK
[2019/01/01 18:17:45.977605,  2] ../source3/smbd/open.c:1404(open_file)
  DOMAIN\nasadmin opened file **/xgen/collections/sunnyPatchless/sunnyPatchless_jumper_tear/Noise1/p_sweather_thick.xpd read=Yes write=Yes (numopen=2)
[2019/01/01 18:17:53.156651,  2] ../source3/smbd/server.c:807(remove_child_pid)
  Could not find child 17257 -- ignoring

--------------------------------------------------------------------------------------------------------------
/var/log/samba4/log.winbindd

Code:
[2019/01/01 18:02:52.957269,  2] ../source3/lib/interface.c:345(add_interface)
  added interface ixl0 ip=2a02:20c8:2670:351:ae1f:6bff:fe8a:bff8 bcast= netmask=ffff:ffff:ffff:ffff::
[2019/01/01 18:02:52.957705,  2] ../source3/lib/interface.c:345(add_interface)
  added interface ixl0 ip=10.1.1.8 bcast=10.1.7.255 netmask=255.255.248.0
[2019/01/01 18:02:52.958327,  2] ../source3/lib/interface.c:345(add_interface)
  added interface ixl0 ip=2a02:20c8:2670:351:ae1f:6bff:fe8a:bff8 bcast= netmask=ffff:ffff:ffff:ffff::
[2019/01/01 18:02:52.958350,  2] ../source3/lib/interface.c:345(add_interface)
  added interface ixl0 ip=10.1.1.8 bcast=10.1.7.255 netmask=255.255.248.0
[2019/01/01 18:02:52.981445,  1] ../source3/lib/tdb_validate.c:480(tdb_validate_and_backup)
  tdb '/var/db/samba4/winbindd_cache.tdb' is valid
[2019/01/01 18:02:52.996900,  1] ../source3/lib/tdb_validate.c:490(tdb_validate_and_backup)
  Created backup '/var/db/samba4/winbindd_cache.tdb.bak' of tdb '/var/db/samba4/winbindd_cache.tdb'
[2019/01/01 18:02:52.998420,  2] ../source3/winbindd/winbindd_util.c:283(add_trusted_domain_from_tdc)
  Added domain BUILTIN (null) S-1-5-32
[2019/01/01 18:02:52.998459,  2] ../source3/passdb/pdb_interface.c:161(make_pdb_method_name)
  No builtin backend found, trying to load plugin
[2019/01/01 18:02:52.999101,  2] ../source3/winbindd/winbindd_util.c:283(add_trusted_domain_from_tdc)
  Added domain server (null) S-1-5-21-3067501586-3017355786-1614438915
[2019/01/01 18:02:52.999175,  2] ../source3/winbindd/winbindd_util.c:283(add_trusted_domain_from_tdc)
  Added domain DOMAIN DOMAIN.NO S-1-5-21-4100299945-2672209705-340494855
[2019/01/01 18:02:53.001357,  0] ../lib/util/become_daemon.c:124(daemon_ready)
  STATUS=daemon 'winbindd' finished starting up and ready to serve connections


--------------------------------------------------------------------------------------------------------------
/var/log/samba4/log.wb-DOMAIN

Code:
  ../source3/rpc_client/cli_pipe.c:564: RPC fault code DCERPC_FAULT_ACCESS_DENIED received from host qdc.DOMAIN.no!
[2019/01/01 17:41:58.766389,  1] ../source3/winbindd/winbindd_cm.c:2827(cm_connect_sam)
  Unwilling to make SAMR connection to domain DOMAIN without connection level security, must set 'winbind sealed pipes = false' and 'require strong key = false' to proceed: NT_STATUS_DOWNGRADE_DETECTED
[2019/01/01 17:43:53.113069,  0] ../source3/winbindd/winbindd.c:243(winbindd_sig_term_handler)
  Got sig[15] terminate (is_parent=0)
[2019/01/01 17:54:12.531636,  2] ../source3/winbindd/winbindd_pam.c:2108(winbind_dual_SamLogon)
  NTLM CRAP authentication for user [DOMAIN]\[andreas.aanerud] returned NT_STATUS_WRONG_PASSWORD
[2019/01/01 17:54:12.558842,  2] ../source3/winbindd/winbindd_pam.c:2108(winbind_dual_SamLogon)
  NTLM CRAP authentication for user [DOMAIN]\[andreas.aanerud] returned NT_STATUS_WRONG_PASSWORD
[2019/01/01 17:57:45.923919,  0] ../source3/winbindd/winbindd.c:243(winbindd_sig_term_handler)
  Got sig[15] terminate (is_parent=0)
[2019/01/01 18:05:01.965476,  2] ../source3/winbindd/winbindd_pam.c:2108(winbind_dual_SamLogon)
  NTLM CRAP authentication for user [DOMAIN]\[Alexander.VPN] returned NT_STATUS_NO_SUCH_USER
 

Attachments

  • Screenshot 2019-01-01 at 17.34.21.png
    Screenshot 2019-01-01 at 17.34.21.png
    54.4 KB · Views: 996
Last edited:

dvc9

Explorer
Joined
May 2, 2012
Messages
72
Did add the option, to SMB Auxiliary parameters, in the GUI.
" force unknown acl user = yes "

Did remove the multichannel support.

Running

Code:
robocopy "D:\feature\***" "\\SERVER.DOMAIN.no\production\****" /copy:DATSO /mir /fft /r:1 /w:1 /mt:32 /zb /np /ndl /xjd
 
Last edited:

dvc9

Explorer
Joined
May 2, 2012
Messages
72
So I got the same error again, on a 32.3 GB file,
What happens, is that SMB Drops the connections slowly, so we see a decrease in speed,
then suddenly we are at 29 Bytes or similar.

and then the file times out.

Strait after the time out, then it continues with smaller files, and continues working, as if nothing where wrong.
Screenshot 2019-01-02 at 13.51.10.png
 

dvc9

Explorer
Joined
May 2, 2012
Messages
72
Will do.

FYI, before I added that, i got more errors on smaller files.
Screenshot 2019-01-02 at 14.25.25.png


Adding Tunable

Variable : net.inet.tcp.reass.maxqueuelen
Value : 1448

To sysctl
 

dvc9

Explorer
Joined
May 2, 2012
Messages
72
ok, i did test until 19 ich, and I do get the same errors.
Same as before...

Builds up, all files, then somehow it suddenly hangs, all connection Bytes drops, and its off.
Screenshot 2019-01-02 at 19.34.34.png


and, while it runs, its also dropping browsing of files, so it hangs if trying to accsess the Network Drive
Screenshot 2019-01-02 at 20.28.00.png


Ill try ,

case sensitive = yes
socket options = TCP_NODELAY

what do you think ?
 

dvc9

Explorer
Joined
May 2, 2012
Messages
72
ok, i did test until 19 ich, and I do get the same errors.
Same as before...

Builds up, all files, then somehow it suddenly hangs, all connection Bytes drops, and its off.
View attachment 27486

and, while it runs, its also dropping browsing of files, so it hangs if trying to accsess the Network Drive
View attachment 27487

So just in an kinda desperate move, I tried :
following in Auxiliary parameters:

force unknown acl user = yes
case sensitive = yes
socket options = TCP_NODELAY
ea support = no
map archive = no
map hidden = no
map readonly = no
map system = no

Screenshot 2019-01-02 at 21.03.18.png


Still the same kind of error, one file bounces, and kinda "kick starts" something, and the rest follows.
all the files are of some size, while the others are smaller size.

Weird !
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Supermicro 2-Port SFP+ Intel 10GbE LAN card PCI-e LP X710 DA
I have a server at work where I moved the card to a different slot and it started locking up under heavy load. The only thing that appeared to get it going again was a reboot of the system. I moved it back to the original slot and it started working properly, although slowly. It might be as simple as replacing it with a better network card.
 
Last edited:

dvc9

Explorer
Joined
May 2, 2012
Messages
72
Perhaps, tho the strange thing... I did my "backup plan" of enabling FTP and then just used FileZilla to send the files.
I got 400 / 500 MB Write, and 5-7 Gbit transfer rate.

FileZilla option lost all ACL´s so my belief is that there is something to do with ACL´s
Case sensitivity and yeah weird dos attributes ... maybe some NetBIOS stuff, as it always hangs on the "
the specified network name is no longer available"

So at the moment, I'm using ZFS Snapshot, Send and Receive to move all files to another Dataset, where I enabled some ( wrong ) attributes, just to see if I can force the system into a good experience.

( We have like 350 folders, with small 5000 +/- jpgs that are less than 1MB in them, + some folders with huge Simulation files, and to browse a directory with 5000 jpgs where killing me :P )
 

dvc9

Explorer
Joined
May 2, 2012
Messages
72
ok, so update.

so when running smbstatus , then I see the same files "hanging" from 11.30 ( its now 14.36 ) is that normal ?


Code:
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03798.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03798.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03798.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03826.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03826.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03826.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03809.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03809.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03809.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03788.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03788.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03788.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03794.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03794.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03794.jpg   Thu Jan  3 11:30:53 2019
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,546
ok, so update.

so when running smbstatus , then I see the same files "hanging" from 11.30 ( its now 14.36 ) is that normal ?


Code:
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03798.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03798.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03798.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03826.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03826.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03826.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03809.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03809.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03809.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03788.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03788.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03788.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x17019f    RDWR       LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03794.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_NONE  0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03794.jpg   Thu Jan  3 11:30:53 2019
5940         26334      DENY_ALL   0x100180    RDONLY     LEASE(RWH)       /mnt/FastStorage/production   production/footage/q044.0/shots/q044.0_s000.9/output_block/v001/q044.0_s000.9_block_v001.03794.jpg   Thu Jan  3 11:30:53 2019

Use smbstatus to identify the PID of your SMB session, run the command "smbcontrol <pid> debug 5" and re-run the test until it fails. Once it fails, _immediately_ copy /var/log/samba4/log.smbd off the server, tar it up and send it to me in a PM.
 

dvc9

Explorer
Joined
May 2, 2012
Messages
72
Thanks for the replies.
I think I have stabilised it...

So, this is the Dataset created.
Screenshot 2019-01-02 at 21.47.52.png


I know that its bad to turn of Sync, but I where desperate to test everything,
get it stable, and then turn on a feature pr time.

The Auxiliary parameters of SMB are :

socket options = TCP_NODELAY IPTOS_LOWDELAY
hide files = yes
store dos attributes = no
ea support = no
map archive = no
map hidden = no
map system = no
map readonly = no
strict sync = no
sync always = no

I will turn on one feature at a time, and then report back the findings.
at the moment, Directory listings are amazing fast.

Transfer rates are around 50/60 MB/s on Single thread, Small JPG Files. ( 200k )
Screenshot 2019-01-03 at 14.39.07.png

on bigger files, its constant 108MB with a single thread.
If multithread Robocopy, then its slower actually... so there is still some research to do.

First priority is to turn on Sync, on both SMB.conf and ZFS Dataset.
then ill test more.
 

dvc9

Explorer
Joined
May 2, 2012
Messages
72
ok, so i did forward and backwards with testing, and the problem still occurs.

What have I learned ?

The settings in smb.conf do help,
with particular things, like folder browsing and such.
It also looks like it solves the RoboCopy problem, but it does not.
it just takes longer time.

ACL attributes and Dos attributes

"store dos attributes = no" cant be no, as we are using extensively ACL´s on all files and folders.
so when that is set, I just get the errors about the Flags.

However, when using that, and using /copy:DT.
Then im able to sync the folders, and no error 64 is triggered.


Could we limit ourself to /copy:DT?
we need the overship from /copy:DATSO, or /copyall.
/copy:DATSO, and /copyall gives the same result. , error 64.

The other settings do work, and they do speed up folder browsing,
especially on folders containing 20 / 40 000 small jpgs.

Back to Robocopy.
So the error still is there, but Robocopy uses longer time to create it,
while mappings and ea is turned off. I testet more threads, and other options.
But /copy:DATSO and /copyall is always getting stopped, after a given time.

I decided now to test 2 other softwares, to see if the error still is created.

EMCCopy, ( V4.2 ) its the same as Robocopy but different.
This is much faster than Robocopy. The error gets also created.
and it's the same with ACL´s, if ACL´s are transferred then copy failes,
the time of that faile variates.

Beyond Compare (v4)
This is slow, compared to the others, Robocopy and EMCCopy,
but it does get the sync job done, without error.
I think it doesn't write the full ACL setts, therefor it works.

--------------

What I'm thinking, is that its probably a combination of ACL´s and lots of connections at the same time.
The question, is Samba hanging, since there are to many write requests and it needs to check the ACL´s at the same time ?

@andos Thanks for the help, I'll send you the log files in a PM Now.

I'll forward any new findings on the forum.
 
Last edited:

dvc9

Explorer
Joined
May 2, 2012
Messages
72
Hey!

So, i have fixed some solutions, and unfortunately,
the workaround required some changes that i dont like.

First off, the dataset had to be a UNIX share type, not windows.
Screenshot 2019-01-17 at 19.54.24.png


The permissions, were still set with Active Directory Users and Groups
Screenshot 2019-01-17 at 19.59.27.png


and the SMB Axulery parameters where

socket options = TCP_NODELAY IPTOS_LOWDELAY
case sensitive = false

This solved the Robocopy error from an "harsh" windows environment,
that where based on recursive ACL strategy, ( all ACLs where parented )


- So two lessions.
one , ACL sucks... dont do recursive , and forget to copy the main folder.
two , FreeNAS is mostly tuned for good samba speed ( thanks iXsystems! )
 

dvc9

Explorer
Joined
May 2, 2012
Messages
72
I have a server at work where I moved the card to a different slot and it started locking up under heavy load. The only thing that appeared to get it going again was a reboot of the system. I moved it back to the original slot and it started working properly, although slowly. It might be as simple as replacing it with a better network card.

Hey Chris!
Just wanted you to know,

We never got the server 100%, there where always something on the networking side. I even updated drivers n all, but still there where times where we either got 2846% Package storm, or LDAP wo simply disconnected all users etc.

So I thought on what you stated, and we got ourself a Solarstorm SFN5162F SFP+ Server Adapter.
Without any tuning, anything at all, just plug n play, we got full speed, and excellent small file performance.

Wont buy Intel cards anytime soon.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
So I thought on what you stated, and we got ourself a Solarstorm SFN5162F SFP+ Server Adapter.
Without any tuning, anything at all, just plug n play, we got full speed, and excellent small file performance.
What kind of transceivers or cables did you use with that?
 
Top