Mastakilla
Patron
- Joined
- Jul 18, 2019
- Messages
- 203
Hi everyone,
I have a weird problem... I was running the solnet-array-test script and during the parallel array seek test, after a couple days of running, suddenly
But I heard that the seek-array test was still running, as my HDDs were still rambling... So I waited for a few more days and today, finally the script has completed and the SSH window that ran it, did return the prompt to me.
I wonder how I can debug this... I didn't reboot yet, as I'd like to investigate this issue a bit further first...
fyi,
I'm not sure if it is related, but the solnet-array-test completed without errors, but did have weird performance... This is the second time I run it (first time with a zpool and second time without a zpool) and the first time the parallel seek-array test took between 3d4h till 4d4h per HDD, which I already found a big variance.
But now the second run it became even more crazy... One HDD took 3d6h, 4 HDDs took more than 4 days and 3 HDDs took more than 5 days (up till 5d11h!)
When the GUI was still working, I was able to take some HDD IO screenshots and there it is very clear when it becomes slower
da4 is the only fast / normal one. With da5 and da6 you can see performance drop to around 40MB/sec near the end and sometimes restore a little later. The other HDDs, from which I didn't take a screenshot yet (because they weren't completed yet), have an even much longer performance drop to 40MB/sec... Again, not sure if this is related, but I do find this very weird...
I have a weird problem... I was running the solnet-array-test script and during the parallel array seek test, after a couple days of running, suddenly
- My GUI became unreachable on both interfaces
- My IPMI became unreachable (ping timeout)
- My SSH login also stopped working on both interfaces
Code:
C:\Users\m4st4>ssh -vvv root@192.168.0.10 OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5 debug3: Failed to open file:C:/Users/m4st4/.ssh/config error:2 debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2 debug2: resolve_canonicalize: hostname 192.168.0.10 is address debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to 192.168.0.10 [192.168.0.10] port 22. debug1: Connection established. debug3: Failed to open file:C:/Users/m4st4/.ssh/id_rsa error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_rsa.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_rsa type -1 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_rsa-cert error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_rsa-cert.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_rsa-cert type -1 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_dsa error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_dsa.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_dsa type -1 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_dsa-cert error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_dsa-cert.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_dsa-cert type -1 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_ecdsa error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_ecdsa.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_ecdsa type -1 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_ecdsa-cert error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_ecdsa-cert.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_ecdsa-cert type -1 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_ed25519 error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_ed25519.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_ed25519 type -1 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_ed25519-cert error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_ed25519-cert.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_ed25519-cert type -1 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_xmss error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_xmss.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_xmss type -1 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_xmss-cert error:2 debug3: Failed to open file:C:/Users/m4st4/.ssh/id_xmss-cert.pub error:2 debug1: key_load_public: No such file or directory debug1: identity file C:\\Users\\m4st4/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_for_Windows_7.7 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0-hpn14v15 debug1: match: OpenSSH_8.0-hpn14v15 pat OpenSSH* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to 192.168.0.10:22 as 'root' debug3: hostkeys_foreach: reading file "C:\\Users\\m4st4/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file C:\\Users\\m4st4/.ssh/known_hosts:3 debug3: load_hostkeys: loaded 1 keys from 192.168.0.10 debug3: Failed to open file:C:/Users/m4st4/.ssh/known_hosts2 error:2 debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2 debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none debug2: compression stoc: none debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,none debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,none debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none debug2: compression stoc: none debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ObBF/vvqMsbX8ltJx+asuBqgEceleflY+OksclB1HHQ debug3: hostkeys_foreach: reading file "C:\\Users\\m4st4/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file C:\\Users\\m4st4/.ssh/known_hosts:3 debug3: load_hostkeys: loaded 1 keys from 192.168.0.10 debug3: Failed to open file:C:/Users/m4st4/.ssh/known_hosts2 error:2 debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2 debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2 debug1: Host '192.168.0.10' is known and matches the ECDSA host key. debug1: Found key in C:\\Users\\m4st4/.ssh/known_hosts:3 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey after 134217728 blocks debug3: unable to connect to pipe \\\\.\\pipe\\openssh-ssh-agent, error: 2 debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory debug2: key: C:\\Users\\m4st4/.ssh/id_rsa (0000000000000000) debug2: key: C:\\Users\\m4st4/.ssh/id_dsa (0000000000000000) debug2: key: C:\\Users\\m4st4/.ssh/id_ecdsa (0000000000000000) debug2: key: C:\\Users\\m4st4/.ssh/id_ed25519 (0000000000000000) debug2: key: C:\\Users\\m4st4/.ssh/id_xmss (0000000000000000) debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Trying private key: C:\\Users\\m4st4/.ssh/id_rsa debug3: no such identity: C:\\Users\\m4st4/.ssh/id_rsa: No such file or directory debug1: Trying private key: C:\\Users\\m4st4/.ssh/id_dsa debug3: no such identity: C:\\Users\\m4st4/.ssh/id_dsa: No such file or directory debug1: Trying private key: C:\\Users\\m4st4/.ssh/id_ecdsa debug3: no such identity: C:\\Users\\m4st4/.ssh/id_ecdsa: No such file or directory debug1: Trying private key: C:\\Users\\m4st4/.ssh/id_ed25519 debug3: no such identity: C:\\Users\\m4st4/.ssh/id_ed25519: No such file or directory debug1: Trying private key: C:\\Users\\m4st4/.ssh/id_xmss debug3: no such identity: C:\\Users\\m4st4/.ssh/id_xmss: No such file or directory debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password debug3: failed to open file:C:/dev/tty error:3 debug1: read_passphrase: can't open /dev/tty: No such file or directory root@192.168.0.10's password: debug3: send packet: type 50 debug2: we sent a password packet, wait for reply debug3: receive packet: type 52 debug1: Authentication succeeded (password). Authenticated to 192.168.0.10 ([192.168.0.10]:22). debug1: channel 0: new [client-session] debug3: ssh_session2_open: channel_new: 0 debug2: channel 0: send open debug3: send packet: type 90 debug1: Requesting no-more-sessions@openssh.com debug3: send packet: type 80 debug1: Entering interactive session. debug1: pledge: network debug1: console supports the ansi parsing debug3: Successfully set console output code page from:850 to 65001 debug3: Successfully set console input code page from:850 to 65001 debug3: receive packet: type 80 debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0 debug3: receive packet: type 91 debug2: channel_input_open_confirmation: channel 0: callback start debug2: fd 3 setting TCP_NODELAY debug2: client_session2_setup: id 0 debug2: channel 0: request pty-req confirm 1 debug3: send packet: type 98 debug2: channel 0: request shell confirm 1 debug3: send packet: type 98 debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug3: receive packet: type 99 debug2: channel_input_status_confirm: type 99 id 0 debug2: PTY allocation request accepted on channel 0 debug2: channel 0: rcvd adjust 65536 debug3: receive packet: type 99 debug2: channel_input_status_confirm: type 99 id 0 debug2: shell request accepted on channel 0 debug2: client_check_window_change: changed debug2: channel 0: request window-change confirm 0 debug3: send packet: type 98 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 debug3: send packet: type 100 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 debug3: send packet: type 100 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 ...
But I heard that the seek-array test was still running, as my HDDs were still rambling... So I waited for a few more days and today, finally the script has completed and the SSH window that ran it, did return the prompt to me.
I wonder how I can debug this... I didn't reboot yet, as I'd like to investigate this issue a bit further first...
- How can I restore / restart my GUI?
- Perhaps I can restart one of my two interfaces (the one on which I'm not logged in atm)?
- Which log files should I check?
fyi,
I'm not sure if it is related, but the solnet-array-test completed without errors, but did have weird performance... This is the second time I run it (first time with a zpool and second time without a zpool) and the first time the parallel seek-array test took between 3d4h till 4d4h per HDD, which I already found a big variance.
But now the second run it became even more crazy... One HDD took 3d6h, 4 HDDs took more than 4 days and 3 HDDs took more than 5 days (up till 5d11h!)
When the GUI was still working, I was able to take some HDD IO screenshots and there it is very clear when it becomes slower
da4 is the only fast / normal one. With da5 and da6 you can see performance drop to around 40MB/sec near the end and sometimes restore a little later. The other HDDs, from which I didn't take a screenshot yet (because they weren't completed yet), have an even much longer performance drop to 40MB/sec... Again, not sure if this is related, but I do find this very weird...
