Vega Alpha
Dabbler
- Joined
- Mar 19, 2016
- Messages
- 17
I knew this day would come.
Six years ago, I purchased FreeNAS Mini with 32GB ECC RAM, 1 4 TB Data Set Mirrored via 2 7200 RPM Western Digital NAS Red, 1 4 TB Data Set Striped via 1 7200 RPM Western Digital Enterprise Black Amazon Refurbished.
I installed software:
Plugin: Plex-Pass
Jail: Gogs a Git server <- Old FreeNAS warren jail
Jail: PostgreSQL with PostgreSQL Admin <- Plans to move to community plugin
Plugin: GitLab <- Currently migrating to.
Samba Shares to drop movies, music, and pictures into folders so Plex will find and later show them in its GUI.
In December 2020, the Western Digital Black strip failed SMART manual backup, and restoring this non-critical data was successful.
December 2021 One of the drives in the Mirror began to fail. This is critical data.
In console:
Dec 19 16:10:48 server (ada1:ahcich10:0:0:0): Error 5, Retries exhausted
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): RES: 41 40 a0 01 8d 40 d5 00 00 00 00
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): Retrying command, 3 more tries remain
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): RES: 41 40 17 02 8d 40 d5 00 00 00 00
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): Retrying command, 2 more tries remain
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): RES: 41 40 90 01 8d 40 d5 00 00 00 00
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): Retrying command, 1 more tries remain
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): RES: 41 40 50 01 8d 40 d5 00 00 00 00
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): Retrying command, 0 more tries remain
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): RES: 41 40 f7 01 8d 40 d5 00 00 00 00
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): Error 5, Retries exhausted
Dec 19 16:11:32 server GEOM_MIRROR: Cannot open consumer ada1p1 (error=1).
Dec 19 16:11:32 server GEOM_MIRROR: Device swap1 destroyed.
Dec 20 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 20 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 21 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 21 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 22 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 22 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 23 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 23 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 23 16:04:58 server 1 2021-12-23T16:04:58.699007-06:00 server.local sshd 56057 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:04:58 server 1 2021-12-23T16:04:58.699219-06:00 server.local sshd 56057 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:04:58 server 1 2021-12-23T16:04:58.796591-06:00 server.local sshd 56058 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:04:58 server 1 2021-12-23T16:04:58.796791-06:00 server.local sshd 56058 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:51:29 server 1 2021-12-23T16:51:29.032764-06:00 server.local sshd 57912 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:51:29 server 1 2021-12-23T16:51:29.032962-06:00 server.local sshd 57912 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:51:29 server 1 2021-12-23T16:51:29.082248-06:00 server.local sshd 57913 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:51:29 server 1 2021-12-23T16:51:29.082451-06:00 server.local sshd 57913 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 24 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 24 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 25 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 25 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 26 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 26 00:00:00 server syslog-ng[1176]: Configuration reload finished;
I am a Windows SDK Software Developer of Windows desktop applications, Java Web Middle Ware Developer, and HTML, CSS, and JavaScript Front End Developer, with very light knowledge of using the Linux command line, barely sufficient to deploy my Web Applications into Tomcat, WebLogic, WebSphere, and Fuse application servers. When I purchased the FreeNAS server, I thought it would be easy to manage but instead found myself in a nightmare of confusion. I needed more Linux knowledge, so I was willing to put great effort into FreeBSD, which I did for about a year of casual weekends. I found very little translated into Linux for my job, so I stopped applying myself to FreeBSD. My practical Linux skills laid stagnated so far as have been needed by my particular needs in my Software Developer career. That said. You can give me instructions, and I will likely have the aptitude to follow instructions, understand definitions, and so on, but will not know enough of the specifics to get the job done myself. I'm a software developer knowing C/C++, Java, JavaScript, 6502 assembly. (Yes, that's right—Assembly for an 8-bit 6502 on a Commodore 64 back in the '80s. I tell my kids I'm old as dirt.) I know software developer kits and application frameworks, but nearly no system management, system admin, hardware configuration, OS configuration. I'm not far down the Dev/Ops path yet.
Would you mind helping me save 25 years of family photos and home videos?
I purchased a FreeNAS Mini from iXSystems. I think the hardware and software support hot swap. So when the new drive comes in the mail, I pop the old drive out and slide the new drive in. And TrueNAS begins to block by block copy the existing drive with data to the new drive. I will see no interruption of service from the outside. A couple of days later, I will receive an email from my server saying the two halves of the Mirror are once again in sync. No command line nor GUI actions to take. Ok, I know better. That's an extreme. I've been reading the manuals for an hour now, but I still don't know what is wrong with the Mirror. I only know that something is wrong, and I'm not sure precisely what variation of the documented procedure I take. I say it that way because a facade for something fundamentally complicated never goes as planned. I've been in software development for 30+ years. Stuff happens, and today I don't feel prepared.
Where do I look to analyze the situation? I was on the "Pools" page. It says "volume-1" is "degraded," but little more. Then articles around the web have me typing in "diskinfo" and "zsf" commands on the command line. I'm following instructions. I know I'll get there. But how many weekends will it take? A process consuming a lot of my time is not what I promised my wife when I spent $3000 on this machine. I think I oversold myself 6 years ago when I described to my wife my effort when a drive eventually failed. Maybe I should just go back to the TrueNAS docs. When I left it, I was reading a lot that didn't seem to apply, but at the same time I was not positive as to what I can and can not do when the new drive arrives in the mail.
Where is my mentor from when I was 20? I wonder if he is alive today? Can anyone with experience step me through this my first time?
Six years ago, I purchased FreeNAS Mini with 32GB ECC RAM, 1 4 TB Data Set Mirrored via 2 7200 RPM Western Digital NAS Red, 1 4 TB Data Set Striped via 1 7200 RPM Western Digital Enterprise Black Amazon Refurbished.
I installed software:
Plugin: Plex-Pass
Jail: Gogs a Git server <- Old FreeNAS warren jail
Jail: PostgreSQL with PostgreSQL Admin <- Plans to move to community plugin
Plugin: GitLab <- Currently migrating to.
Samba Shares to drop movies, music, and pictures into folders so Plex will find and later show them in its GUI.
In December 2020, the Western Digital Black strip failed SMART manual backup, and restoring this non-critical data was successful.
December 2021 One of the drives in the Mirror began to fail. This is critical data.
In console:
Dec 19 16:10:48 server (ada1:ahcich10:0:0:0): Error 5, Retries exhausted
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): RES: 41 40 a0 01 8d 40 d5 00 00 00 00
Dec 19 16:10:55 server (ada1:ahcich10:0:0:0): Retrying command, 3 more tries remain
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): RES: 41 40 17 02 8d 40 d5 00 00 00 00
Dec 19 16:11:02 server (ada1:ahcich10:0:0:0): Retrying command, 2 more tries remain
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): RES: 41 40 90 01 8d 40 d5 00 00 00 00
Dec 19 16:11:09 server (ada1:ahcich10:0:0:0): Retrying command, 1 more tries remain
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): RES: 41 40 50 01 8d 40 d5 00 00 00 00
Dec 19 16:11:16 server (ada1:ahcich10:0:0:0): Retrying command, 0 more tries remain
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 f0 00 8d 40 d5 00 00 05 00 00
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): RES: 41 40 f7 01 8d 40 d5 00 00 00 00
Dec 19 16:11:23 server (ada1:ahcich10:0:0:0): Error 5, Retries exhausted
Dec 19 16:11:32 server GEOM_MIRROR: Cannot open consumer ada1p1 (error=1).
Dec 19 16:11:32 server GEOM_MIRROR: Device swap1 destroyed.
Dec 20 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 20 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 21 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 21 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 22 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 22 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 23 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 23 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 23 16:04:58 server 1 2021-12-23T16:04:58.699007-06:00 server.local sshd 56057 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:04:58 server 1 2021-12-23T16:04:58.699219-06:00 server.local sshd 56057 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:04:58 server 1 2021-12-23T16:04:58.796591-06:00 server.local sshd 56058 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:04:58 server 1 2021-12-23T16:04:58.796791-06:00 server.local sshd 56058 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:51:29 server 1 2021-12-23T16:51:29.032764-06:00 server.local sshd 57912 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:51:29 server 1 2021-12-23T16:51:29.032962-06:00 server.local sshd 57912 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:51:29 server 1 2021-12-23T16:51:29.082248-06:00 server.local sshd 57913 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 23 16:51:29 server 1 2021-12-23T16:51:29.082451-06:00 server.local sshd 57913 - - _secure_path: /mnt/volume-1/users/jerry/.login_conf is group writeable by non-authorised groups
Dec 24 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 24 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 25 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 25 00:00:00 server syslog-ng[1176]: Configuration reload finished;
Dec 26 00:00:00 server syslog-ng[1176]: Configuration reload request received, reloading configuration;
Dec 26 00:00:00 server syslog-ng[1176]: Configuration reload finished;
I am a Windows SDK Software Developer of Windows desktop applications, Java Web Middle Ware Developer, and HTML, CSS, and JavaScript Front End Developer, with very light knowledge of using the Linux command line, barely sufficient to deploy my Web Applications into Tomcat, WebLogic, WebSphere, and Fuse application servers. When I purchased the FreeNAS server, I thought it would be easy to manage but instead found myself in a nightmare of confusion. I needed more Linux knowledge, so I was willing to put great effort into FreeBSD, which I did for about a year of casual weekends. I found very little translated into Linux for my job, so I stopped applying myself to FreeBSD. My practical Linux skills laid stagnated so far as have been needed by my particular needs in my Software Developer career. That said. You can give me instructions, and I will likely have the aptitude to follow instructions, understand definitions, and so on, but will not know enough of the specifics to get the job done myself. I'm a software developer knowing C/C++, Java, JavaScript, 6502 assembly. (Yes, that's right—Assembly for an 8-bit 6502 on a Commodore 64 back in the '80s. I tell my kids I'm old as dirt.) I know software developer kits and application frameworks, but nearly no system management, system admin, hardware configuration, OS configuration. I'm not far down the Dev/Ops path yet.
Would you mind helping me save 25 years of family photos and home videos?
I purchased a FreeNAS Mini from iXSystems. I think the hardware and software support hot swap. So when the new drive comes in the mail, I pop the old drive out and slide the new drive in. And TrueNAS begins to block by block copy the existing drive with data to the new drive. I will see no interruption of service from the outside. A couple of days later, I will receive an email from my server saying the two halves of the Mirror are once again in sync. No command line nor GUI actions to take. Ok, I know better. That's an extreme. I've been reading the manuals for an hour now, but I still don't know what is wrong with the Mirror. I only know that something is wrong, and I'm not sure precisely what variation of the documented procedure I take. I say it that way because a facade for something fundamentally complicated never goes as planned. I've been in software development for 30+ years. Stuff happens, and today I don't feel prepared.
Where do I look to analyze the situation? I was on the "Pools" page. It says "volume-1" is "degraded," but little more. Then articles around the web have me typing in "diskinfo" and "zsf" commands on the command line. I'm following instructions. I know I'll get there. But how many weekends will it take? A process consuming a lot of my time is not what I promised my wife when I spent $3000 on this machine. I think I oversold myself 6 years ago when I described to my wife my effort when a drive eventually failed. Maybe I should just go back to the TrueNAS docs. When I left it, I was reading a lot that didn't seem to apply, but at the same time I was not positive as to what I can and can not do when the new drive arrives in the mail.
Where is my mentor from when I was 20? I wonder if he is alive today? Can anyone with experience step me through this my first time?
Last edited: