SOLVED Extremely Slow network speeds (or failed transfers) since 9.1.1 upgrade

[shackrock]

I just upgraded via the GUI from 8.3.1. 64-bit Version. I also performed the zpool upgrade with no issues.

Here's some base specs:

Build FreeNAS-9.1.1-RELEASE-x64 (a752d35)
Platform Intel(R) Core(TM) i3 CPU 540 @ 3.07GHz
Memory 4003MB

Supermicro X8SIL-F
Intel Core i3-530 CPU
Kingston 2 x 2GB ECC Unbuffered DDR3 1333 RAM
Corsair 650TX PSU
I have a 14.1TB RAID-Z1 array. It's 2 ZPools total.

Before this upgrade, transfer speeds using CIFS from my laptop to the server (pure gigabit network, nothing else is allowed on it, all wired) - were between 30-70MBps depending on what else was happening. Now, with nothing else going on, Windows literally times out on most transfers. Sometimes it will linger around 10kBps for a few minutes before saying "check your network connection..."

Any ideas what happened here? I'm happy to post any logs as required. Just let me know how to get to them.

Thanks all.

Update: Just restarted the server again, still no change. Now it takes around 4 minutes to open a folder that resides on FreeNAS (used to be <1 second).

 

[SmallGuy]

At a glance, 14To with 4Go memory seems to be optimistic.
Take a look at the hardware recommendations (8Go minimum).

 

[SmallGuy]

Have a look on the swap utilisation.

 

[shackrock]

At a glance, 14To with 4Go memory seems to be optimistic.
Take a look at the hardware recommendations (8Go minimum).

Have a look on the swap utilisation.​

​

I hear you on RAM, but like I said this wasn't an issue before... and this thing only sees one user at a time. ​

​

This appears to me to be software related rather than hardware seeing as how it was working at least reliably around 30-70MBps before the upgrade.​

​

Swap utilization graph shows 100% free. Physical memory shows around 90% free.​

 

[cyberjock]

Yeah, it is software related. FreeBSD 9 needs more RAM than 8. So yes, upgrading your RAM will fix your problem. I would normally say you could go back to 8.x, but you said you did the zpool upgrade so that's not an option. You will probably need to go to 12GB or so to get the performance back you are used to.

 

[SmallGuy]

As you just restart, Is there some Heavy activity on your drives?
Think it take time to mount 14TB.
[edit] Cyberjok draw faster than me. :)

 

[shackrock]

As you just restart, Is there some Heavy activity on your drives?
Think it take time to mount 14TB.

Looks pretty dead to me on the reporting tab.

I'll grab some extra RAM and see what happens I suppose...

 

[shackrock]

As you just restart, Is there some Heavy activity on your drives?
Think it take time to mount 14TB.
[edit] Cyberjok draw faster than me. :)

Hey, as a follow up to this... Just woke up this morning and now the rig is transferring at 115MBps.... haha!! Does that mean it was really working to build the ZFS array after a restart for several hours?

Right now my load times on the system are averaging 1.0 and sometimes higher/lower. Not sure what it's doing right now, literally nothing is using the NAS, it's all internal. Here's my "display system processes" screen:

Code:

last pid: 28506;  load averages:  0.87,  0.90,  0.92  up 0+16:07:55    08:28:29
38 processes:  1 running, 37 sleeping
 
Mem: 117M Active, 143M Inact, 3072M Wired, 136M Buf, 522M Free
ARC: 2576M Total, 350M MFU, 2112M MRU, 34K Anon, 60M Header, 54M Other
Swap: 20G Total, 20G Free
 
  PID USERNAME    THR PRI NICE  SIZE    RES STATE  C  TIME  WCPU COMMAND
4430 root          1  20    0 92300K 17148K select  1  5:38  0.00% smbd
1827 root          1  20    0 10372K  4376K select  0  1:24  0.00% devd
3046 root          7  20    0  122M 11908K uwait  2  0:44  0.00% collectd
2906 root          6  20    0  394M  146M usem    0  0:15  0.00% python2.7
27990 root          1  20    0 88204K 13792K select  0  0:07  0.00% smbd
2635 uucp          1  20    0 18324K  7748K WCTRL  1  0:07  0.00% usbhid-ups
2520 root          1  20    0 22212K  3888K select  2  0:02  0.00% ntpd
2103 root          1  20    0 12032K  1732K select  3  0:01  0.00% syslogd
3713 root          1  52    0  147M 52128K ttyin  3  0:01  0.00% python2.7
2645 uucp          1  22    0 20332K  3560K nanslp  1  0:01  0.00% upsmon
2637 uucp          1  20    0 20352K  3572K select  0  0:01  0.00% upsd
2856 root          1  20    0 73724K  7616K select  3  0:00  0.00% nmbd
3175 root          1  52    0 14124K  1808K nanslp  1  0:00  0.00% cron
2187 root          4  20    0  9900K  1548K rpcsvc  0  0:00  0.00% nfsd
8869 www          1  20    0 26024K  6164K kqread  1  0:00  0.00% nginx
2864 root          1  20    0 82532K 11256K select  1  0:00  0.00% smbd
2640 uucp          1  20    0 20328K  3536K nanslp  1  0:00  0.00% upslog
2209 root          1  52    0 14120K  1812K rpcsvc  2  0:00  0.00% rpc.lockd

UPDATE: ok maybe I spoke too soon again. Shortly after my first few transfers, speeds dropped to around 10-20MBps. Still much better than last night. It looks like my memory is indeed about to max itself out. I'll try upgrading memory and report back. However I'm still interested to know what FreeNAS does after a reboot, and how long that takes, any info on that?

 

[cyberjock]

The problem is that as pools get bigger and there is more data causing more metaslabs to be used, more metadata(and regular data) exist, your cache size has to increase appropriately. Without the cache, virtually all actions such as mounting the pool and moving/copying data to/from the pool causes your unacceptably small cache size to cause serious performance problems. When the cache is being flushed of data it will need a short time later, you can expect absolutely horrible performance.

But yeah, short is that you shouldn't even be using 4GB of RAM with ZFS. The manual makes it clear 8GB is the minimum, and for a reason. Many users have had a single kernel panic from insufficient RAM and their entire pool was lost. Permanently. And you are welcome to search the forums as 2 people in the last 2 days have learned that if your pool is unmountable, there are no recovery options. You just get to kiss your data goodbye.

 

[shackrock]

The problem is that as pools get bigger and there is more data causing more metaslabs to be used, more metadata(and regular data) exist, your cache size has to increase appropriately. Without the cache, virtually all actions such as mounting the pool and moving/copying data to/from the pool causes your unacceptably small cache size to cause serious performance problems. When the cache is being flushed of data it will need a short time later, you can expect absolutely horrible performance.

But yeah, short is that you shouldn't even be using 4GB of RAM with ZFS. The manual makes it clear 8GB is the minimum, and for a reason. Many users have had a single kernel panic from insufficient RAM and their entire pool was lost. Permanently. And you are welcome to search the forums as 2 people in the last 2 days have learned that if your pool is unmountable, there are no recovery options. You just get to kiss your data goodbye.

Wow, ok good to know. This rig was built about 4 years ago with V .7 of FreeNAS, just haven't upgraded anything since then except adding drives and the software. I suppose I'll shut it down for now until the RAM arrives tomorrow. Thanks.

 

[cyberjock]

I had the same problems you had back in January. Pool performance sucked and there is no warning that you need more RAM. I had 12GB of RAM. I didn't want to buy another stick until I could prove definitively that I needed it. Well, after 2 weeks of trying to get it to work I dropped the money on another 8GB stick. Poof, performance was back. If you just follow the 1GB of RAM per TB of disk space, you can't go wrong.

 

[shackrock]

I had the same problems you had back in January. Pool performance sucked and there is no warning that you need more RAM. I had 12GB of RAM. I didn't want to buy another stick until I could prove definitively that I needed it. Well, after 2 weeks of trying to get it to work I dropped the money on another 8GB stick. Poof, performance was back. If you just follow the 1GB of RAM per TB of disk space, you can't go wrong.

Cool. FYI not sure if I'd call it solved yet, til I try it out, but hopefully it will solve it for me =).

 

[SmallGuy]

I'm confident upgrading your RAM solve your problem, anyway this the first step you have to do, as sooner or later you would have trouble with this lack of RAM.
So don't worry.

 

[cyberjock]

Cool. FYI not sure if I'd call it solved yet, til I try it out, but hopefully it will solve it for me =).

I know. And to be honest(and don't take offense to this.. you aren't alone) the sooner people realize that RAM is extremely important for ZFS the less time many of us moderators will spend telling people to do what the manual, my noobie presentation, the FAQ, and stickies in the forums, and about 300 threads that already say to do. The moderators' #1 complaint is about the excessive number of threads that could have been solved with a simple "RTFM" and "Add more RAM". I've gotten to the point where I just close my browser tab when I see people cutting their RAM short(you were kind of lucky that posted). If they can't figure it out from all of the places the manual writers and moderators have pointed it out, should we really spend more time trying to tell someone to do it again? All of us mods do not get paid for our time. So why continue to hash out the same problem every day or two? We get sick of it after a while...

Anyway, let us know how the RAM thing goes. Hint: I already know what the end result will be. ;)

Edit: I didn't change it to solved. But I wouldn't be surprised if another mod did since this problem is the most common problem we see in the forums.

 

[shackrock]

I know. And to be honest(and don't take offense to this.. you aren't alone) the sooner people realize that RAM is extremely important for ZFS the less time many of us moderators will spend telling people to do what the manual, my noobie presentation, the FAQ, and stickies in the forums, and about 300 threads that already say to do. The moderators' #1 complaint is about the excessive number of threads that could have been solved with a simple "RTFM" and "Add more RAM". I've gotten to the point where I just close my browser tab when I see people cutting their RAM short(you were kind of lucky that posted). If they can't figure it out from all of the places the manual writers and moderators have pointed it out, should we really spend more time trying to tell someone to do it again? All of us mods do not get paid for our time. So why continue to hash out the same problem every day or two? We get sick of it after a while...

Anyway, let us know how the RAM thing goes. Hint: I already know what the end result will be. ;)

Edit: I didn't change it to solved. But I wouldn't be surprised if another mod did since this problem is the most common problem we see in the forums.

Haha, understood.

Any thoughts to what FreeNAS was doing for the first few hours though?

 

[cyberjock]

Not a clue. I know ZFS does some stuff in the background, and if your pool performs poorly for any reason(failing disk, insufficient RAM, overly busy CPU or underpowered CPU) tasks that normally take a few seconds can suddenly take hours or days. I'd focus less on what "might" have been going on and focus on getting your hardware up to something that higher than 1/2 of the minimum requirements as called out in the manual.

If you go to 16GB of RAM and then still have problems, then we can see what is wrong and do some troubleshooting. But as long as you have a system with less than the minimum requirements don't expect much sympathy or time from us volunteers trying to track down a problem that might not be a problem at all. All of us mods are volunteers. ;)

This isn't much different from people that try to stream HD videos over wireless and complain that FreeNAS is slow. Well, no kidding! Wireless isn't meant to be fast, and you're lucky if you can stream HD videos at all over wireless. ;)

 

[shackrock]

Hi, just wanted to update.

After a couple days of the server being on with the 4GB of RAM, my transfers started going up to 100MBps - everything was working well. However I ordered extra RAM anyhow, and it was backordered but finally arrived and I installed it tonight.

Take a look at the attached image. Now my question is: I was performing well with 4GB of memory, and the physical memory was using almost all of the 4GB I had (although with a bigger swap size). With my upgrade to 12GB I'm still having FreeNAS us almost all of my physical memory/RAM, but with a smaller swap size now.

I think the server IS responding faster, but the transfer speeds are about the same. I guess my point is, it seems like things were performing well after I let the server get itself together for a couple days - even with 4GB. Is it really a concern of system stability with low RAM? Just wondering, thanks!

 

[warri]

ZFS is designed to take all your free RAM, so that is normal. It'll occupy as much as it can get. There has been a thread about that in the forums some months ago.

The system can run stable with 4 GB, but you might run into problems when trying to recover from disk failures. We had a user who wasn't able to mount their pool until they upgraded RAM.

So since you already have the additional RAM, why not play it safe and keep it?

 

[shackrock]

ZFS is designed to take all your free RAM, so that is normal. It'll occupy as much as it can get. There has been a thread about that in the forums some months ago.

The system can run stable with 4 GB, but you might run into problems when trying to recover from disk failures. We had a user who wasn't able to mount their pool until they upgraded RAM.

So since you already have the additional RAM, why not play it safe and keep it?

Thanks for the info, that makes sense. I am definitely keeping the RAM - just wanting to get educated further on the topic is all really. In my case, I have a fairly older motherboard and the RAM I need to match my existing 4GB was hard to find, so I won't let go of this now =).