SOLVED Can Someone Interpret This For Me?

[NiceTry]

After installing miniDLNA and starting it on my system, transfer speeds to FreeNAS AFP server has slowed to a crawl from 50-80MB/sec to less than 5MB/sec.

I used the installation process from the GUI to select and install miniDLNA. I provided storage references and IPv4 gateway address.

If I turn off miniDLNA and stop the jail and reboot, the performance returns to normal.

On stopping the jail, I see this on the console:

"Jun 24 16:00:50 ServerA kernel: arp: 192.168.1.94 moved from 02:3b:bc:00:0a:0a to 00:24:21:ec:c4:e5 on epair0b
Jun 24 16:15:32 ServerA kernel: epair0a: link state changed to DOWN Jun 24 16:15:32 ServerA kernel: epair0b: link state changed to DOWN
Jun 24 16:15:32 ServerA kernel: re0: link state changed to DOWN Jun 24 16:15:32 ServerA kernel: bridge0: link state changed to DOWN
Jun 24 16:15:32 ServerA kernel: re0: promiscuous mode disabled Jun 24 16:15:33 ServerA ntpd[2021]: sendto(198.55.111.5) (fd=22): No route to host
Jun 24 16:15:36 ServerA kernel: re0: link state changed to UP Jun 24 16:16:24 ServerA kernel: ifa_del_loopback_route: deletion failed
Jun 24 16:16:24 ServerA kernel: Freed UMA keg (udp_inpcb) was not empty (20 items). Lost 2 pages of memory.
Jun 24 16:16:24 ServerA kernel: Freed UMA keg (udpcb) was not empty (336 items). Lost 2 pages of memory.
Jun 24 16:16:24 ServerA kernel: Freed UMA keg (tcptw) was not empty (100 items). Lost 2 pages of memory.
Jun 24 16:16:24 ServerA kernel: Freed UMA keg (tcp_inpcb) was not empty (30 items). Lost 3 pages of memory.
Jun 24 16:16:24 ServerA kernel: Freed UMA keg (tcpcb) was not empty (12 items). Lost 3 pages of memory.
Jun 24 16:16:24 ServerA kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Jun 24 16:16:24 ServerA kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required"

Can someone interpret the last two lines? What needs cleanup and how is it done? Or, do I have
other configuration issues going on?

 

[joeschmuck]

Here is the answer to the question you posted:

The last two lines are simply stating that further hook cleanup is required. It will occur and these are normal messages.

I hate asking over and over for this but we need to know what version of FreeNAS you are using, your hardware makeup, etc... if you want some good assistance. This is posted in the forum rules.

I have one question for you... How many media files do you have? If you have a lot, that would cause the slow down until the scan of your media was complete.

 

[NiceTry]

Joe, thanks for the info re: hook cleanup.
I waited until the scan completed before testing. Modest library. 600 video, 1600 music. 1.4TB total. Scan takes less than 15 minutes.
BTW, the slowdown occurs on writing to FreeNAS. Reads consistently 60-80MB/s. Sometimes even faster.
Another BTW: Symptom occurs when jail is running or has been run. It's not necessary that the miniDLNA plugin is ever turned on to evoke the slowdown. Once the jail has started, the slowdown occurs. Another footnote is that the first time a write-after-jail-start happens, the slowdown occurs after 3-4 GB are transferred. Then it bogs down until creeping along at a few MB/s. Subsequent attempts to write chug along slowly even after waiting some time between attempts (I was thinking cache filling problem, but...).

When jail is stopped, reboot is required to clear problem. I've turned off autostart for the jail. If I remove the jail, all is well. My uninformed guess is that there is an unintended competition for the re0 ethernet port by the jail processes and FreeNAS but that does not explain the problem not showing up until 3-4GB have passed. I'm hoping better minds than mine have an idea. I also have no idea how to effect a change.

9.2.1.5, 4-1TB RaidZ1 Iz4, 4GB RAM, 2.4GHz Core 2 Duo E4600. Sorry about that. I usually include the specs in my posts.
This configuration ran under 8.x, 9.2 and 9.2.1.4. Weird symptoms (crashes, DLNA drop-outs) occurred about same time as upgrade to 9.2.1.5. Thinking I had somehow corrupted the USB system drive, I performed a clean install of 9.2.1.5 (as opposed to GUI upgrade path) and replaced one drive whose SMART report noted bad sectors. Volumes still reported "HEALTHY" and all drives were "ONLINE." I replaced it anyway. Those actions stabilized primary fileserver function. Slowdown symptom may have been present prior to 9.2.1.5 upgrade but I didn't notice.
I'm very pleased with how FreeNAS is doing its file service. Now it's down to getting miniDLNA/jail to operate without interfering with the server function.
I think that completes the history. I have limited diagnostics and experience with FreeBSD so please be patient.
I hope someone has some insight. Otherwise I have another external option for the DLNA server I can use.

 

[solarisguy]

I am seeing a correlation between your 4GB of RAM, and slowdown observed after 3-4GB have been transferred.

Try to borrow 8 or 16GB of RAM and test whether the slowdown occurs after the amount transferred is around the size of the RAM.

 

[NiceTry]

Good eye, solarisguy. I was wondering that too and was hoping someone might have an explanation other than the experimental coincidence. My 2-slot mobo uses DDR2 memory that for some reason is expensive, even on eBay. Good suggestion though. I'm fishing around to find 2x4GB DDR2 240pin boards to eliminate that as an issue. It may be cheaper to get a new socket 775 mobo with 8GB of DDR3 memory than to buy the DDR2 boards. A friend has a 32-bit setup with 2GB RAM and has no problem, even using ZFS (he claims). Custom jails. It's on a 100Mbit network so maybe he can't test it adequately. Perhaps the 64-bit version is fussier.

 

[solarisguy]

Yes, 64-bit version needs 8GB minimum, 16GB is better.

 

[NiceTry]

8GB it is. I'm on the hunt for memory.

 

[solarisguy]

Your idea of a new system was a better one (since apparently with your memory type borrowing some RAM might be very difficult).

 

[joeschmuck]

Although the recommended system RAM is 8GB minimum for the version of FreeNAS which you are running, I'm still a bit perplexed about why there isn't a slow down with the jail disabled, and it only happens during a write operation. Write operations typically do not stick around in RAM very long so it's not a heavy RAM user whereas reading operations do cache to the RAM more so.

Now there are some odd reports about CIFS with respect to 9.2.1.4 and 9.2.1.5 and those could be related to the issue. I'd roll back to 9.2.1.3 and try again.

But no matter how you slice it, if you are going to use ZFS file system, you should have ECC RAM and at least 8GB of it for FreeNAS 9.2.x flavors.

An alternative for MiniDLNA is Plex, you just need to turn on the DLNA service within the configuration GUI of Plex. I still prefer MiniDLNA for simple just DLNA support however this could be an option.

 

[solarisguy]

An alternative for MiniDLNA is Plex, you just need to turn on the DLNA service within the configuration GUI of Plex. I still prefer MiniDLNA for simple just DLNA support however this could be an option.

Plex is not known to cause slowdowns you are experiencing. However, on occasions Plex appears to exhaust RAM.

 

[joeschmuck]

I think that is because Plex does much more than DLNA, but you can choose to not use the transcoding or servicing of the Plex portion, that might work. It's worth a try but I think rolling back to 9.2.1.3 is the best thing to try first.

 

[NiceTry]

I've found PLEX does too much, actually. And its organization can be irritating. For instance, it groups music into genre. Great function. But you can't play everything in a genre because its further organization by artist gets in the way. A lot of navigating. It misidentified many of my videos and missed cataloging about 150 files. I've found miniDLNA has a more useful and simple sorting scheme. Plus, the problem seems to be with the jail, not necessarily the plugin unless the install for miniDLNA does something odd. Perhaps installing the plex server plugin is worth a shot just as a diagnostic. Or, I could just create an empty jail and see what happens. Opinion?

As far as the weirdness with slowdown on writes and the interaction with jails, I'm baffled. I don't know enough about how FreeNAS is designed to even intelligently guess. I'm just working through observation and experiment. I appreciate everyone's ideas.

I did install cyberjock's CIFS patch. In my fumbling around, I needlessly turned on CIFS and couldn't turn it off. The patch worked.

I've never regressed versions before. Is it as simple as burning a new USB with 9.2.1.3 and plugging it in?

 

[solarisguy]

Is it as simple as burning a new USB with 9.2.1.3 and plugging it in?

Yes, it is that simple. You need to reconfigure miniDLNA once again and re-index the files.

When changing a major version you might be against backward ZFS incompatibility, but on 9.2.1.x versions there is no problem.

 

[solarisguy]

I meant, install 9.2.1.3 on a new USB (or one that was zeroed)

 

[NiceTry]

Installed 9.2.1.3 on a USB. Boot hangs at a point where it seems to be checking some off-site address. Will try entering single/safe user mode and set IP parameters. Also, the boot log complains about smartd not starting. Right now family is enjoying movies and won't let me muck with it.

 

[solarisguy]

Boot hangs

Did you wait forever ?

 

[NiceTry]

Pretty much. I waited at least 10 minutes with no change to the boot screen. I'm trying 9.2.1.2. If that works, I'll GUI upgrade to 9.2.1.3.

 

[NiceTry]

9.2.1.3 installed. Autotune selected. Had to fuss through boot options to get it to come up first time then made changes through GUI. Initial tests have good server performance read and write. Giving it some time to build databases and do some testing before attempting miniDLNA install.

Auto-import of ZFS worked perfectly.

 

[solarisguy]

I recall that when I booted 9.2.1.5 for the first time, I had to wait more than 10 minutes. More like 30 minutes. I had actually given up, but got distracted and when I returned the console menu was waiting for me.

Now..., if you had to play with boot options... That is interesting. What went wrong? What did you have to change from the defaults?

 

[NiceTry]

I recall that when I booted 9.2.1.5 for the first time, I had to wait more than 10 minutes. More like 30 minutes. I had actually given up, but got distracted and when I returned the console menu was waiting for me.

Now..., if you had to play with boot options... That is interesting. What went wrong? What did you have to change from the defaults?

Since the boot hung on what appeared to be a network routing problem, I wanted to make sure the gateway and static IP were correct. Boot Options just happened to appear and I grabbed at it. It may have been a fluke that I was able to get to the console menu by intervening at that first FreeNAS boot screen. I hit the spacebar when it popped up and selected "Boot Options." A couple returns later, I had the Console. Magic, I guess. Not sure how it happened or why but I was happy to be able to set up the networking and gateway. I'm not sure going through the Boot Options selection did anything either but I was able to set basic networking parameters. FreeNAS was not working so I rebooted after that. Maybe if I were more patient, it just would have gotten there anyhow. There were fewer exceptions cited as it booted. Smartd started and reported on the disks and up popped the console screen again. From the web GUI, I finished the static networking, named the machine, auto-imported ZFS, set users, email and selected auto-tune.

Nothing really custom, just finishing the package through the menus.

I'll try my luck with miniDLNA tomorrow. Why did you and joeschmuck recommend going back to 9.2.1.3?