72 TB brutto storage for backup

Status
Not open for further replies.

wussy

Explorer
Joined
Jan 11, 2012
Messages
50
Hi guys,

At work we are going with EMC Storages. One of them is dedicated for videos and photos. There are aprox. 40 TB data on it. We were searching for a backup solution for the datas.
The situation was that it should be as cheap as possible.

We decided to go with a case from XCASE (http://www.xcase.co.uk) . They have quite cheap cases, a really good service and fast shipping.
The case we use was the following:

X-Case RM 424 - 24 Hotswap Bay 6GB Mini SAS (SATA/SAS Backplane) - Rail Kit included with 900 Watt Psu for 600 Euro. (http://www.xcase.co.uk/ProductDetails.asp?ProductCode=CASE-XCASE-RM424)
It can take 24 drives and has 6GBs Backpanels already built in. You get everything shipped with it what you need.

As motherboard we got an Intel® Server Board S5520HC for 375 Euro.
We use two Intel Xenon 5600 and 64 GB RAM.

For the SAS Connection to the back panels we use an Adaptec Raid 52445. It has 6 internal and one external SFF connectors.
Every disk is used as single volume. With this configuration Freenas 8.0.4 64 bit can see the disks.


After flashing the newest firmware on the raid controller every things seems to work fine.
It´s now running with 8x 3 TB discs because we are waiting for the SAS Cables to be delivered.

The plan was to use one big Raidz2 with 2 spare disks. After making some research and testing I realized that the spare discs don´t pop in automated.
That´s really disappointing.
So we decided to set up a normal RAIDZ2 without spares.
Also the replacing for disks don´t work for me. I tried it on the GUI but even on the shell it didn´t worked. I think I have to play around a little more with the shell commands.
Also I didn´t managed to get SMART running until now.

The worse thing about it is that we won´t realize when a disk is gone and even if we would´t be able to change it in this state :smile:

Means we have to work on a solution, but as the storage is "only" for backup our EMC it isn´t a deal breaker.

One note here: I really can´t understand why all this and more (like hotplugin) is working on the ix system but not in the FreeNAS. I can understand that the want to sell the IX Storages and it´s not necessary to put all buissnes functionality in the free system but in my opinion it is a bad reputation when the free variant has not an easy way to change disks or even a bullet proofed notification about disk health.
I think that many companies that are middle-sized (means 50 employes) will get the free system a shot and when everything runs smoothly go with the IX systems. We were really thinking about replacing our EMC (and we have three of them) because the cost a lot of money but in my situation I can´t say let´s go and buy an IX System because there´s no good feeling after having this kind of troubles with the free system and IXsystems machines are too expensive to say okay let´s try it and see what will happen.
And yes i know then there is support and they are tested and so on, but in my opinion it could be a great way to advertise the commercial product when the basics for a NAS work great in the free system.

Maybe somebody has an idea how I can get following things to work:

replacing disks
get spares popped in
SMART
maybe hot plugging (only a niche to have)

I know you need LOG files to help. Maybe you can tell me what LOGS would be interesting?!

Has somebody checked a Freenas with Nagios without SNMP Traps? Has somebody some input for me concerning NAGIOS checks?

Will post some performance datas after having all disks online. Could take up to two weeks :-(

Critical words and opinions are very welcome.
 

wussy

Explorer
Joined
Jan 11, 2012
Messages
50
Some pictures

image3.jpg image2.jpg image1.jpg image.jpg
 

wussy

Explorer
Joined
Jan 11, 2012
Messages
50
Hi,
Here my performance datas:

Write:
dd if=/dev/zero of=/mnt/store/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 164.120493 secs (654239946 bytes/sec)

Write 2:
dd if=/dev/random of=/mnt/store/tmp.dat bs=512k count=20480 conv=sync
20480+0 records in
20480+0 records out
10737418240 bytes transferred in 170.497767 secs (62976885 bytes/sec)

Read:
dd if=/mnt/store/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 161.410506 secs (665224248 bytes/sec)

Read 2:
dd if=/mnt/store/tmp.dat of=/dev/zero bs=512k count=20480 conv=sync
20480+0 records in
20480+0 records out
10737418240 bytes transferred in 3.290633 secs (3263025187 bytes/sec)


Config:
FreeNAS Version FreeNAS-8.0.4-RELEASE-x64 (10351)
Plattform 2x Intel(R) Xeon(R) CPU E5607 @ 2.27GHz
Speicher 65461MB
Adaptec Raid 52445

RaidZ2 24x 3TB (one spare) with 24 3TB Barracuda 7200 RPM 64MB 6GB/s (ST300DM001)
 

monarchdodra

Explorer
Joined
Feb 15, 2012
Messages
79
HI, being an enthusiast home user who has more or less run into the same problems as you, here is the info I can give you (as far as I have understood it):

-FreeNAS 8 is a relatively new project, and a lot of "important features" are not quite fully stable yet. Its getting there though.
-FreeNAS 8.0.4 uses a kind of database for its discs, meaning that doing anything not via the UI will desync the UI from the actual zpools :/ What's more, if you do anything via the command line, it will not show up in the GUI, confusing you into thinking that "it isn't working"
-FreeNAS 8.2.0 (BETA 2) doesn't actually use a database for its drives, so you can do anything you want via CLI, and it will show up in the GUI.

FreeNAS 8 is built on FreeBSD 8, itself having a bug with its ZFS implementation that prevents failed drives from being automatically detected, and hot spares from kicking in. Hot spares should should be useable by hand though. According to the developers, this issue is in their "Top 5".
See here: http://forums.freenas.org/showthread.php?6299-HDD-failure-several-problems

My suggestion is to try 8.2.0 BETA 2, and see if it has enough functionality for you. I noticed a lot of difference between the two.
 

wussy

Explorer
Joined
Jan 11, 2012
Messages
50
Thx for your answer. Do you know what the differences between 8.2 Beta and 8.0.4 are?
And most important do you know why 8.2.0 is in BETA?

It´s really important for me that it´s stable.
Do you have any experience?

I´ve already seen the Thread you have linked but i was thinking changing disks from the CLI works without desyncing the GUI

Thx again for answering
 

wussy

Explorer
Joined
Jan 11, 2012
Messages
50
Okay found it. But it don´t seem to have the problems with hotspares fixed.
Here the featureset:


New plug-in API allows the installation of non-core NAS features. The API provides the ability to install arbitrary software on a FreeNAS™ system using an easy-to-use packaging method that automatically creates the GUI extensions needed to control the application, all in a format that will survive upgrades.

BitTorrent, DLNA/uPNP, and iTunes plugins are available and instructions for creating your own plugins are here.

Built-in DAAP, DLNA, and torrent support.

An informational icon indicates which graphical fields contain a tooltip.

ZFS is now integrated in that any ZFS operations that are performed at the command line are now reflected in the GUI. This means that you can now create pools, datasets, snapshots, and zvols in either the command line or the GUI and they will stay in sync with each other.

Support for multipath devices on systems containing dual expander SAS backplanes, SAS drives, or dual expander JBODs with SAS drives. Such hardware will be automatically configured for multipath.

The GUI now includes a webshell which allows you to access a root shell from within a web browser.

"Create Volume" was renamed to "Volume Manager".

Extending existing ZFS pools is more intuitive.

Creating a volume now uses a multiselect widget instead of checkboxes to improve ease of use.

When you create or import a ZFS volume, a periodic scrub task is automatically created to run every 35 days. These tasks can be managed in Storage -> ZFS Scrubs.

An autotuning script is now available and disabled by default. It sets various tunables and sysctls based on system resources and components. The predetermined values are exposed through the GUI from the Sysctls and Tunables panes.

GUI is now compatible with Android and iOS mobile devices.

FreeNAS™ can now be configured as an iSCSI initiator.

Okay will test it out. Has anybody experinces how stable this beta is?!
 

monarchdodra

Explorer
Joined
Feb 15, 2012
Messages
79
Hi, I can understand that from a business perspective, you do not want to be using "BETA" software.

AFAIK 8.2.0 is a branch to incorporates the FreeNAS 8's Plugin system. The rest should be just as stable as 8.0.4 (or more stable).
You can find a list of release notes in the , and in particular,
[URL="http://www.freenas.org/about/news/item/release-notes-for-freenas-820-beta2?category_id=136"]8.2.0 BETA 2
.

My experience overall is that 8.2.0, while being a beta, is also newer, more robust, and more functional (I've seen myself a few bugs in both 8.0.3 and 8.0.4 that were fixed in 8.2.0). You may want to stay away from the plugins though, that is the part in beta. Relevant to you are:
ZFS can be manipulated from the CLI <-- Manipulating ZFS through the CLI in 8.0.3 or 8.0.4 DOES desync the GUI. Don't do it.
The GUI now includes a webshell. <-- Can be useful for a quick developper debug if you can't ssh.

For an exact list of changes, you can also look at the tickets road map.

Worst case scenario, if anything doesn't work with the UI, you can always fallback to command line. ZFS is built into FreeBSD's kernel, so that's kind of a robustness guarantee (in the sense that it is just as robust in all version of freenas)

Good luck
 

wussy

Explorer
Joined
Jan 11, 2012
Messages
50
Hi,
have already installed the beta.

Looks like changing disks works from gui now. that would be great if this is really working.Makes live more easy.
If only the plugins are beta it won´t bother me at all.

THX again for answering and for making the effort. Will post on when i have tested everthing out.

Best regards
 

wussy

Explorer
Joined
Jan 11, 2012
Messages
50
Hi,
Seems like Freenas 8.2 don´t utilize all the RAM. The write / read performance is much weaker then before.

memory.jpg

Tried also to run the auto tuning script. nothing changed.
Has anybody an advise for me?
 

wussy

Explorer
Joined
Jan 11, 2012
Messages
50
Think I solved the problem by adding systables

kern.ipc.nmbclusters 50000 Maximum number of mbuf clusters allowed
vfs.zfs.arc_max 60675M Maximum ARC size
vm.kmem_size 49168M Size of kernel memory
vm.kmem_size_max 61461M

Are they correct when having 65461MB RAM total?

Best regards
 
Status
Not open for further replies.
Top