sactobob
Dabbler
- Joined
- May 12, 2015
- Messages
- 23
I'm working on a server that ultimately will be used for critical backups of VM images, but also have the ability to spin up a VM (temporarily) if needed. Currently I have 10x3tb nas drives and 2x512 Samsung SSD Pros, 2xquad 5620s and 72GB ECC DDR3. My initial thoughts is raid10 the nas drives with 5x5 and mirror the Samsung (only need 200GB for one datastore). The SSD speeds are of course crazy fast, but I don't seem to be getting performance that I would expect from a 5x3tb stripe either locally or over a 1gb dedicated port.
So to make sure, in a striped volume speed is not limited to 1 drive like ZFS correct? What's also different from what I was expecting is to see 2 volumes mirror (like my hardware raids), but when I use the wizard setup, I see 5 mirrored volumes, it appears the data is spread over all 5 as total space available reflects 5x3tb. Is this correct with the wizard setup? It also leads me to believe, unlike hardware raid10, that I can lose more then one mirror (of course as long as it's not the same mirrored pair) and still have a working datastore?
Here's a snippet of an iozone on a vm over iscsi:
....
Are these normal speeds? I plan on bonding 2x1gb ports for final install. The iozone locally yields faster speeds, but still not what I would expect of 5 stripes. So am I getting a stripe of 5 volumes that are mirrored with all data spread across (and pulled from) the stripes with Wizard or do I need to manually set this up? Thanks for any help!
-Bob
So to make sure, in a striped volume speed is not limited to 1 drive like ZFS correct? What's also different from what I was expecting is to see 2 volumes mirror (like my hardware raids), but when I use the wizard setup, I see 5 mirrored volumes, it appears the data is spread over all 5 as total space available reflects 5x3tb. Is this correct with the wizard setup? It also leads me to believe, unlike hardware raid10, that I can lose more then one mirror (of course as long as it's not the same mirrored pair) and still have a working datastore?
Here's a snippet of an iozone on a vm over iscsi:
Code:
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
64 4 232652 752329 998622 1336792 1119387 842003 735831 877797 1255511 821391 852702 821391 1124074
64 8 293434 1163036 1304314 2298136 1780008 1279447 863676 1330167 1638743 1336792 1210227 831569 1492919
64 16 323102 1304314 1599680 2133730 2067979 1768283 1124074 1638743 1991276 1780008 1526887 1138375 1828508
64 32 438043 1304314 2203800 2772930 2467108 2006158 1330167 2444640 1879725 2379626 1562436 1183548 1933893
64 64 500060 1518251 2772930 3057153 2892445 2203800 1392258 2298136 1492919 1734015 1828508 743988 1991276
128 4 258508 748976 1015251 1440082 1175271 921183 766076 955616 992724 508012 477301 914904 1332828
128 8 310724 976473 1526043 1911894 1206978 1471662 1229084 1598754 1852520 576161 631014 1266785 1772988
128 16 325809 1024942 2035099 2905056 2464907 2066432 1409828 1618028 2464907 1162547 1254941 1373754 2004703
128 32 576780 1346196 2409592 3380677 3124872 2660335 1710838 2784517 2784517 1561553 1421023 1406136 2558895
128 64 511887 1217930 2132084 4012317 3895854 1684006 1967960 2985839 2027414 1778862 1319723 1561553 2132084
128 128 597981 1440082 3536566 3867787 3657016 1778862 1967960 1885042 2166499 1939522 1561553 976473 2453642
256 4 309185 762164 1208072 1463250 880256 962301 910872 1019886 1103757 880256 844929 1052888 1274008
256 8 385018 1027695 1718521 2187712 1632308 1652404 1376935 1802167 1855098 1185399 1108314 1471270 1924938
256 16 536654 1286217 2152625 3823789 2783115 2131261 1752173 2486631 2975955 1400278 1376935 1506359 2228578
256 32 797843 1534342 2812272 4132864 3770085 3454704 2065659 2719671 3557725 1375171 1497953 1437779 2582316
256 64 820398 1579483 2747508 4132864 3756894 2458168 1752173 3410808 3511189 1778290 1523457 1497953 2539563
256 128 794889 1642294 3275543 3935921 2152625 1729594 2228578 130343 3159869 1842366 1729594 1629830 2639446
256 256 839644 1683493 4496299 4652146 4404088 2148318 2392442 2165650 2345409 2242541 1805197 927393 2699161
512 4 331207 758612 1158418 1402845 1105920 872320 1049187 1030558 1032044 837942 760762 970041 1267097
512 8 438011 1073309 1723770 2216629 1635797 1476130 1463057 1766304 1856390 1343179 1108203 1305615 1823291
512 16 585176 1361920 2081286 3201349 2560168 2438090 2057359 3012719 2560168 1640796 1551867 2081286 2304647
512 32 792486 1580419 2548017 4305250 3177664 3527371 2256220 3934520 4447925 1734911 1546280 1904125 2958758
512 64 839580 1501940 2750343 4447925 3366987 2995907 2427068 3934520 4163357 1947291 1541840 1842059 2509312
512 128 876951 1560891 3437042 5331314 3970896 2572435 2405320 91789 3628697 1931527 1882427 1854787 2875558
512 256 915847 1695195 3201349 4871723 3762197 2152206 2314583 171185 3459188 2207515 1803387 1605228 2370799
512 512 988345 1983258 5177083 5278892 5384786 2357784 2572435 2216629 2860238 2235086 1801873 1038532 2979282
1024 4 374818 718005 1203474 1442326 1130312 844168 981750 1042770 1130312 817340 724179 1141730 1333507
1024 8 531633 1000969 1646334 2353657 1882266 729344 1395464 1771262 1783768 1284839 1029029 1284839 1893052
1024 16 591201 1220575 1969440 3355952 2789291 2509027 1848247 2867506 2348509 1630707 1389594 1617808 2321847
1024 32 851195 1460969 2410457 4195100 3908759 3251778 2265510 4018473 3618930 1964935 1512415 1889720 2910252
1024 64 942952 1307525 2448941 4488860 4379019 3271594 2260740 4079544 4634161 1599731 1590254 1786736 2867506
1024 128 1054547 1572783 2586074 4146499 3140041 2694787 2565989 77929 4379019 1796451 1692394 1787480 2892612
1024 256 976172 1689730 3181915 4972145 4570058 2852271 2813041 133476 4199201 1939207 1835608 1819280 2688041
1024 512 1019987 1771993 3835457 5536137 5507740 2250080 2908281 190617 3984917 2220992 1878973 2598591 4574926
1024 1024 1595571 2567523 7472033 8199542 8000971 3849207 4550690 3001782 4614246 3894582 2586074 1580306 4634161
2048 4 524710 1233688 2083516 2793651 2060032 1004458 1814210 1830447 2068963 1042239 763562 1972971 2450202
2048 8 909873 1571864 3136379 4502520 3452815 1504681 2772014 3536690 3663388 2144371 1708146 2649737 3671217
2048 16 1026545 2066474 3784436 5611381 5020987 3908414 3396834 3194702 2569678 1652925 1445434 3012103 4130162
2048 32 1386411 2301200 4741090 7425563 6523312 5754221 4552633 7189398 6603549 3683812 1659632 2406275 4796687
2048 64 1588726 2546821 4643695 8498106 7286980 6649556 4178378 8634786 7782107 3663388 2531808 3555722 4994712
2048 128 1670606 2589040 4983122 8357547 8059569 7262337 4562305 67956 4751580 1779262 2089598 3093459 5056455
2048 256 1768638 1899669 4686767 7088541 7609754 5225638 3999400 88558 4656281 1899669 2079481 2805512 4312636
2048 512 1784066 2455104 2913021 3089009 3329682 2734075 2258247 123388 4112366 1840251 1729468 1657071 2615847
2048 1024 1113723 1788524 3465352 4120256 4213218 2303669 3234398 197702 3542524 2085540 1250207 1596106 2585923
2048 2048 961939 1867050 3819776 4301837 3764533 1738569 2409650 1673209 2503765 1381950 1409617 915107 1802788
4096 4 455979 714226 1139036 1171902 979423 757262 935172 1026533 935121 799852 916513 1888560 2354114
4096 8 884657 1563255 1830994 2517637 3119752 2505155 2550529 3495533 3416292 1665070 1655602 2553562 3486312
4096 16 1074157 1351276 3813675 5926533 5271829 1851716 2205712 2868663 3023131 1805024 2031724 3091124 4235870
4096 32 1662974 2427973 4096486 6919538 6804432 5826043 4366128 7392988 6847827 3706707 2514321 3324403 4801789
4096 64 1736251 2420789 4357270 7877762 7877762 6451792 4171080 8788530 6597984 3608605 2465960 3319264 5075585
4096 128 1780148 2532855 4796426 8514134 8240513 7158858 4643451 67579 4751325 2972395 2529871 3212508 5026578
4096 256 1842581 2634219 4367238 7572208 7502762 6420447 4108241 119201 7774379 2292236 2498960 3139133 4551192
4096 512 1838440 2707279 4791076 8193352 8193352 5389246 4526013 128521 7099689 3618486 2162674 1706585 2268027
4096 1024 1157064 1657040 3168077 3494822 3509816 2516162 2747545 145129 4442909 1629380 1776283 1639018 2152378
4096 2048 1063057 1558009 3153539 3878244 3540193 1566962 2033889 209460 3531460 1631081 1471281 1575440 1748977
4096 4096 1071411 1592526 2962145 2929815 2825261 1511406 2419425 1582697 2340643 1497051 1213198 673250 1304885
8192 4 452174 602973 1129444 1147167 921078 691081 1063347 1073781 955105 660487 671471 1088063 1115184
8192 8 588373 806381 1501224 1598439 1358962 1149931 1604410 1955612 1475755 938487 923157 1330493 1361924
8192 16 684882 1330081 2214071 2229732 1829805 1606585 1886673 3012742 2020708 1383977 1325565 1734131 1880993
8192 32 982020 1910062 4354953 5970616 6154564 4920620 4336814 7407022 6145757 2594008 1969738 3477070 4222755
8192 64 1447347 2078150 4652131 7230880 7453619 5829805 2579403 8543361 5830794 3271793 1928826 2319283 2611357
8192 128 1202238 2420244 3133924 3844314 3905045 3258451 2190498 66780 2398785 1597176 1509534 1657264 2129937
8192 256 1175141 1591922 2455002 3273040 3328849 2971317 2559993 113266 3787530 2282007 1346025 2875338 3223903
8192 512 1345604 2229153 2579210 3192451 3168897 2925775 3114889 120148 5527801 1880581 1527520 1891972 1942566
8192 1024 1045455 1885328 2797840 2897891 3691893 2581147 2735254 129348 5204597 2023207 1367616 1983611 2175933
8192 2048 1066416 1746648 2793064 2996191 2798752 2290831 2605614 149317 4059608 1236548 1460202 1835181 1909107
8192 4096 1075900 1671859 2617325 3840876 2612350 1445337 2778833 218622 2836411 1610048 1287035 881905 1513657
8192 8192 1089650 1566591 2238301 2263665 2297879 1545382 2241953 1479759 2285650 1663683 1427681 725795 1359876....
Are these normal speeds? I plan on bonding 2x1gb ports for final install. The iozone locally yields faster speeds, but still not what I would expect of 5 stripes. So am I getting a stripe of 5 volumes that are mirrored with all data spread across (and pulled from) the stripes with Wizard or do I need to manually set this up? Thanks for any help!
-Bob
Code:
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
64 4 182882 429630 1210227 1679761 1204796 492717 1066042 500060 1310683 500060 444572 969761 1429325
64 8 484710 901377 2052169 3057153 1947927 939223 1336792 942521 2379626 901377 842003 1638743 2379626
64 16 673098 1363961 4564786 4564786 3363612 1599680 1780008 1526887 3203069 1722886 1204796 1780008 2923952
64 32 720041 2379626 4564786 7100397 4564786 2467108 3022727 2561267 4274062 2689580 1518251 1562436 3588436
64 64 1780008 2203800 7940539 7940539 5860307 2133730 2467108 2133730 3738358 1947927 2203800 1304314 3791156
128 4 144327 455822 1406136 1455701 1243315 509943 1000121 506097 1217930 465304 465708 1185654 1451764
128 8 392792 826202 2905056 2985839 2367096 969421 1539168 962469 2464907 1007629 836500 1663139 2367096
128 16 531131 1291158 3053774 4407601 3560017 1642783 2166499 1658003 3380677 1471662 1598754 2066432 2985839
128 32 927549 2714132 5325799 6406138 4934216 2248149 1967960 2660335 4934216 2511022 1802755 2066432 3785961
128 64 1618028 2558895 6045455 8548124 6114306 2286447 3199360 2843510 5325799 2248149 2608629 2420455 4012317
128 128 2286447 2969325 7082197 8548124 6727225 2558895 3297617 2727923 4407601 2727923 3053774 1278855 3982553
256 4 222039 445202 1543163 1561112 1319408 498069 1148625 474098 1354356 517265 457143 1108314 1415041
256 8 362651 907792 2783115 2812272 2392442 907792 1718521 898674 1985448 822913 812944 2102055 2371308
256 16 891956 1224606 3823789 4332998 3197509 1639786 3159869 1543163 3545976 1579483 1639786 2692393 3078337
256 32 1347557 2000242 5569035 6398720 5117791 2665657 3935921 2699161 5022044 2081678 2015259 2366082 3935921
256 64 2223962 2665657 7314033 7735574 6554972 3705040 4572895 3499745 6398720 3009318 2726577 3326279 4332998
256 128 2639446 3043436 7571923 7965107 7120034 3326279 4496299 3511189 5810112 3421677 3654598 2509882 4197489
256 256 2872459 3499745 7735574 8024634 7314033 2783115 4332998 3410808 5569035 3511189 3823789 1266495 4197489
512 4 296990 458785 1344020 1579257 1336492 474078 1086890 462440 1168504 433152 505908 1160923 1394645
512 8 656561 971357 2830083 2796910 2317080 949875 1803387 915847 2402629 966113 819085 2216629 2337255
512 16 884172 1347393 4061007 4228947 3768799 1689859 2474613 1667552 2708713 1267845 1747618 2891043 3046915
512 32 1438555 2073249 5624545 6156675 5331314 2979282 3104171 2337255 5886650 2256220 3029721 3762197 3905895
512 64 2039772 3064306 6395018 7871843 6909408 2975154 5278892 4375425 7410080 2815243 3064306 3304808 4532414
512 128 2753870 3795443 5439343 8265729 7647578 5067142 4827914 4447925 8140399 3201349 3556580 3239989 4420457
512 256 2910635 3684733 5639315 7758090 7435738 3158966 4494470 4228947 6018636 3634839 3788747 2811557 4237291
512 512 3140488 4000485 8394979 8362289 7988981 3259661 5453155 3684733 5870558 3736016 3905895 1450212 4411377
1024 4 367568 495165 1519371 1565332 1333507 480754 1098511 476487 1368780 511987 532556 1135391 1390944
1024 8 593981 950044 2730767 2791104 2375790 950885 2142335 897438 903099 584126 794510 1572783 2265510
1024 16 895194 1351126 3999762 4095102 3696803 1668074 3556008 1657133 3712781 1812371 1378445 2255990 3092561
1024 32 1508167 2086145 3937426 6172653 5507740 2877110 3541348 2877110 4095102 2015654 2822283 2908281 4110780
1024 64 1927891 3401130 6479979 8000971 4920874 3121783 5277632 4589593 8183918 2908281 3314514 3808251 4674510
1024 128 3335105 3999762 6208343 8326715 7869040 3311958 5330028 5226256 8391792 3403826 3955557 3239514 4550690
1024 256 3101494 3791442 6059442 7883484 7590887 4898425 5219904 4789183 7524394 3390391 4048778 3142339 4282950
1024 512 3003881 3999762 6918376 8457895 8137402 3751699 5383475 4249053 6963241 3791442 4064103 2910252 4451639
1024 1024 3469823 4095102 8750850 8822754 8610501 3863055 5682634 3643490 5958564 3631168 4321737 1528563 4531485
2048 4 376754 471125 1561009 1559876 1310279 474849 1309280 487848 1331606 512012 473097 1333259 1390450
2048 8 726494 859783 2547576 2805512 2370418 875558 2288326 941176 2409650 1014422 1007876 1870302 2384234
2048 16 1044393 1822292 4104506 4330028 3757946 1756343 3657149 1692329 3702868 1843410 1705094 3051697 3112514
2048 32 1503627 2135840 5949508 6110328 5535444 3024831 5475461 2968386 5251194 2905140 2102384 2832339 4112366
2048 64 1858566 3470953 5816563 7318020 6802261 3084572 5293261 4321315 5119743 3287628 3472356 3567536 4666399
2048 128 3102397 4188565 5475461 8190205 8029434 5535444 5916724 5868220 8987113 3744839 3938881 3715681 4552633
2048 256 3606479 3924484 5264067 7874853 7670915 5095447 4889518 5503525 7589584 5044577 4031308 3351769 4303993
2048 512 3102397 4130162 5738844 8464610 8292998 5567735 6167360 5264067 7616502 3612546 4376355 3246623 4469720
2048 1024 3126106 4469720 7368238 8830061 8531869 3922692 6167360 4623699 7013303 3969827 4472047 2909075 4511980
2048 2048 5147354 4432815 8574451 8866519 5104531 3112514 5447681 5689430 6518362 4215286 4347560 1511565 4472047
4096 4 387660 460798 1531209 1588844 1311559 454965 1130492 486753 1303598 528516 472379 1351808 1415401
4096 8 773700 944893 2146730 2830382 2318218 923956 2295912 926497 2294686 979925 845932 2250793 2412291
4096 16 1102205 1801994 2927818 4231697 3706707 1658960 3686820 1672526 3489144 1836082 1722498 3068488 3162828
4096 32 1527261 2146730 5689064 6225028 5549401 2830848 3605576 2888438 5218980 3029528 3091124 3189841 4167033
4096 64 1953188 3737349 6500617 7939654 7476641 3325046 4922872 4768469 6758923 4226492 3471518 3531460 4735608
4096 128 3167493 4032070 5319164 8548024 8142867 5483860 5006072 6160296 5139356 2952980 3972402 3919833 4666153
4096 256 3059199 3697135 5179642 7906767 7642951 5204750 5818150 5519094 8013724 3857345 4200656 3214912 4274869
4096 512 4934183 4153935 6735075 8501494 8292220 4054910 5198450 5397712 7969117 3673419 4104315 3537277 4447510
4096 1024 3838385 4476482 7392988 8770583 8655693 4248440 6389405 5367357 8013724 4214051 3599532 3319264 4516494
4096 2048 3778448 4003879 6361016 8752710 8660056 5818150 6379914 5044289 6724530 4352854 4145915 2936325 4226492
4096 4096 3640724 4214051 6872481 8480511 8256354 5603704 5936773 4027344 6158088 4063542 4505833 946820 2934319
Last edited:
