nemesis1782
Contributor
- Joined
- Mar 2, 2021
- Messages
- 105
@Arwen: I'll get back to you on most of the things you said in your last two posts. Your posts deserve a well thought through reply and I feel I have some reading up to do before I can you a give you a meaning full reply.
As for the bricked system, yeah I could do that and thank you for the advice! I will not be doing that, for many reasons however they're personal and I prefer not discussing them in an open forum. If you'd like to know why I do mind sharing the reasons through PM though, just ask! You've been a great help!
On the topic fragmentation:
Ah ok then it makes sense. So just to verify I understood what you said:
- A virtual block device like a iSCSI LUN probably has a lot of continuous changing data and thus fragmentation logicly becomes an issue.
-->This is then mittigated by a rule of use stating 50% usage where it won't become such an issue.
--> Another way of mitigating this might be to stop the VM during a service window in my use case moving the iSCSI lun away and back if I understand you correctly. I mean for me I could shutdown the VM's in the night for a few hours and do this. My LUN's will not be to big anyway.
- Searching data and a empty slot to put data is slowed considerably by fragmentation in ZFS and thus for "fragmentation heavy" workloads you need a lot free space to mittigate this as much as possible.
It seems to me that ZFS itself is ill suited for iSCSI then. I mean iSCSI is a virtual block device. If and when I start using iSCSI, I do not want this. Basicly when using iSCSI the consumer is responsible for any data in the block device. You want the block device to claim a piece of disk space on something that is preferably fast and has some disk level redundancy. If you need further redundany you setup replication and a second redundant iSCSI endpoint.
Often for iSCSI we would use raid5 or 6 for speed and some parity. Then have the SAN replicate the data to another DC with the same setup over a 10GB dark fibre. VMWare would then switch (which almost never happened) to the backup iSCSI endpoint if the main one died. In my case for anything iSCSI
Is there really no way to have storage in TrueNAS if ZFS just doesn't fit the use case for the pool you're making?
Personally I'm against abstract rules to simplify because as clearly can be seen around the internet to much regurgitation happens without understanding what they are regugitating. Thank you for taking the time to explain it!
On the BPR:
Yeah, that would be great. Basicly it's just moving reference around. Not sure how that would fix fragmentation though. It wouldprobably fix many of the search issue. This of course is pure conjecture ;)
On the iSCSI part:
Well my question is more, if that is the limitation why even use ZFS for that use case instead of traditional methods. Since I do not have any actual experience with ZFS yet I'll go deeper into this once I have a test system and performed some comparissons.
Facts:
- You have to rely on the hardware RAID card and it's drivers for telling you, and potentially recovering from, bad disk blocks or bad disks.
--> Agreed, that is what they're made for and I do see how this goes against the ZFS philisophy. When looking at it as a Architect I would say that is a consiquence of the abstraction. You add a extra level of indirection which has pro's and con's. I do see how heavy the cons are though even if they are difficult to quantify.
- If the RAID card fails, you generally have to replace with a similar model. (Or server with similar model of RAID card.) You can't just put the disks into another server and import your ZFS pool.
--> Yeah agreed that is a huge dissadvantage, for the DC's we'd have a replacement configured lying on the shelves in a preconfigured state
--> Not just a simular model, also the exact same firmware and configuration
--> Hardware RAID has some undeniable advantages though, which I will not be going into further since we've concluded it is a bad idea in this case and for my use case
- OS side SMART won't work
--> Yeah obviously since you're abstracting the hardware. SMART is handled by the RAID card. (Unless you use it as a IT in which case it should work fine)
- RAID cards can and do limit writing speeds. This is because ZFS bundles writes into transactions that will write across multiple disks. They can be larger than a hardware RAID card's battery backed up cache memory.
--> Hmm, yes I hadn't considered that. I would formulate it differently however I do agree. In the case of ZFS and how it handles data writes it'll defenitly impact performace SEVERELY!
- RAID cards can and do limit reading speeds. ZFS can trigger reads across all your data disks. This is especially noticeable during a scrub, which can read for hours. Even days. (But a ZFS scrub can't fix anything because it does not control the redundancy... see Note 1)
--> Well yeah in the proposed theoretical I assumed data scrubbing on zfs would be dissabled. Since that is the RAID cards responsibility.
- Last, ZFS will run fine on the hardware based RAID card, UNTIL IT DOESN'T!
--> Now we don't want that do we :P
- Their have been KNOWN cases of ZFS finding bugs in hardware RAID card firmware, (or in user implementations), the HARD way, meaning data loss. For example, not setting the hardware RAID card to automatically change write cache from write later to write through during battery failure. Then a crash occurs and you've lost data that was supposed to be "securely written".
--> Yes, however i would like to argue that a misconfigured ZFS system would probaly be just as bad. Furthermore the code base for ZFS (haven't dared to take a peak yet) is probably extremely complicated and thus has a lot of room for possible bugs. Not saying it's bad or not mature this is just a fact of software.
Other software vendor:
So yes, I assume your talking about synology in this case. This is a setup I am currently using. However in the past i have had a volume crash due to this leaving me with a read only version of my data (12TB at the time). Synology was/is convinced that this is due to memory issues eventhough numerous tests on the memory by me as well as Synology have come back possitive. Just to be clear I've been using Synologies for 10 years now and have had 2 "Fatal" issues. Both of which support helped me resolve. The other one being the stupic C2000 bug which isn't really Synologies fault.
Note1:
Cool. I definitly see the benefits of ZFS and I think TrueNAS is the best option for freeNAS software. However everything has con's and I would like to be as well informed as I can be before handing the keys to the missus :P
The rest I'll come back to. Again thnx. you've really helped me jump start my trek into ZFS and TrueNAS!
As for the bricked system, yeah I could do that and thank you for the advice! I will not be doing that, for many reasons however they're personal and I prefer not discussing them in an open forum. If you'd like to know why I do mind sharing the reasons through PM though, just ask! You've been a great help!
On the topic fragmentation:
Ah ok then it makes sense. So just to verify I understood what you said:
- A virtual block device like a iSCSI LUN probably has a lot of continuous changing data and thus fragmentation logicly becomes an issue.
-->This is then mittigated by a rule of use stating 50% usage where it won't become such an issue.
--> Another way of mitigating this might be to stop the VM during a service window in my use case moving the iSCSI lun away and back if I understand you correctly. I mean for me I could shutdown the VM's in the night for a few hours and do this. My LUN's will not be to big anyway.
- Searching data and a empty slot to put data is slowed considerably by fragmentation in ZFS and thus for "fragmentation heavy" workloads you need a lot free space to mittigate this as much as possible.
It seems to me that ZFS itself is ill suited for iSCSI then. I mean iSCSI is a virtual block device. If and when I start using iSCSI, I do not want this. Basicly when using iSCSI the consumer is responsible for any data in the block device. You want the block device to claim a piece of disk space on something that is preferably fast and has some disk level redundancy. If you need further redundany you setup replication and a second redundant iSCSI endpoint.
Often for iSCSI we would use raid5 or 6 for speed and some parity. Then have the SAN replicate the data to another DC with the same setup over a 10GB dark fibre. VMWare would then switch (which almost never happened) to the backup iSCSI endpoint if the main one died. In my case for anything iSCSI
Is there really no way to have storage in TrueNAS if ZFS just doesn't fit the use case for the pool you're making?
Personally I'm against abstract rules to simplify because as clearly can be seen around the internet to much regurgitation happens without understanding what they are regugitating. Thank you for taking the time to explain it!
On the BPR:
Yeah, that would be great. Basicly it's just moving reference around. Not sure how that would fix fragmentation though. It wouldprobably fix many of the search issue. This of course is pure conjecture ;)
On the iSCSI part:
Well my question is more, if that is the limitation why even use ZFS for that use case instead of traditional methods. Since I do not have any actual experience with ZFS yet I'll go deeper into this once I have a test system and performed some comparissons.
Facts:
- You have to rely on the hardware RAID card and it's drivers for telling you, and potentially recovering from, bad disk blocks or bad disks.
--> Agreed, that is what they're made for and I do see how this goes against the ZFS philisophy. When looking at it as a Architect I would say that is a consiquence of the abstraction. You add a extra level of indirection which has pro's and con's. I do see how heavy the cons are though even if they are difficult to quantify.
- If the RAID card fails, you generally have to replace with a similar model. (Or server with similar model of RAID card.) You can't just put the disks into another server and import your ZFS pool.
--> Yeah agreed that is a huge dissadvantage, for the DC's we'd have a replacement configured lying on the shelves in a preconfigured state
--> Not just a simular model, also the exact same firmware and configuration
--> Hardware RAID has some undeniable advantages though, which I will not be going into further since we've concluded it is a bad idea in this case and for my use case
- OS side SMART won't work
--> Yeah obviously since you're abstracting the hardware. SMART is handled by the RAID card. (Unless you use it as a IT in which case it should work fine)
- RAID cards can and do limit writing speeds. This is because ZFS bundles writes into transactions that will write across multiple disks. They can be larger than a hardware RAID card's battery backed up cache memory.
--> Hmm, yes I hadn't considered that. I would formulate it differently however I do agree. In the case of ZFS and how it handles data writes it'll defenitly impact performace SEVERELY!
- RAID cards can and do limit reading speeds. ZFS can trigger reads across all your data disks. This is especially noticeable during a scrub, which can read for hours. Even days. (But a ZFS scrub can't fix anything because it does not control the redundancy... see Note 1)
--> Well yeah in the proposed theoretical I assumed data scrubbing on zfs would be dissabled. Since that is the RAID cards responsibility.
- Last, ZFS will run fine on the hardware based RAID card, UNTIL IT DOESN'T!
--> Now we don't want that do we :P
- Their have been KNOWN cases of ZFS finding bugs in hardware RAID card firmware, (or in user implementations), the HARD way, meaning data loss. For example, not setting the hardware RAID card to automatically change write cache from write later to write through during battery failure. Then a crash occurs and you've lost data that was supposed to be "securely written".
--> Yes, however i would like to argue that a misconfigured ZFS system would probaly be just as bad. Furthermore the code base for ZFS (haven't dared to take a peak yet) is probably extremely complicated and thus has a lot of room for possible bugs. Not saying it's bad or not mature this is just a fact of software.
Other software vendor:
So yes, I assume your talking about synology in this case. This is a setup I am currently using. However in the past i have had a volume crash due to this leaving me with a read only version of my data (12TB at the time). Synology was/is convinced that this is due to memory issues eventhough numerous tests on the memory by me as well as Synology have come back possitive. Just to be clear I've been using Synologies for 10 years now and have had 2 "Fatal" issues. Both of which support helped me resolve. The other one being the stupic C2000 bug which isn't really Synologies fault.
Note1:
Cool. I definitly see the benefits of ZFS and I think TrueNAS is the best option for freeNAS software. However everything has con's and I would like to be as well informed as I can be before handing the keys to the missus :P
The rest I'll come back to. Again thnx. you've really helped me jump start my trek into ZFS and TrueNAS!