Does TrueNAS Core support what I want to do?

[pavan_pudipeddi]

Currently, my client is using "Synology NAS" for securely storing some sensitive files (only the client team of less than 10 people will have access to these files). They're in the process of migrating to TrueNAS Core (on prem).

They want us to automate some of the stuff they do. A cron job should do the following (not limited to) :

• Access a given directory on TrueNAS
• If there are any files, process those files
• Place the original and updated files in another directory on TrueNAS system.

We will be using NodeJs/Go-lang for the automation.

I know how to do this in a Linux based system and placing the resultant files in either on a Linux directory or on any cloud storage. I went through the documentation and searched forums, but still am not clear how do I go about the TrueNAS integration.

My questions are -

• How do I achieve this in TrueNAS?
• I learned that TrueNAS core doesn't support Docker. So, let's assume that our cron job is running in a Docker container on a different machine/server altogether. Is it possible for such a program to access the TrueNAS machine within the same network (but on a different machine) through SFTP/SSH etc.?

Any help is greatly appreciated.

 

[sretalla]

let's assume that our cron job is running in a Docker container on a different machine/server altogether. Is it possible for such a program to access the TrueNAS machine within the same network (but on a different machine) through SFTP/SSH etc.?

Yes.

API calls would normally be the way to get TrueNAS to do something from outside the system:

CORE API Reference

Describes how to access the API documentation in TrueNAS CORE.

www.truenas.com

But if you just want to move files around, SSH should be enough.

• How do I achieve this in TrueNAS?
• I learned that TrueNAS core doesn't support Docker.

I'm not sure that Docker is the right answer here, but it can be had via a linux VM in TrueNAS if needed.

If all you're doing is a cron job, CORE has that in the Tasks section of the interface already, no need for any additional services or components.

If you will run a script with node/go, you should do that in a jail, where you can install whatever you want and mount the needed storage locally.

 

[pavan_pudipeddi]

Thank you for such a quick turn around.

I am looking at some API examples now.

In the meanwhile, here are a few more details that might help explaining my rationale behind choosing Docker and APIs -

• We will have to read PDF files and modify PDF files, create Excel files, connect to 3rd party API to post some data there. This is the reason for choosing cron jobs.
• The reason for choosing Docker is that the template of the PDF files we will be reading from and writing to will keep changing (once/twice a year). So, we felt that keeping our app (or cron job) within a Docker container would make the deployments easier whenever something changes.

I'm not sure that Docker is the right answer here, but it can be had via a linux VM in TrueNAS if needed.

So you're saying that we can have TrueNAS => LinuxVM => Docker? (I haven't used VMs much, will figure this out).

Thanks again.

 

[sretalla]

So you're saying that we can have TrueNAS => LinuxVM => Docker?

Yes. Plenty of people were/are doing that.

• We will have to read PDF files and modify PDF files, create Excel files, connect to 3rd party API to post some data there. This is the reason for choosing cron jobs.
• The reason for choosing Docker is that the template of the PDF files we will be reading from and writing to will keep changing (once/twice a year). So, we felt that keeping our app (or cron job) within a Docker container would make the deployments easier whenever something changes.

I suspect a jail can do all that (including hosting its own crontab) and will use significantly fewer system resources in the process, but working in a FreeBSD jail can have downsides if you're used to working in linux, so the choice is there for you to make.

 

[pavan_pudipeddi]

I see. I think I have a better idea of my options now.

One last question - is there a way for me to run TrueNAS core on a Mac (my computer) or in any Cloud VM?
If you could point me towards relevant documentation/resources, I'd be very grateful.

 

[jgreco]

I see. I think I have a better idea of my options now.

One last question - is there a way for me to run TrueNAS core on a Mac (my computer) or in any Cloud VM?
If you could point me towards relevant documentation/resources, I'd be very grateful.

There are ways, but they're almost universally all bad. VMware Fusion is a type 2 hypervisor which is strongly advised against, and most cloud VM's are under-resourced and/or have unsuitable attributes such as undesirable storage controllers. Please see

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

for known compatible ways to virtualize.

But if you just want to move files around, SSH should be enough.

NFS would also be a good way if you wanted the container/VM/etc to have full access to manipulate the files.

 

[sretalla]

One last question - is there a way for me to run TrueNAS core on a Mac (my computer) or in any Cloud VM?

I think you're asking because you want a test/dev environment, in which case @jgreco's advice is valid, but not as important as you're not really storing the important data there... making sure to have external copies of any development you do in such a VM.

If you're suggesting that will be the final solution for the client, forget it (on your PC).

 

[jgreco]

Fair enough. I had decided not to go back into the thread to comment on several other things, which would be irrelevant in a dev environment as well. Otherwise follow the rules if you value your data.

 

[pavan_pudipeddi]

I think you're asking because you want a test/dev environment

Almost - I am hoping to do a quick PoC to get my head around it understand all the challenges involved in implementing the client's requirements.

If you're suggesting that will be the final solution for the client, forget it (on your PC).

Of course it's not the final solution for the client

There are ways, but they're almost universally all bad.

This is scary, discouraging and challenging at the same time.

Based on what I have read so far, here are the 3 approaches I am thinking and the possible challenges in each of those -

Note: this is a step by step breakdown of the solution. I am trying to understand the possible challenges we may encounter with TrueNAS. I understand if this is not in scope for this community to address, but would be glad if you could point me in the right direction. Appreciate your patience and effort in assisting me

Approach 1:

• Have TrueNAS and our app/job on different machines in the same network
• Let the client upload files to the parent directory in TrueNAS
• The cron job runs periodically. It connects to the TrueNAS (through SFTP/SSH) and downloads any available files this directory
• The processed files will then be put back on the TrueNAS system (there are going to be at least 3 files, so I am thinking of a gzipped file).

Possible challenges in this approach:

• Uploading files from PHP web app to TrueNAS directory - is this technically feasible? (common for all the 3 approaches)
• Does TrueNAS support support SSH/SFTP connections? - it does (from the first response I received)
• Does TrueNAS support "gzip" files? - I am fairly certain that it does

Approach 2:

• Have TrueNAS and our app/job on same machines (TrueNAS Core => Linux => Docker)
• Let the client upload files to the parent directory in TrueNAS
• The cron job runs periodically and performs required file manipulations/creations directly in TrueNAS

Possible challenges in this approach:

• Setting up Docker and handling deployments - certainly feasible, but wouldn't be straightforward
• File manipulation directly on TrueNAS (this is common for Approach 3 as well)

Approach 3:

• Have TrueNAS and our app/job on same machines (our job runs on VM, instead of docker)
• Let the client upload files to the parent directory in TrueNAS
• The cron job runs periodically and performs required file manipulations/creations directly in TrueNAS

Possible challenges in this approach:

• Setting up Docker and handling deployments - certainly feasible, but wouldn't be straightforward
• File manipulation directly on TrueNAS (this is common for Approach 3 as well)

Based on your experience with TrueNAS, do you see any other challenges? Also, which of these approaches would you recommend?

 

[sretalla]

Perhaps I didn't make it clear enough the first time around, but I would add a 4th option and select that.

Have TrueNAS host the files and your app/cronjob in a jail on that same box.

Your app gets direct access to the data (via jail mounts) and can run with the needed software (pkg is available for node and go for installation in a jail) and a crontab set up in the jail (or on the host if you prefer a GUI) to launch the code.

For me, that's the absolute zenith of simplicity and will most likely work best also.

Potential challenges are for you to learn how to work in jails and understanding the jail mounts required.

Make yourself a TrueNAS VM (in VirtualBox) and test it out.

 

[pavan_pudipeddi]

Perhaps I didn't make it clear enough the first time around, but I would add a 4th option and select that.

Have TrueNAS host the files and your app/cronjob in a jail on that same box.

Your app gets direct access to the data (via jail mounts) and can run with the needed software (pkg is available for node and go for installation in a jail) and a crontab set up in the jail (or on the host if you prefer a GUI) to launch the code.

For me, that's the absolute zenith of simplicity and will most likely work best also.

Potential challenges are for you to learn how to work in jails and understanding the jail mounts required.

Make yourself a TrueNAS VM (in VirtualBox) and test it out.

Understood. Thanks a ton.

I'll give a shot at this.

Thank you @sretalla and @jgreco for your time and guidance. Really appreciate it.

 

[jgreco]

This is scary, discouraging and challenging at the same time.

My gut feeling is that this is because I think you're paddling upriver.

TrueNAS is a fileserver. For example, it doesn't "support gzip files". What does that even mean? Files are files. TrueNAS is essentially just BSD UNIX with lots of fileservice stuff added. It kinda expects a network client to attach via something like NFS (UNIX-y), AFP (Apple), or SMB (Windows) and then access files through that. This is easy and powerful; for example, I have a video processing workflow for captured ATSC TV streams that involves multiple VM's including both Windows and Linux that convert from a .ts, pull out closed caption subtitles, convert to .srt subtitles, and then re-encode it ending up in a .mkv file. You don't do the work on the NAS. If the only things you needed to do were FreeBSD-based, you could do it in a jail, which has the significant benefit of not running lots of traffic over the network, but also limits your workflow flexibility. All the systems just have a datastore mounted and do their work on the shared file tree. But "support gzip files" ... is it supposed to do what Windows does with built-in ZIP file manipulation capabilities? No, it doesn't really do that, yet you are certainly not forbidden to work with gzip files.

Try to view the NAS not so much as a computer or server but really more as a smart network hard disk. You appear overly focused on SSH and SFTP, for example. Or let's consider the following:

Uploading files from PHP web app to TrueNAS directory - is this technically feasible? (common for all the 3 approaches)

WHAT? That doesn't make much sense. That would mean you'd need some sort of temporary directory on whatever machine was running the PHP web app, and then have a separate step to somehow copy that over to TrueNAS. Why would you do that? Why not just mount the ingest directory for whatever you're doing directly on the PHP web app machine? That way the PHP app would have a chance to do things like report failures such as disk full, etc. File comes into PHP via web and then immediately hairpins back out onto the network to the NAS via NFS for storage. Clean, elegant. Fairly resilient to many errors. Avoids unnecessary handling. You could run the PHP web app in a TrueNAS jail to eliminate one network hop as well.

 

[pavan_pudipeddi]

My gut feeling is that this is because I think you're paddling upriver.

TrueNAS is a fileserver. For example, it doesn't "support gzip files". What does that even mean? Files are files. TrueNAS is essentially just BSD UNIX with lots of fileservice stuff added. It kinda expects a network client to attach via something like NFS (UNIX-y), AFP (Apple), or SMB (Windows) and then access files through that. This is easy and powerful; for example, I have a video processing workflow for captured ATSC TV streams that involves multiple VM's including both Windows and Linux that convert from a .ts, pull out closed caption subtitles, convert to .srt subtitles, and then re-encode it ending up in a .mkv file. You don't do the work on the NAS. If the only things you needed to do were FreeBSD-based, you could do it in a jail, which has the significant benefit of not running lots of traffic over the network, but also limits your workflow flexibility. All the systems just have a datastore mounted and do their work on the shared file tree. But "support gzip files" ... is it supposed to do what Windows does with built-in ZIP file manipulation capabilities? No, it doesn't really do that, yet you are certainly not forbidden to work with gzip files.

Try to view the NAS not so much as a computer or server but really more as a smart network hard disk. You appear overly focused on SSH and SFTP, for example. Or let's consider the following:



WHAT? That doesn't make much sense. That would mean you'd need some sort of temporary directory on whatever machine was running the PHP web app, and then have a separate step to somehow copy that over to TrueNAS. Why would you do that? Why not just mount the ingest directory for whatever you're doing directly on the PHP web app machine? That way the PHP app would have a chance to do things like report failures such as disk full, etc. File comes into PHP via web and then immediately hairpins back out onto the network to the NAS via NFS for storage. Clean, elegant. Fairly resilient to many errors. Avoids unnecessary handling. You could run the PHP web app in a TrueNAS jail to eliminate one network hop as well.

Thanks for the detail.

I read a bit more about Jails and have a slightly better idea about it now. I understand where you're coming from.

I'll play around with TrueNAS and Jails and shall knock at the community doors as and when the needs arises.

Thanks again for all the inputs. Really appreciate it.