zephyrus00jp
Cadet
- Joined
- Feb 25, 2016
- Messages
- 9
Thank you for sharing this great piece of software, FreeNAS and
providing the prepackaged plugin of owncloud.
I have a couple of questions regarding owncloud plugin.
I have run an installation of owncloud on Debian GNU/Linux on my main
desktop machine at home.
But now, I obtained an old HP Microserver which I intend to use as
24x365 home NAS, I thought it is a good idea to run owncloud there.
But when I tried to use owncloud plugin and learn how it fares, I realized I
am facing a few issues. So here are the questions.
Major Question 1: the version of bundled php in owncloud plugin is too old.
It is 5.4 and 5.6 seems to be the preferred choice today.
Is there a plan to update the version of the PHP in the prepared
plugin of owncloud?
Or is there a way for a third-party contributor like me to create
such a plugin EASILY? (I wonder if there is a shell script or
something to automate the creation of the prepared PLUGIN with updated PHP 5.6 and share such plugin?)
Major Question 2: can we switch to MySQL database from SQLite in the
owncloud plugin ? (It seems that the prepared PLUGIN is is pre-configured to
use SQLite if I am not mistaken. A fresh install of "raw" owncloud
usually asks us to select sqlite or mysql, but that questoin is not shown on
the first web login to owncloud plugin.)
Being able to convert SQlite to MySQL during run-time is very nice.
The suggested "php occ ..." command in owncloud manual for database conversion does not seem
to work with the prepackaged PLUGIN.
(The command needs to be issued under "www" user which owns owncloud's
config.php. Otherwise the above command refuses to run at all. Currently
config.php is owned by www. Usually, I tinker with the configuration
files under root account. I found out "su www ..." does not work.
"www" user is not usable from "su".
I see the error message: www user is not currently available, etc.
Also, the non-standard location of the PHP scripts, etc. [under /var/pbi/...]
may be interfering with the conversion process. But I am not sure abou this.
At least, on Debian GNU/linux installation of owncloud, when I
temporarily changed the ownership of config.php to "root" and tried
the conversion of database from the command-line, it worked there.
Debian uses "www-data" user for owncloud and thus config.php is owned by it.
But running such a command under root does not sound nice. I may change the permission of
files in unexpected ways.)
Anyway, I could not convert the SQlite db to mysql db easily.
Sub-question 2-b:
What is the recommended manner to update the database (from SQLite to
MySQL) from the viewpoint of the developer of the plugin who created it:
the choice of users such as www, the particular file layout chosen
and other factors obviously affects it.
This is NOT a general owncloud question, but very particular to the way the packaging of PLUGIN was
done. That is why I am asking this question No 2-b here.
Sub-question 2-c: plugin of owncloud that works with mysql
If it is not easy to change the database in the prepared PLUGIN, is
there a plan to offer owncloud plugin that pre-configures MySQL?
(Well, the user can create the MySQL database, user, and its password
from the Jail's shell feature once, and can proceed to setup owncloud
from its own web interface as is documented in owncloud manual after
the jail is populated with the prepackaged owncloud plugin..)
So I think it is just a matter preparing plugin of owncloud with necessary
mysql binaries, etc. in a desired directory hierarchy in avance.
Again, a standard script to install the desired packages to the
desired places in the selected file layout would be very handy.
BTW, the following is the performance situation gleaned by the use of
DTrace and WHY I want to switch to mysql.
Background: why I want to switch to MySQL: SQLite slows down owncloud.
Using SQLite is a performance hog.
I CAN NOT describe the performance problems below WITHOUT DTrace.
Kudos to FreeBSD developers and FreeNAS developers to incorporate
DTrace in the OS kernel and distribution.
With owncloud server that uses SQLite,
uploading takes a looooong time after many small files are added to
local directory that is managed by owncloud sync app under windows
(and other devices from the horror stories I read) .
Using DTrace on FreeNAS, I found that owncloud with SQLite on the server side
OPENs the database for each upload of a file, and SCANs (READs) a
VERY LARGE portion of database, THEN CLOSEs it at the end.
And the server REPEATs this. You can see the inefficiency immediately.
Real-world Example:
I have about 40 GB worth of files, among which are small data files
that I wanted to share using owncloud. (Now I count 63K+ files, but
this number contains a triplicate of latest data files that triggered the
slow performance issue that led me to post this inquiry.
These data sets contain 44921 files (strange the number is not multiple of three.) totaling 919MB.
(So actually, I began to experience problem when the number of files
exceeded about 18K+ and the size of the data is about 39GB. These were
mostly text files, MS-Office doc/ppt/xls files, JPEG photos and such.
The current figures contain the triplicates of a particular data set,
two of which have been added for performance benchmarks.)
For this dataset, the created SQLite owncloud.db on the server was almost 64MB.
Using DTrace, I found out that during the slow upload phase (a couple
of uploads or so every 10 seconds) after many small files are added to
local Windows PC, I saw owncloud.db is opened on the server, and then a lot
of reads of 1K block totalling a few dozen MB (close to 30-50MB
each 10 sec (!) of this db file, and then the file is closed and the
process repeats for each upload. No wonder the upload of a small file
is so slow. I mean, for uploading a small file of a few KB,
the server reads and scans and answer SQL queries against 64MB
database. Simply reading 30-40 MB takes close to half a second. And interpreting the database metastructure for the data adds overhead, of course.
This process is insane. Obviously, the way owncloud uses SQlite, it recreates the in-core data structure
by reading that many data and handles the SQL queries and start all
over for each uploaded file (!).
Converting sqlite database to mysql seems to correct the situation
SOMEWHAT since now mysql daemon sits on the server with the database
already read from the disk. It does not have to open and read the
database for each upload. This improvement may be a deal killer for
many users, and thus I am posting this inquiry.
Re the improvement by using mysql: Not perfect IMHO.
I tested the owncloud, nginx, mysql, php-fpm combination under another
jail to see the speed improvement. There, the large amount of read of DB for each upload obviously
disappears and, now, according to DTrace, too many calls of "lstat" by
PHP code becomes a problem although the speed measured by elapsed wall
time is probably is 2-3 x faster. But there is a room for improvement:
I mean "so what if one day upload becomes 8 hour upload" (!).
I think the throughput of small file upload is still not something on a par with Dropbox, its main contender.
If you are interested in "lstat" issues , see, for example,
php.webtutor.pl/en/2011/06/02/running-php-on-nfs-huge-performance-problems-and-one-simple-solution/
)
People who complained on the slow owncloud performance can see the innards of the issue using DTrace very easily. I recommend those who interested to try that themselves.
Again, thank you for sharing this great piece of software, FreeNAS and
plugins.
Although I have installed owncloud with mysql under another jail,
I prefer to use prepackaged plugin if the issues of outdated PHP and
database selection(MySQL) are solved.
This is because the more people use the prepackaged PLUGINs, the more
kinks are ironed out in the configuration, and security issues will be
scrutinized by more eyeballs IMHO.
(Debian's packages and their configurations are well thought out IMHO
thanks to such user feedback and contributions.)
TIA
providing the prepackaged plugin of owncloud.
I have a couple of questions regarding owncloud plugin.
I have run an installation of owncloud on Debian GNU/Linux on my main
desktop machine at home.
But now, I obtained an old HP Microserver which I intend to use as
24x365 home NAS, I thought it is a good idea to run owncloud there.
But when I tried to use owncloud plugin and learn how it fares, I realized I
am facing a few issues. So here are the questions.
Major Question 1: the version of bundled php in owncloud plugin is too old.
It is 5.4 and 5.6 seems to be the preferred choice today.
Is there a plan to update the version of the PHP in the prepared
plugin of owncloud?
Or is there a way for a third-party contributor like me to create
such a plugin EASILY? (I wonder if there is a shell script or
something to automate the creation of the prepared PLUGIN with updated PHP 5.6 and share such plugin?)
Major Question 2: can we switch to MySQL database from SQLite in the
owncloud plugin ? (It seems that the prepared PLUGIN is is pre-configured to
use SQLite if I am not mistaken. A fresh install of "raw" owncloud
usually asks us to select sqlite or mysql, but that questoin is not shown on
the first web login to owncloud plugin.)
Being able to convert SQlite to MySQL during run-time is very nice.
The suggested "php occ ..." command in owncloud manual for database conversion does not seem
to work with the prepackaged PLUGIN.
(The command needs to be issued under "www" user which owns owncloud's
config.php. Otherwise the above command refuses to run at all. Currently
config.php is owned by www. Usually, I tinker with the configuration
files under root account. I found out "su www ..." does not work.
"www" user is not usable from "su".
I see the error message: www user is not currently available, etc.
Also, the non-standard location of the PHP scripts, etc. [under /var/pbi/...]
may be interfering with the conversion process. But I am not sure abou this.
At least, on Debian GNU/linux installation of owncloud, when I
temporarily changed the ownership of config.php to "root" and tried
the conversion of database from the command-line, it worked there.
Debian uses "www-data" user for owncloud and thus config.php is owned by it.
But running such a command under root does not sound nice. I may change the permission of
files in unexpected ways.)
Anyway, I could not convert the SQlite db to mysql db easily.
Sub-question 2-b:
What is the recommended manner to update the database (from SQLite to
MySQL) from the viewpoint of the developer of the plugin who created it:
the choice of users such as www, the particular file layout chosen
and other factors obviously affects it.
This is NOT a general owncloud question, but very particular to the way the packaging of PLUGIN was
done. That is why I am asking this question No 2-b here.
Sub-question 2-c: plugin of owncloud that works with mysql
If it is not easy to change the database in the prepared PLUGIN, is
there a plan to offer owncloud plugin that pre-configures MySQL?
(Well, the user can create the MySQL database, user, and its password
from the Jail's shell feature once, and can proceed to setup owncloud
from its own web interface as is documented in owncloud manual after
the jail is populated with the prepackaged owncloud plugin..)
So I think it is just a matter preparing plugin of owncloud with necessary
mysql binaries, etc. in a desired directory hierarchy in avance.
Again, a standard script to install the desired packages to the
desired places in the selected file layout would be very handy.
BTW, the following is the performance situation gleaned by the use of
DTrace and WHY I want to switch to mysql.
Background: why I want to switch to MySQL: SQLite slows down owncloud.
Using SQLite is a performance hog.
I CAN NOT describe the performance problems below WITHOUT DTrace.
Kudos to FreeBSD developers and FreeNAS developers to incorporate
DTrace in the OS kernel and distribution.
With owncloud server that uses SQLite,
uploading takes a looooong time after many small files are added to
local directory that is managed by owncloud sync app under windows
(and other devices from the horror stories I read) .
Using DTrace on FreeNAS, I found that owncloud with SQLite on the server side
OPENs the database for each upload of a file, and SCANs (READs) a
VERY LARGE portion of database, THEN CLOSEs it at the end.
And the server REPEATs this. You can see the inefficiency immediately.
Real-world Example:
I have about 40 GB worth of files, among which are small data files
that I wanted to share using owncloud. (Now I count 63K+ files, but
this number contains a triplicate of latest data files that triggered the
slow performance issue that led me to post this inquiry.
These data sets contain 44921 files (strange the number is not multiple of three.) totaling 919MB.
(So actually, I began to experience problem when the number of files
exceeded about 18K+ and the size of the data is about 39GB. These were
mostly text files, MS-Office doc/ppt/xls files, JPEG photos and such.
The current figures contain the triplicates of a particular data set,
two of which have been added for performance benchmarks.)
For this dataset, the created SQLite owncloud.db on the server was almost 64MB.
Using DTrace, I found out that during the slow upload phase (a couple
of uploads or so every 10 seconds) after many small files are added to
local Windows PC, I saw owncloud.db is opened on the server, and then a lot
of reads of 1K block totalling a few dozen MB (close to 30-50MB
each 10 sec (!) of this db file, and then the file is closed and the
process repeats for each upload. No wonder the upload of a small file
is so slow. I mean, for uploading a small file of a few KB,
the server reads and scans and answer SQL queries against 64MB
database. Simply reading 30-40 MB takes close to half a second. And interpreting the database metastructure for the data adds overhead, of course.
This process is insane. Obviously, the way owncloud uses SQlite, it recreates the in-core data structure
by reading that many data and handles the SQL queries and start all
over for each uploaded file (!).
Converting sqlite database to mysql seems to correct the situation
SOMEWHAT since now mysql daemon sits on the server with the database
already read from the disk. It does not have to open and read the
database for each upload. This improvement may be a deal killer for
many users, and thus I am posting this inquiry.
Re the improvement by using mysql: Not perfect IMHO.
I tested the owncloud, nginx, mysql, php-fpm combination under another
jail to see the speed improvement. There, the large amount of read of DB for each upload obviously
disappears and, now, according to DTrace, too many calls of "lstat" by
PHP code becomes a problem although the speed measured by elapsed wall
time is probably is 2-3 x faster. But there is a room for improvement:
I mean "so what if one day upload becomes 8 hour upload" (!).
I think the throughput of small file upload is still not something on a par with Dropbox, its main contender.
If you are interested in "lstat" issues , see, for example,
php.webtutor.pl/en/2011/06/02/running-php-on-nfs-huge-performance-problems-and-one-simple-solution/
)
People who complained on the slow owncloud performance can see the innards of the issue using DTrace very easily. I recommend those who interested to try that themselves.
Again, thank you for sharing this great piece of software, FreeNAS and
plugins.
Although I have installed owncloud with mysql under another jail,
I prefer to use prepackaged plugin if the issues of outdated PHP and
database selection(MySQL) are solved.
This is because the more people use the prepackaged PLUGINs, the more
kinks are ironed out in the configuration, and security issues will be
scrutinized by more eyeballs IMHO.
(Debian's packages and their configurations are well thought out IMHO
thanks to such user feedback and contributions.)
TIA