[WIP] MusicBrainz mirror on FreeNAS

Status
Not open for further replies.

tonyschuite

Dabbler
Joined
Oct 8, 2012
Messages
12
OK, so working with Sickbeard, Couchpotato, sabnzbd and headphones is a dream. However, Headphones has one major problem: the musicbrainz scraping.
Installing a musicbrainz mirror on Ubuntu is a pain but doable. This is because the musicbrainz team develops on Ubuntu or similar.

Now, you could install a mirror on Ubuntu itself (I am still working on it, but I am getting there). This has the disadvantage that your ubuntu client must be running if the Freenas system is scraping.

I was fed up with the official musicbrainz mirror, so I decided it <u>should be possible</u> to compile on FreeBSD / Freenas.

Below is how far I got. The dependencies is the biggest pain, but so far it is mostly time consuming.
I am posting this in order to create a Howto, as n00b friendly as I can make it. Beware, this contains some advanced shell commands. I myself have never done anything on FreeBSD/freenas, so a lot of this could be done easier. Please tell me so I can change that in the Howto. Also the Musicbrainz server will take A LOT of space! around 8GB (make sure your Jail is big enough).
When I get this running, I will look into the ability to create a clean Database and a script that Headphones can connect to so as to update the needed Artist / Release.

This has all been done on FreeNAS-9.1.1-RELEASE-x64 (a752d35)

TIP: use 'tmux new -s session_name' to start a detached ssh session. these sessions can be resumed any time (provided your freenas box doesn't crash) using the command 'tmux attach -t session_name'.

Let's Begin:

// ___________________________________________

# First we need to do a portsnap inside the jail so we can compile existing ports for FreeBSD
# in ssh
# in jail (jexec 1 csh)

Code:
portsnap fetch extract

# TAKES AGES!!
# if already done with a different plugin --> portsnap fetch update
# Ports tree is already up to date.

# you can use vi as editor but i prefer nano
Code:
cd /usr/ports/editors/nano[/FONT][/COLOR][/FONT][/COLOR]
make install clean


# exit jail and re enter for it to work

##let's install postgres
# use default selected options when prompted
Code:
cd /usr/ports/databases/postgresql92-server
make install clean
cd /usr/ports/databases/postgresql92-contrib
make install clean


#grab a cup of coffee!

Code:
nano /etc/rc.conf


#add
Code:
postgresql_enable="YES"


#go to WebUI
system-->sysctls-->add Sysctl

#add (not sure if the values are the same with every system, but for me this worked)
Variable = kern.ipc.shmall value = 32768
Variable = kern.ipc.shmmax value = 134217728
Variable = kern.ipc.semmap value = 256

#in ssh
#exit jail

Code:
exit
nano /etc/sysctl.conf


#add
security.jail.sysvipc_allowed=1

#exit with :qw

Code:
jls

#look for jail id
Code:
jail -m jid=<insert jailID here> allow.sysvipc=1
 
jexec <insert jailID here> csh
/usr/local/etc/rc.d/postgresql initdb
/usr/local/etc/rc.d/postgresql start
 
#creating test database
su pgsql
createuser -P pg_test

#give it a password, give it no SU priviliges
Code:
/usr/local/bin/createdb db_test -O pg_test --encoding=UNICODE


#no message means SUCCESS!!
Code:
su pgsql
createuser -P -s postgres
# give it a password
exit


#check https://www.wunki.org/posts/2011-05-07-solid-freebasd-server-postgresql-90.html for more options (encryption and remote connection)
# That is where I got this from.

##postgres done
##Next up is the prerequisites of MB

##GIT

Code:
pkg_add -r git


##PhantomJS (not listed in official install guide but it gave me a headache during build)
Code:
cd /usr/ports/lang/phantomjs
make install clean


##PERL no need (preinstalled, perl -v to recheck)

##memcached
Code:
pkg_add -r memcached
 
nano /etc/rc.conf

#add
memcached_enable="YES"
Code:
/usr/local/bin/memcached -d -u nobody


## ICU
Code:
cd /tmp
fetch http://download.icu-project.org/files/icu4c/4.0/icu4c-4_0-src.tgz
tar -zxvf icu4c-4_0-src.tgz
cd icu/source/
./runConfigureICU FreeBSD
gmake
gmake install
echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig


## FINALLY!! MusicBrainz! (sadly we are nowhere near done )
Code:
mkdir /home/musicbrainz/
cd /home/musicbrainz
 
rehash
git clone --recursive git://github.com/metabrainz/musicbrainz-server.git
cd musicbrainz-server
 
cp lib/DBDefs.pm.sample lib/DBDefs.pm
 
nano lib/DBDefs.pm


# change MB_SERVER_ROOT to your desired location (I left it on default)
# add password, host and port to READWRITE, READONLY and SYSTEM
# in my case (local server) server and port are 127.0.0.1 and 5432
# change WEB_SERVER to "127.0.0.1:5000" (we will only be running a local mirror)
# change REPLICATION_TYPE to RT_SLAVE
# change DB_STAGING_SERVER to 0

# run the following to install dependencies:

Code:
su
cpan App::cpanminus
cpanm Carton
cpanm --uninst-shadows ExtUtils::MakeMaker
cpanm .


#take another cup of coffee!
Code:
 
pkg_add -r libxml2
cat Makefile.PL | grep ^requires > cpanfile
carton install --deployment

# in most instances this should complete. if not, edit the cpanfile with the contents of the following:
#

IF! The following three libraries failed to install, install them manually:
Code:
cpanm Hash::Merge
cpanm JSON::Syck
cpanm Net::CoverArtArchive::CoverArt


Code:
cd /home/musicbrainz/musicbrainz-server/postgresql-musicbrainz-unaccent
gmake
gmake install
cd ../
 
cd postgresql-musicbrainz-collate
gmake
gmake install
cd ../
 
vi /usr/local/pgsql/data/postgresql.conf


# remove the # before listen_adresses='localhost'
#quit using :wq

Code:
vi /usr/local/pgsql/data/postgresql.conf

# comment all lines that are uncommented by putting # in front of it
now add
Code:
local all all trust


# Sadly we need a full musicbrainz dump (PUN also intended) to be able to sync changes to the database.
# in a browser, go to http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/LATEST
# copy the code and insert in the following commands

Code:
cd /tmp
mkdir dumps
cd dumps
fetch http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/<insert code here>/mbdump-cdstubs.tar.bz2 && fetch http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/<insert code here>/mbdump-cover-art-archive.tar.bz2 && fetch http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/<insert code here>/mbdump-derived.tar.bz2 && fetch http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/<insert code here>/mbdump-edit.tar.bz2 && fetch http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/<insert code here>/mbdump-editor.tar.bz2 && fetch http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/<insert code here>/mbdump-stats.tar.bz2 && fetch http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/<insert code here>/mbdump.tar.bz2


# more Coffee!

Code:
cd /home/musicbrainz/musicbrainz-server/
./admin/InitDb.pl --createdb --import /tmp/dumps/mbdump*.tar.bz2 --echo


This takes a while.
after that you can run the server with

Code:
plackup -Ilib -r


refer to https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md
for further instructions on auto starting the server and loading changes.

carton exec -- ./admin/replication/LoadReplicationChanges

##___________________________________________
 

extera

Cadet
Joined
Apr 19, 2013
Messages
8
I'm following this!
I would really like this running on FreeNAS, I have the .OVA running in ESX, but i would like it on FreeNAS!

Do you have an idea when you will finish the how-to?

Thanks!
 

tonyschuite

Dabbler
Joined
Oct 8, 2012
Messages
12
it seems there is a part missing.
i got this running a while back.

sadly i am going to Thailand for a month, so I wont be able to redo this untill september at the earliest.
I will see what I can do this weekend.

regards Tony
 

calgarychris

Contributor
Joined
Aug 27, 2011
Messages
124
Definitely keen to see if this can be done!
 

tonyschuite

Dabbler
Joined
Oct 8, 2012
Messages
12
so after redoing my entire NAS, I am back to square one with this one. Up untill the musicbrainz dependencies it is a walk in the park. Right now cpan is giving me grief with older versions of deps installed.

I should mention that I did have it running on my old build.

will post updates!
 

tonyschuite

Dabbler
Joined
Oct 8, 2012
Messages
12
The howto seems to be fairly done.
it is still a huge pain in the A** but i got it working now on Freenas 9.1 x64
 

Trouloulou

Cadet
Joined
Jan 3, 2014
Messages
5
I'm trying to do it...

Really not easy, and thanks for the tip around the ExtUtils::MakeMaker. This class made me crazy before your advice ^^

I gonna share the final result ;)

MANY MANY thanks
 

tonyschuite

Dabbler
Joined
Oct 8, 2012
Messages
12
I was a bit quick on the draw. I now have an error running the server.
I am looking for an answer on the musicbrainz IRC.
So far everything has been compiled and installed. It just gives me a "strict subs" error. Which is a perl problem as far as I know.

Will update
 

Trouloulou

Cadet
Joined
Jan 3, 2014
Messages
5
Any idea about Redis Server...?

It seems that mine is not running ( nothing is listening on 6379 )...

caught exception in MusicBrainz::Server::Controller::Root->begin "Could not connect to Redis server at 127.0.0.1:6379: Connection refused at /usr/local/lib/perl5/site_perl/5.16/Redis.pm
 

Trouloulou

Cadet
Joined
Jan 3, 2014
Messages
5
I've installed /usr/ports/database/redis
And start the server.

Seems to be ok, now my HeadPhone refresh its data to my mirror server.... :)
 

tonyschuite

Dabbler
Joined
Oct 8, 2012
Messages
12
Damn ;) you got it working ;)
What perl version do you have and what freenas version?
 

Trouloulou

Cadet
Joined
Jan 3, 2014
Messages
5
My freenas is 9.1.1 and my perl is 5.16.

The only difference with your tuto, is that i've uninstall the perl package installed when git is installed and i've installed perl via /usr/ports/lang/perl...
 

tonyschuite

Dabbler
Joined
Oct 8, 2012
Messages
12
yes, this has worked for me. I have used these notes several times and got it working.
however, each time i faced difficulties that where different, even on the same system. You will probably face problems not explained in this guide.
my first recommendation would be to do this in a separate jail. chances are you will run into some problems and you wouldn't want to crash the entire system. also, you wont need to have it running 24/7.

right now i deleted the jail i had it running in, because of disk space (need to upgrade my drives to 3TB).
 

joelmusicman

Patron
Joined
Feb 20, 2014
Messages
249
So I *almost* got to the end, then I hit a dead end...

Code:
root@musicbrainz:/home/musicbrainz/musicbrainz-server # ./admin/InitDb.pl --createdb --import /tmp/dumps/mbdump*.tar.bz2 --echo
Can't locate Unicode/ICU/Collator.pm in @INC (@INC contains: /home/musicbrainz/musicbrainz-server/admin/../lib /usr/local/lib/perl5/5.16/BSDPAN /usr/local/lib/perl5/site_perl/5.16/mach /usr/local/lib/perl5/site_perl/5.16 /usr/local/lib/perl5/5.16/mach /usr/local/lib/perl5/5.16 .) at /home/musicbrainz/musicbrainz-server/admin/../lib/MusicBrainz/Server/Translation.pm line 13.
BEGIN failed--compilation aborted at /home/musicbrainz/musicbrainz-server/admin/../lib/MusicBrainz/Server/Translation.pm line 13.
Compilation failed in require at /home/musicbrainz/musicbrainz-server/admin/../lib/DBDefs/Default.pm line 32.
BEGIN failed--compilation aborted at /home/musicbrainz/musicbrainz-server/admin/../lib/DBDefs/Default.pm line 32.
Compilation failed in require at /usr/local/lib/perl5/5.16/parent.pm line 20.
BEGIN failed--compilation aborted at /home/musicbrainz/musicbrainz-server/admin/../lib/DBDefs.pm line 29.
Compilation failed in require at ./admin/InitDb.pl line 35.
BEGIN failed--compilation aborted at ./admin/InitDb.pl line 35.


So I tried to reinstall Unicode::ICU::Collator...

Code:
root@musicbrainz:/home/musicbrainz/musicbrainz-server # cpanm Unicode::ICU::Collator
--> Working on Unicode::ICU::Collator
Fetching http://www.cpan.org/authors/id/T/TO/TONYC/Unicode-ICU-Collator-0.002.tar.gz ... OK
Configuring Unicode-ICU-Collator-0.002 ... N/A
! Configure failed for Unicode-ICU-Collator-0.002. See /root/.cpanm/work/1394246438.85644/build.log for details.


Check the log file, and it says my OS is unsupported... Not sure what to do now, as some googling didn't reveal any obvious answers...

Log file output:
Code:
cpanm (App::cpanminus) 1.7001 on perl 5.016003 built for amd64-freebsd-thread-multi
Work directory is /root/.cpanm/work/1394246438.85644
You have make /usr/bin/make
You have LWP 6.05
You have /usr/bin/tar: bsdtar 2.8.5 - libarchive 2.8.5
You have /usr/bin/unzip
Searching Unicode::ICU::Collator on cpanmetadb ...
--> Working on Unicode::ICU::Collator
Fetching http://www.cpan.org/authors/id/T/TO/TONYC/Unicode-ICU-Collator-0.002.tar.gz
-> OK
Unpacking Unicode-ICU-Collator-0.002.tar.gz
Entering Unicode-ICU-Collator-0.002
Checking configure dependencies from META.yml
Checking if you have ExtUtils::MakeMaker 0 ... Yes (6.90)
Configuring Unicode-ICU-Collator-0.002
Running Makefile.PL
OS unsupported: No icu-config --cppflags found
-> N/A
-> FAIL Configure failed for Unicode-ICU-Collator-0.002. See /root/.cpanm/work/1394246438.85644/build.log for details.
 

joelmusicman

Patron
Joined
Feb 20, 2014
Messages
249
Ok, so I figured out that I did not properly install ICU and revisited that section. Got ICU done but still stuck...
 

joelmusicman

Patron
Joined
Feb 20, 2014
Messages
249
Still stumped. "cpanm ." is failing from quite a few dependencies.

Code:
Installing the dependencies failed:
Module 'Text::Unaccent' is not installed,
Module 'XML::Simple' is not installed,
Module 'GnuPG' is not installed,
Module 'XML::RSS::Parser::Lite' is not installed,
Module 'Redis' is not installed,
Module 'XML::SemanticDiff' is not installed,
Module 'XML::Parser' is not installed,
Module 'Unicode::ICU::Collator' is not installed,
Module 'XML::XPath' is not installed,
Module 'Javascript::Closure' is not installed
! Bailing out the installation for MusicBrainz-Server-0.01.
 

9C1 Newbee

Patron
Joined
Oct 9, 2012
Messages
485
This is REALLY cool. I can tell it is a bit out of my skill set at this point. I encourage you to perfect this and make a step by step guide that even a big ape like me could follow :)
 

joelmusicman

Patron
Joined
Feb 20, 2014
Messages
249
A pbi would be awesome for this project since its pretty much set and forget once replication is set up. I'm still stuck on it and have an Ubuntu VM running this on another machine. I'd still love to switch it to FreeNAS because my eventual goal is to sell the other box.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
PBI is extremely unlikely. Long story short many of the people that wanted to do PBIs gave up and left before 9.1 hit release.

I'm interested in this too, but I don't know if I could figure it out. And if I could figure it out I'm not sure I'd write a guide anyway.
 
Status
Not open for further replies.
Top