TechNazgul RSS

Monday, February 14, 2011

How to move your XBMC library while maintaining your watched history and working thumbnails



It’s fairly common that for one reason or another, xbmc users need to move their video library from one location to another.  Perhaps you’ve been using a single 1TB drive for your library, you’ve filled it up and upgraded to a 3TB drive and need to update the path for every movie and TV show to point to the new drive. 

 

Or, in my case, you’ve been using windows samba-based streaming and want to move to the much more efficient NFS streaming protocol (see the comparison detailed in this post).

 

The simple way to change file locations is just to remove your old library, move the content, and then rescan the new destination to recreate your library in the new location.  Unfortunately, this has a few big disadvantages, mainly that you lose each file’s watched status, watched count, and bookmarks, which is really annoying for large libraries.

 

Prerequisites

 

To use the solution below, there are a few things that will help ensure you can follow the steps without modification:

 

  • Your movies and TV should be sorted into individual folders, with thumbnails, trailers, nfo files, etc. all in each individual movie folder. For example, the directory for the movie 300 would include the following files:

 

300 [top-level movie folder]

-> .actors [folder]

-> extrathumbs [folder]

-> 300.bluray.mkv

-> 300.bluray.nfo

-> 300.bluray.tbn

-> 300.bluray-fanart.jpg

-> 300.bluray-trailer.mp4

 

  • If your library is not stored in this structure, try the ‘export library’ function within XBMC (settings –> video –> export library –> separate files) to dump all of these files to the movie directories.
  • You’ll want to download a tool like SQLite (if you are NOT using a shared MySQL library) or Navicat Lite (if you are using MySQL to host a shared library for multiple machines).  These will allow you to browse your database and update the paths where necessary.  I used Navicat, so those of you not using a shared library will have to modify the instructions slightly below to do the same thing in SQLite, which I’m not familiar with.
  • If using MySQL, you’ll need to remember your root password to the DB so that you can connect to it and manipulate the table data.
  • To be safe, you should make a copy of your video DB before starting this as you are mucking around with the sql database and there is no easy way to undo mistakes you make.  In Navicat, this is as easy as, right click on xbmc_video (your video database, and select “dump SQL file.”

 

image

 

Updating the video database with the new path

 

With those items laid out as pre-requisites, here is how it’s done:

 

  • Close xbmc
  • Move / copy your content to the new location
  • Open your database using one of the tools above
  • Open the ‘path’ table within the xbmc_video database (or MyVideos.db for non MySQL).  See the screenshot below for an example using Navicat Lite. [the username/password in my smb path has been blocked out]

 

image

 

  • You now need to open the MySQL console in Navicat to enter in the find/replace command for the path.  The MySQL console can be accessed using the F6 key, or under Tools\Console in the main menu.
  • You’ll use a command like the one below to do the find/replace.  This is a good succinct explanation of the find/replace command in MySQL if you want more info.

mysql> update path set strPath = replace(strPath, ‘old path’, ‘new path’);

 

  • To provide a real example, here is what I used:

mysql> update path set strPath = replace(strPath, ‘smb://username:password@servername/Movies/’, ‘/mnt/movies/’);

 

  • This replaces all SMB paths with nfs mounted paths in my root filesystem. Note that if you have different variations of the old path in your library (some with password, some without, or a separate smb://user:pass@server:/TV/ (/pictures, etc.) you’ll need to issue the above command several times until all old paths are replaced with the new location.  Make sure you have all items with the old path replaced and then move on below.

 

Now that this much is done, you have a library with an updated path pointing to your new file locations. However, a few problems remain to be resolved.  First, we have to update your sources.xml file to point to the new location as well.  This file is what tells XBMC where to look for future movie/TV updates.

 

  • Open your userdata folder and edit the file sources.xml.  Find all instances of your old path (smb://username:password@servername/Movies/ in my example) and replace with your new path.

 

Now, the hardest problem to solve.  How to get XBMC to rehash all of your thumbnail files so that they still work with the new file locations.  The reason this does not work automatically is that XBMC creates cached thumbnails based on the full path to the movie location.  For example, if the original path was smb://username:password@servername/Movies/300, XBMC creates and caches a thumbnail based on a thumbnail hash for that directory in the userdata/Thumbnails directory.

 

If you change that path in the library like we did above from smb:// to /mnt/movies/300, the cached thumbnail will no longer work and you get a big ugly screen capture or no thumbnail at all in place of the thumbnail you want.  The only way to force XBMC to rehash the thumbnail is to remove the item from your library or to manually refresh the information for every file in your library through a series of remote/keyboard clicks, so it’s totally impractical for a large library.

There are a lot of posts on the XBMC forums that involve using scripts to rehash the directory, but I couldn’t get any of these to work.  As I was about ready to give up and just delete my library and start over, I stumbled onto a much easier solution that makes XBMC do all the heavy lifting.

 

Fixing the xbmc thumbnail cache after moving your library

 

  • Start in Navicat, create a new database by right-clicking anywhere in the “connections” frame, call it anything you like… in this case, xbmc_video10 (yes, it took me 10 iterations to figure this out).  Enter the same options as indicated below.

 

 

image

 

  • This is all you need to do for now with Navicat.
  • Edit advancedsettings.xml in your userdata folder to change the database name from the original name to the new name xbmc_video10.

 

<videodatabase>
    <type>mysql</type>
    <host>mysqlserverIP address</host>
    <port>3306</port>
    <user>xbmc</user>
    <pass>xbmc</pass>
    <name>xbmc_video10</name>
</videodatabase>

 

  • Reopen xbmc.  You’ll be starting with a blank video database.
  • Optionally: you can delete all files in the thumbnail directory before starting this rescrape if you don’t want the extra thumbnails taking up space.  I didn’t do this originally as I wasn’t sure it was going to work when I started, but I’m fairly certain you could without issue.
  • Update your xbmc library (right-click somewhere in the library and choose update library).  This will force a rescrape of all of the sources you had previously defined in the new location.  While scraping the new locations, xbmc will properly create the cached thumbnail file based on the new location.  If you have  large library, this will take a fair amount of time.
  • When done, navigate through your TV & Movie libraries and make sure everything is working.  Thumbnails should show up, you should be able to play all files, etc.
  • Close xbmc

 

So at this point we have all of our thumbnails working, but we’ve lost all of our watched history, watched counts, proper ordering for date added the library, etc.  We have only one more step to pair the working thumbnails with complete library history.

 

  • Edit your advanced settings.xml file again.  This time, revert the change you made to the videodatabase name and enter the name of the original database. Save & close.

 

The next time you reopen xbmc, it will access your original library with updated paths, watched file history, and it will find all of the properly cached thumbnails that were created when you had it manually rescrape the library.  Victory!

 

 

Technorati Tags: ,,,

Tuesday, February 8, 2011

NFS vs. SMB throughput for HD media streaming using the NETGEAR XAVB5001 500 Mbps Powerline Adapter



NETGEAR XAVB5001 500 Mbps Powerline Adapter Review:


XAVB5001After recently dropping my satellite subscription and moving to MythTV, I needed to find a way to get HD video streams from my antenna and HD Homerun tuner to a TV in the house without wired internet.  Wireless (G or N) didn’t prove to be reliable or fast enough, so the Netgear 500 mbps (theoretical) powerline adapters seemed like a good bet.

Testing was performed in a ~3-year old house and the 2 adapters were several rooms apart from one another on the same floor.  With these adapters, performance can typically vary significantly based on wiring within the house, so YMMV.

As a preview of the results, while they didn’t achieve anywhere near the advertised 250 mbps (~31 MB/s) in each direction, they did perform much better than older generation powerline adapters I’ve used.  When I plugged in the adapters, the Netgear software reported that I was getting ~220 Mbps in each direction, which was encouraging, but ultimately not even close to accurate.

In addition to testing the adapters, I also wanted to test the efficiencies of different network protocols to see what would work best for streaming HD content to my XBMC / MythTV box.  Below are the results.

Testing was done using commands like the following on a Ubuntu Linux machine that tests reads and write speeds to network-mounted disk. 

Write: time dd if=/dev/zero of=/mnt/storagesmb/testfilesmb1 bs=16k count=16384
Read:  time dd if=/mnt/storagesmb/testfilesmb1 of=/dev/null bs=16k

Since the focus of my testing was primarily to verify read speeds from a media frontend to a backend storage server, I did not do significant write testing.

Disk Read/Write Test Results:

Protocol Test Type File Size Rsize (nfs) Throughput
SMB Write 256MB na 03.8 MB/s
SMB Read 256MB na 03.9 MB/s
NFS v3 Write 256MB na 05.4 MB/s
NFS v3 Read 256MB 16,384 09.3 MB/s
NFS v3 Read 256MB 32,768 10.8 MB/s

Next, a separate set of tests was done using iperf between two Ubuntu Linux machines.  The server resides on a large RAID array capable of reads and writes upwards of 500 MB/s, so disk speeds were not a limitation in this testing. 

In theory, the iperf test should remove most inefficiencies related to network protocol and show raw throughput. As it turns out, the results are very similar to the NFS testing above, which speaks well of the NFS protocol.

iperf results:

***ONE DIRECTIONAL TEST
------------------------------------------------------------
Client connecting to 192.168.1.111, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.182 port 42245 connected with 192.168.1.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  90.8 MBytes  75.9 Mbits/sec (9.49 MB/s)


***BIDIRECTIONAL (SIMULTAENOUS) TEST
------------------------------------------------------------
Client connecting to 192.168.1.111, TCP port 5001
TCP window size: 55.0 KByte (default)
------------------------------------------------------------
[  5] local 192.168.1.182 port 60753 connected with 192.168.1.111 port 5001
[  4] local 192.168.1.182 port 5001 connected with 192.168.1.111 port 59780
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  57.4 MBytes  48.0 Mbits/sec (6.00 MB/s)
[  4]  0.0-10.1 sec  43.0 MBytes  35.7 Mbits/sec (4.46 MB/s)

Combined => ~83.7 Mbits/sec (10.46 MB/s)



SMB vs. NFS Conclusion:
While this is probably no surprise to many, NFS is vastly more efficient.  Using the same commands and testing read speeds with a 32k rsize, nfs outperforms smb by ~280%

The difference was noticeable on XBMC performance as well.  When my network shares were mounted using SMB, I was unable to smoothly stream high bitrate uncompressed 1080p bluray rips (i.e. Avatar).  However, when using NFS, it played absolutely everything I could throw at it.

Netgear XAVB5001 Conclusion:

Bottom line, if your wiring is fairly modern and free of interference, while these adapters fall far short of their advertised 500 Mbps speeds, they should be more than capable of streaming your uncompressed HD content if you can serve your files via NFS.  Even if you are stuck using SMB (Windows network file sharing), most of your content will work without issue, but high bitrate uncompressed bluray will likely not play 100% smoothly.


Followers

Facebook