TechNazgul RSS

Monday, February 14, 2011

How to move your XBMC library while maintaining your watched history and working thumbnails



It’s fairly common that for one reason or another, xbmc users need to move their video library from one location to another.  Perhaps you’ve been using a single 1TB drive for your library, you’ve filled it up and upgraded to a 3TB drive and need to update the path for every movie and TV show to point to the new drive. 

 

Or, in my case, you’ve been using windows samba-based streaming and want to move to the much more efficient NFS streaming protocol (see the comparison detailed in this post).

 

The simple way to change file locations is just to remove your old library, move the content, and then rescan the new destination to recreate your library in the new location.  Unfortunately, this has a few big disadvantages, mainly that you lose each file’s watched status, watched count, and bookmarks, which is really annoying for large libraries.

 

Prerequisites

 

To use the solution below, there are a few things that will help ensure you can follow the steps without modification:

 

  • Your movies and TV should be sorted into individual folders, with thumbnails, trailers, nfo files, etc. all in each individual movie folder. For example, the directory for the movie 300 would include the following files:

 

300 [top-level movie folder]

-> .actors [folder]

-> extrathumbs [folder]

-> 300.bluray.mkv

-> 300.bluray.nfo

-> 300.bluray.tbn

-> 300.bluray-fanart.jpg

-> 300.bluray-trailer.mp4

 

  • If your library is not stored in this structure, try the ‘export library’ function within XBMC (settings –> video –> export library –> separate files) to dump all of these files to the movie directories.
  • You’ll want to download a tool like SQLite (if you are NOT using a shared MySQL library) or Navicat Lite (if you are using MySQL to host a shared library for multiple machines).  These will allow you to browse your database and update the paths where necessary.  I used Navicat, so those of you not using a shared library will have to modify the instructions slightly below to do the same thing in SQLite, which I’m not familiar with.
  • If using MySQL, you’ll need to remember your root password to the DB so that you can connect to it and manipulate the table data.
  • To be safe, you should make a copy of your video DB before starting this as you are mucking around with the sql database and there is no easy way to undo mistakes you make.  In Navicat, this is as easy as, right click on xbmc_video (your video database, and select “dump SQL file.”

 

image

 

Updating the video database with the new path

 

With those items laid out as pre-requisites, here is how it’s done:

 

  • Close xbmc
  • Move / copy your content to the new location
  • Open your database using one of the tools above
  • Open the ‘path’ table within the xbmc_video database (or MyVideos.db for non MySQL).  See the screenshot below for an example using Navicat Lite. [the username/password in my smb path has been blocked out]

 

image

 

  • You now need to open the MySQL console in Navicat to enter in the find/replace command for the path.  The MySQL console can be accessed using the F6 key, or under Tools\Console in the main menu.
  • You’ll use a command like the one below to do the find/replace.  This is a good succinct explanation of the find/replace command in MySQL if you want more info.

mysql> update path set strPath = replace(strPath, ‘old path’, ‘new path’);

 

  • To provide a real example, here is what I used:

mysql> update path set strPath = replace(strPath, ‘smb://username:password@servername/Movies/’, ‘/mnt/movies/’);

 

  • This replaces all SMB paths with nfs mounted paths in my root filesystem. Note that if you have different variations of the old path in your library (some with password, some without, or a separate smb://user:pass@server:/TV/ (/pictures, etc.) you’ll need to issue the above command several times until all old paths are replaced with the new location.  Make sure you have all items with the old path replaced and then move on below.

 

Now that this much is done, you have a library with an updated path pointing to your new file locations. However, a few problems remain to be resolved.  First, we have to update your sources.xml file to point to the new location as well.  This file is what tells XBMC where to look for future movie/TV updates.

 

  • Open your userdata folder and edit the file sources.xml.  Find all instances of your old path (smb://username:password@servername/Movies/ in my example) and replace with your new path.

 

Now, the hardest problem to solve.  How to get XBMC to rehash all of your thumbnail files so that they still work with the new file locations.  The reason this does not work automatically is that XBMC creates cached thumbnails based on the full path to the movie location.  For example, if the original path was smb://username:password@servername/Movies/300, XBMC creates and caches a thumbnail based on a thumbnail hash for that directory in the userdata/Thumbnails directory.

 

If you change that path in the library like we did above from smb:// to /mnt/movies/300, the cached thumbnail will no longer work and you get a big ugly screen capture or no thumbnail at all in place of the thumbnail you want.  The only way to force XBMC to rehash the thumbnail is to remove the item from your library or to manually refresh the information for every file in your library through a series of remote/keyboard clicks, so it’s totally impractical for a large library.

There are a lot of posts on the XBMC forums that involve using scripts to rehash the directory, but I couldn’t get any of these to work.  As I was about ready to give up and just delete my library and start over, I stumbled onto a much easier solution that makes XBMC do all the heavy lifting.

 

Fixing the xbmc thumbnail cache after moving your library

 

  • Start in Navicat, create a new database by right-clicking anywhere in the “connections” frame, call it anything you like… in this case, xbmc_video10 (yes, it took me 10 iterations to figure this out).  Enter the same options as indicated below.

 

 

image

 

  • This is all you need to do for now with Navicat.
  • Edit advancedsettings.xml in your userdata folder to change the database name from the original name to the new name xbmc_video10.

 

<videodatabase>
    <type>mysql</type>
    <host>mysqlserverIP address</host>
    <port>3306</port>
    <user>xbmc</user>
    <pass>xbmc</pass>
    <name>xbmc_video10</name>
</videodatabase>

 

  • Reopen xbmc.  You’ll be starting with a blank video database.
  • Optionally: you can delete all files in the thumbnail directory before starting this rescrape if you don’t want the extra thumbnails taking up space.  I didn’t do this originally as I wasn’t sure it was going to work when I started, but I’m fairly certain you could without issue.
  • Update your xbmc library (right-click somewhere in the library and choose update library).  This will force a rescrape of all of the sources you had previously defined in the new location.  While scraping the new locations, xbmc will properly create the cached thumbnail file based on the new location.  If you have  large library, this will take a fair amount of time.
  • When done, navigate through your TV & Movie libraries and make sure everything is working.  Thumbnails should show up, you should be able to play all files, etc.
  • Close xbmc

 

So at this point we have all of our thumbnails working, but we’ve lost all of our watched history, watched counts, proper ordering for date added the library, etc.  We have only one more step to pair the working thumbnails with complete library history.

 

  • Edit your advanced settings.xml file again.  This time, revert the change you made to the videodatabase name and enter the name of the original database. Save & close.

 

The next time you reopen xbmc, it will access your original library with updated paths, watched file history, and it will find all of the properly cached thumbnails that were created when you had it manually rescrape the library.  Victory!

 

 

Technorati Tags: ,,,

20 comments:

  1. This is genius... I've been thinking about how to do this, as I'll be upgrading my server box with new storage soon. THANKS for figuring this out before I broke something :D

    ReplyDelete
  2. Excellent, cheers! Prefect guide for updating multiple libraries after a server name change.

    ReplyDelete
  3. I was caught out for a little while until I realised that you must use " rather than ' in the mysql replace string. After that it was a breeze.

    ReplyDelete
  4. Interesting that you had to use double quotes. The MySQL manual shows it as single quotes and I think that's what I used as well, though I haven't done it for awhile and can't be sure. In any case, thanks for posting back so that others can try that as well if they run into troubles.

    ReplyDelete
  5. Hi,
    Good read. One thing I'm wondering, I restored my library + thumbnails using a different method, but I didnt have the problem where I had to fix the xbmc thumbnail cache. At least I dont think so, all my thumbs are visible in my new library, or am I missing something?

    Anyway, I went from file location to samba based locations. Exported my library from xbmc to 1 file and cleaned it. Got a text editor and replaced all "c:\my_media..." to "smb://...". Did the symbolic link trick for the thumbs, added the advanced.xml, started XBMC and imported the library again.

    Can it be that changing the file locations within the exported file, instead of mysql, you dont have the cached thumb problem?

    ido

    ReplyDelete
  6. Hey Ido - I'd say consider yourself lucky as I've never heard of it working that way, but I'm glad it did for you. :-)

    ReplyDelete
  7. I moved my TV Shows folder to a different drive. First copied all shows to new directory on new drive. Then exported the XBMC database, in the XML file renamed the path name of the old folder to the new one (via editor find & replace) re- imported XBMC database.Eliminated duplicate entries of the TV Shows in old directory by setting the old folder in Video-Files screen -right-click -> SET CONTENT to ....The process didn't change the movies content that didn't move and preserved the "watched" status for the TV Shows ..hope this helps..
    -

    ReplyDelete
  8. Eliminated duplicate entries of the TV Shows in old directory by setting the old folder in Video-Files screen -right-click -> SET CONTENT to as per link below:
    http://wiki.xbmc.org/index.php?title=Set_Content_%28Video_Library%29

    ReplyDelete
  9. Not sure if mine was a special case, but I had to do one extra set of replaces on the MySQL database: the 'strFilename' column in the 'files' table required updating, to correct the paths of stacked files.

    e.g. stack://smb://Nick9150/Videos_P/Movies/Woodstock (1970) Part1.avi , smb://Nick9150/Videos_P/Movies/Woodstock (1970) Part2.avi

    Perhaps this is because I'm not storing my movies in movie-specific folders.

    ReplyDelete
  10. thanks for this! work perfectly as described, and saved me hours of hassle, thanks!

    ReplyDelete
  11. Great guide! (I am about to run the Thumbnail-poart now)
    Make sure you also change any "" that you might find in advancedsettings.xml to reflect the new servername!

    ReplyDelete
    Replies
    1. "" is supposed to say pathsubstitution..

      Delete
  12. Can I transfer from external on imac to external on router then set up shared library easily?
    i.e. SQLite to transfer first then MySQL to set up the shared library?

    ReplyDelete
  13. I followed these instructions carefully and everything works fine. I am very pleased with the results.

    ReplyDelete
  14. great guide! this was helpful for me...

    ReplyDelete
  15. Great solution for multiple libraries.. congrats!

    ReplyDelete
  16. I like this text very much because it offers many useful information.
    http://www.findacarmadrid.com/

    ReplyDelete
  17. Thank you very much for this excellent guide. I am hosting mySQL on my Synology. I recently moved all my movies to a different location on the NAS hence coming here :) I am at the final stage now - xbmc thumbnail cache. I create a blank Database using Navicat called MyVideos80:

    Default Charter Set = latin 1 (no option for cp 1252)
    Default Collation = latin1_swedish_ci

    I then add MyVideos80 to advancedsettings.xml in the appropriate place. I then deleted thumbnails folder and then opened XBMC. When i go to update library, nothing happens however. The library is blank and and when i hit update library for a split second it says "Compressing database" and thats it. If i revery back to my main database, I can see and play all my movies as normal (minus the thumbnails).

    Do you have any suggestions?

    ReplyDelete
  18. Sorry I can't help Thomas, it's been so long since I first did this that I don't remember the details. If you figure it out, please do post back here as this post gets a lot of search traffic. I'm sure others would benefit if you're able to solve the problem and post the solution.

    ReplyDelete
  19. For those using Frodo (v12) updating Thumbnails/fanart/etc has been made even easier than TechNazgul originally posted.
    XBMC now stores the original location of the artwork in a database table.
    So now we just need to run the same replace method on the art table, url column as we did on the path table, strPath column.
    e.g. UPDATE art SET url = replace(url,'smb://oldserver/','smb://newserver');

    ReplyDelete

Followers

Facebook