TechNazgul RSS

Friday, March 25, 2011

Switching between XBMC and MythTV via remote control



Assuming you’ve tackled the annoying problem of properly auto-starting and resuming irexec (explained in part 1 and part 2), the next thing you might want to do is to be able to switch between XBMC and MythTV on the same machine.  I do this mainly so that I can watch LiveTV on MythTV, and use XBMC for everything else.

 

As it turns out, much of the hard work has been done for us and is available on Google Code, though I’ve modified the code very slightly below to work better in my setup.

 

So, let’s begin:

 

1. Create your appswitch.pl script

I put mine in ~/home/scripts/appswitch.pl.

 

Be sure to replace any curly quotes ‘’ with straight quotes '' when creating your script.  Also check all variables and paths in the “Global vars, paths…” section to ensure they work in your setup.

 

What this script does:

This script checks for what processes are running.  If neither xbmc or mythtv is running, it starts xbmc.  If one or the other is running, it kills the running process and starts the opposite (i.e. kills xbmc and starts myth, or vice-versa).

 

#!/usr/bin/perl -w
##############################################################
## Name: appswitch.pl
##
## Purpose: Executed by irexec this script will cycle between mythtv and xbmc using a single button.
##     
## (C)opyright 2008 Arksoft.
##                                                                                                
## Author: Arkay
##
## Ver 1.0: 4-08-2008.  Initial version.
##

## Modified slightly in March of 2011 by TechNazgul
##############################################################
# Require and Use Clauses.
##############################################################

use strict;                                     #Keeps code neat.
use Getopt::Std;                            #Getopt module for option preprocessing.
use vars qw/ $opt_d $opt_h /;     #Option Processing vars.
use POSIX qw(strftime);               #Time routine we need.

##############################################################
# Prototype definitions
##############################################################

sub logmsg(@);                  #Message logger so we can track what's been going on.
sub process_opts();         #Option processing.. Nothing exiting for this script.
sub do_command($);      #Execute a shell command for lazy perl programmers :)
sub check_proc($);          #Check if a process is running.
sub startproc($);

##############################################################
# Constant Definitions.
##############################################################

my ($TRUE) = 1;
my ($FALSE) = 0;

##############################################################
# Global vars, paths, commands to call.
##############################################################

my ($LOG) = "/home/xbmc/scripts/appswitch.log";         #Log location.
my ($LOGSIZE) = 1024;                                               #Maximum log size in kbytes, self pruning.
my ($DEBUG) = $FALSE;                                               #Debugging default is off.
my ($BASENAME) = $0;                                                #How was the program called?

my ($MYTHTV)="/usr/bin/mythfrontend";                      #Process name to start
my ($XBMC)="/usr/bin/xbmc";                                    #Process name to start
my ($KMYTHTV)="mythfrontend.real";                         #Process name to use with killall
my ($KXBMC)="xbmc.bin";                                             #Process name to use with killall
my ($KMPLAYER)="mplayer";                                       #Also kill any external players
my ($KILLALL)="/usr/bin/killall";                                   #kill command.
my ($AUDIOHACK)="/usr/bin/iecset audio on";             #Shitty Alsa hack with intel HDA

##############################################################
# The Mainline.
##############################################################
MAIN:
{
        my ($command);
        process_opts();

        logmsg "$BASENAME started : PID($$)";

        my ($myth)=check_proc($MYTHTV);
        my ($xbmc)=check_proc($XBMC);

        SWITCH:
        {
                if (($myth == $TRUE ) && ($xbmc == $TRUE))
                {
                        logmsg "Killing $XBMC and $MYTHTV";
                        $command="$KILLALL $KMYTHTV"; do_command($command);
                        $command="$KILLALL $KMPLAYER"; do_command($command);
                        $command="$KILLALL $KXBMC"; do_command($command);
                        sleep(2);

                        $myth=check_proc($MYTHTV);
                        if ($myth == $TRUE)
                        {
                                logmsg "Killing $MYTHTV (forced)";
                                $command="$KILLALL -9 $KMYTHTV"; do_command($command);
                                $command="$KILLALL -9 $KMPLAYER"; do_command($command);
                        }

                        $xbmc=check_proc($XBMC);
                        if ($xbmc == $TRUE)
                        {
                                logmsg "Killing $XBMC (forced)";
                                $command="$KILLALL -9 $KXBMC"; do_command($command);
                        }
                        do_command($AUDIOHACK);
                        startproc("export XBMC_PLATFORM_MODE=1 ; $XBMC");
                        last SWITCH;
                }
                               
                if ($myth == $TRUE)
                {
                        logmsg "Killing $MYTHTV";
                        $command="$KILLALL $KMYTHTV"; do_command($command);
                        $command="$KILLALL $KMPLAYER"; do_command($command);
                        sleep(2);
                        $myth=check_proc($MYTHTV);
                        if ($myth == $TRUE)
                        {
                                logmsg "Killing $MYTHTV (forced)";
                                $command="$KILLALL -9 $KMYTHTV"; do_command($command);
                                $command="$KILLALL -9 $KMPLAYER"; do_command($command);
                        }
                        do_command($AUDIOHACK);
                        startproc("export XBMC_PLATFORM_MODE=1 ; $XBMC");
                        last SWITCH;
                }

                if ($xbmc)
                {
                        logmsg "Killing $XBMC";
                        $command="$KILLALL $KXBMC"; do_command($command);
                        sleep(2);
                        $xbmc=check_proc($XBMC);
                        if ($xbmc == $TRUE)
                        {
                                logmsg "Killing $XBMC (forced)";
                                $command="$KILLALL -9 $KXBMC"; do_command($command);
                        }
                        do_command($AUDIOHACK);
                        startproc($MYTHTV);
                        last SWITCH;
                }
               do_command($AUDIOHACK);
               startproc("export XBMC_PLATFORM_MODE=1 ; $XBMC");
        }

        logmsg "$BASENAME Completed.";
}

##############################################################
# startproc()
# Execute a given command.
##############################################################
sub startproc($)
{
        my ($proc)=@_;
        logmsg "Starting $proc";
        exec("export DISPLAY=:0; $proc &");
}

##############################################################
# check_proc($)
# Check if processes are running that should stop shutdown from occuring.
##############################################################
sub check_proc($)
{
        my ($proc)=@_;
        my (@output);
        my ($command);
        my ($count)=0;
        my ($running)=$FALSE;

        logmsg "Checking for active process $proc.";

        $command="ps -ef | grep \"$proc\" | grep -v grep |wc -l";      
        @output=do_command($command);

        if (@output)
        {
                $count=$output[0];
                chomp ($count);
        }

        if ($count > 0)
        {
                logmsg "Found active process : $proc ($count running).";
                $running=$TRUE;
        }
        return($running);
}

##############################################################
# logmsg
# Little routine to write to the log file.
# Rotates around $LOGSIZE bytes.
##############################################################
sub logmsg(@)
{
        my ($string)=@_;
        my $time=scalar localtime;
        my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks);
        my (@lines,$line);

        ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks)=stat("$LOG");

        if (defined($size))
        {
                $size=$size/1024;                               #size in kbyte

                if ($size >= $LOGSIZE)
                {
                        unlink ("$LOG.old") if (-e("$LOG.old"));
                        rename ($LOG,"$LOG.old");
                }
        }

        print "$time : $string\n" if ($DEBUG==$TRUE);

        if (open (LOG,">>$LOG"))
        {
                if ($string =~ /\n/)
                {
                        @lines=split(/\n/,$string);
                        foreach $line (@lines)
                        {
                                print LOG "$time : $line\n";
                        }
                }
                else
                {
                                print LOG "$time : $string\n";
                }
                close (LOG);
        }
        else
        {
                print "Unable to open LOG $LOG : $!";
        }
}

##############################################################
# process_opts()
# Set Global option flags dependant on command line input.
##############################################################
sub process_opts()
{
        getopts('dh');

        $DEBUG=$TRUE if ($opt_d);      
        exit(usage(1)) if ($opt_h);
}

##############################################################
# usage()
# Output Relevant Usage strings if incorrect opts are given.
##############################################################
sub usage()
{
        my($ucode)=@_;

        if ($ucode == 1)
        {
                print "Usage: $BASENAME [-dh]\n";
                return(0);
        }
}

##############################################################
# sub do_command($)
# use system call to execute command. Returns output of command in array.
##############################################################
sub do_command($)
{
    my ($command)=@_;
    my (@output);
    my ($exit_value)=0;

    logmsg "Executing $command" if ($DEBUG == $TRUE);

    @output=`$command`;

    $exit_value = $? >> 8;

    if ($exit_value != 0)
    {
        logmsg "Error executing $command : $!";
    }
    return(@output);
}

 

 

2. Configure irexec to call the script via remote button press

In your home directory, you should already have a .lircrc file.  You need to add to that the following code, either directly in that file, or by placing it in your ~/.lirc directory and calling it via ‘include’ in your .lircrc file.  In my case, I’m doing it via an inclusion.

 

In ~/.lircrc, add the following:

 

include ~/.lirc/irexec

 

Then, create ~/.lirc/irexec as (replace /home/xbmc with /home/USERNAME)

 

begin
     prog = irexec
     button = LiveTV
     config = /home/xbmc/scripts/appswitch.pl &
end

 

I used the remote button LiveTV as it wasn’t used for anything else by either XBMC or MythTV by default.

 

3. Restart lirc to load the new configuration

sudo /etc/init.d/lirc restart

 

4. Ensure irexec is running and test

 

Check for irexec by typing “ps –ef|grep irexec”  If it isn’t, fire it up manually with:

irexec –d

Then, press the LiveTV button on your remote and wait… If it doesn’t work as planned, kill all running irexec processes and restart irexec without the –d flag.  This will keep it running in the foreground and allow you to watch the script output as it progresses through it. If you see nothing, something is wrong with your irexec script.  If you see script output in your terminal window, then the irexec is being successfully called via remote. You can also experiment with different options of calling irexec as root or as your user, which sometimes yields differing results with the app restart script.

 

Technorati Tags: ,,

Monday, February 14, 2011

How to move your XBMC library while maintaining your watched history and working thumbnails



It’s fairly common that for one reason or another, xbmc users need to move their video library from one location to another.  Perhaps you’ve been using a single 1TB drive for your library, you’ve filled it up and upgraded to a 3TB drive and need to update the path for every movie and TV show to point to the new drive. 

 

Or, in my case, you’ve been using windows samba-based streaming and want to move to the much more efficient NFS streaming protocol (see the comparison detailed in this post).

 

The simple way to change file locations is just to remove your old library, move the content, and then rescan the new destination to recreate your library in the new location.  Unfortunately, this has a few big disadvantages, mainly that you lose each file’s watched status, watched count, and bookmarks, which is really annoying for large libraries.

 

Prerequisites

 

To use the solution below, there are a few things that will help ensure you can follow the steps without modification:

 

  • Your movies and TV should be sorted into individual folders, with thumbnails, trailers, nfo files, etc. all in each individual movie folder. For example, the directory for the movie 300 would include the following files:

 

300 [top-level movie folder]

-> .actors [folder]

-> extrathumbs [folder]

-> 300.bluray.mkv

-> 300.bluray.nfo

-> 300.bluray.tbn

-> 300.bluray-fanart.jpg

-> 300.bluray-trailer.mp4

 

  • If your library is not stored in this structure, try the ‘export library’ function within XBMC (settings –> video –> export library –> separate files) to dump all of these files to the movie directories.
  • You’ll want to download a tool like SQLite (if you are NOT using a shared MySQL library) or Navicat Lite (if you are using MySQL to host a shared library for multiple machines).  These will allow you to browse your database and update the paths where necessary.  I used Navicat, so those of you not using a shared library will have to modify the instructions slightly below to do the same thing in SQLite, which I’m not familiar with.
  • If using MySQL, you’ll need to remember your root password to the DB so that you can connect to it and manipulate the table data.
  • To be safe, you should make a copy of your video DB before starting this as you are mucking around with the sql database and there is no easy way to undo mistakes you make.  In Navicat, this is as easy as, right click on xbmc_video (your video database, and select “dump SQL file.”

 

image

 

Updating the video database with the new path

 

With those items laid out as pre-requisites, here is how it’s done:

 

  • Close xbmc
  • Move / copy your content to the new location
  • Open your database using one of the tools above
  • Open the ‘path’ table within the xbmc_video database (or MyVideos.db for non MySQL).  See the screenshot below for an example using Navicat Lite. [the username/password in my smb path has been blocked out]

 

image

 

  • You now need to open the MySQL console in Navicat to enter in the find/replace command for the path.  The MySQL console can be accessed using the F6 key, or under Tools\Console in the main menu.
  • You’ll use a command like the one below to do the find/replace.  This is a good succinct explanation of the find/replace command in MySQL if you want more info.

mysql> update path set strPath = replace(strPath, ‘old path’, ‘new path’);

 

  • To provide a real example, here is what I used:

mysql> update path set strPath = replace(strPath, ‘smb://username:password@servername/Movies/’, ‘/mnt/movies/’);

 

  • This replaces all SMB paths with nfs mounted paths in my root filesystem. Note that if you have different variations of the old path in your library (some with password, some without, or a separate smb://user:pass@server:/TV/ (/pictures, etc.) you’ll need to issue the above command several times until all old paths are replaced with the new location.  Make sure you have all items with the old path replaced and then move on below.

 

Now that this much is done, you have a library with an updated path pointing to your new file locations. However, a few problems remain to be resolved.  First, we have to update your sources.xml file to point to the new location as well.  This file is what tells XBMC where to look for future movie/TV updates.

 

  • Open your userdata folder and edit the file sources.xml.  Find all instances of your old path (smb://username:password@servername/Movies/ in my example) and replace with your new path.

 

Now, the hardest problem to solve.  How to get XBMC to rehash all of your thumbnail files so that they still work with the new file locations.  The reason this does not work automatically is that XBMC creates cached thumbnails based on the full path to the movie location.  For example, if the original path was smb://username:password@servername/Movies/300, XBMC creates and caches a thumbnail based on a thumbnail hash for that directory in the userdata/Thumbnails directory.

 

If you change that path in the library like we did above from smb:// to /mnt/movies/300, the cached thumbnail will no longer work and you get a big ugly screen capture or no thumbnail at all in place of the thumbnail you want.  The only way to force XBMC to rehash the thumbnail is to remove the item from your library or to manually refresh the information for every file in your library through a series of remote/keyboard clicks, so it’s totally impractical for a large library.

There are a lot of posts on the XBMC forums that involve using scripts to rehash the directory, but I couldn’t get any of these to work.  As I was about ready to give up and just delete my library and start over, I stumbled onto a much easier solution that makes XBMC do all the heavy lifting.

 

Fixing the xbmc thumbnail cache after moving your library

 

  • Start in Navicat, create a new database by right-clicking anywhere in the “connections” frame, call it anything you like… in this case, xbmc_video10 (yes, it took me 10 iterations to figure this out).  Enter the same options as indicated below.

 

 

image

 

  • This is all you need to do for now with Navicat.
  • Edit advancedsettings.xml in your userdata folder to change the database name from the original name to the new name xbmc_video10.

 

<videodatabase>
    <type>mysql</type>
    <host>mysqlserverIP address</host>
    <port>3306</port>
    <user>xbmc</user>
    <pass>xbmc</pass>
    <name>xbmc_video10</name>
</videodatabase>

 

  • Reopen xbmc.  You’ll be starting with a blank video database.
  • Optionally: you can delete all files in the thumbnail directory before starting this rescrape if you don’t want the extra thumbnails taking up space.  I didn’t do this originally as I wasn’t sure it was going to work when I started, but I’m fairly certain you could without issue.
  • Update your xbmc library (right-click somewhere in the library and choose update library).  This will force a rescrape of all of the sources you had previously defined in the new location.  While scraping the new locations, xbmc will properly create the cached thumbnail file based on the new location.  If you have  large library, this will take a fair amount of time.
  • When done, navigate through your TV & Movie libraries and make sure everything is working.  Thumbnails should show up, you should be able to play all files, etc.
  • Close xbmc

 

So at this point we have all of our thumbnails working, but we’ve lost all of our watched history, watched counts, proper ordering for date added the library, etc.  We have only one more step to pair the working thumbnails with complete library history.

 

  • Edit your advanced settings.xml file again.  This time, revert the change you made to the videodatabase name and enter the name of the original database. Save & close.

 

The next time you reopen xbmc, it will access your original library with updated paths, watched file history, and it will find all of the properly cached thumbnails that were created when you had it manually rescrape the library.  Victory!

 

 

Technorati Tags: ,,,

Tuesday, February 8, 2011

NFS vs. SMB throughput for HD media streaming using the NETGEAR XAVB5001 500 Mbps Powerline Adapter



NETGEAR XAVB5001 500 Mbps Powerline Adapter Review:


XAVB5001After recently dropping my satellite subscription and moving to MythTV, I needed to find a way to get HD video streams from my antenna and HD Homerun tuner to a TV in the house without wired internet.  Wireless (G or N) didn’t prove to be reliable or fast enough, so the Netgear 500 mbps (theoretical) powerline adapters seemed like a good bet.

Testing was performed in a ~3-year old house and the 2 adapters were several rooms apart from one another on the same floor.  With these adapters, performance can typically vary significantly based on wiring within the house, so YMMV.

As a preview of the results, while they didn’t achieve anywhere near the advertised 250 mbps (~31 MB/s) in each direction, they did perform much better than older generation powerline adapters I’ve used.  When I plugged in the adapters, the Netgear software reported that I was getting ~220 Mbps in each direction, which was encouraging, but ultimately not even close to accurate.

In addition to testing the adapters, I also wanted to test the efficiencies of different network protocols to see what would work best for streaming HD content to my XBMC / MythTV box.  Below are the results.

Testing was done using commands like the following on a Ubuntu Linux machine that tests reads and write speeds to network-mounted disk. 

Write: time dd if=/dev/zero of=/mnt/storagesmb/testfilesmb1 bs=16k count=16384
Read:  time dd if=/mnt/storagesmb/testfilesmb1 of=/dev/null bs=16k

Since the focus of my testing was primarily to verify read speeds from a media frontend to a backend storage server, I did not do significant write testing.

Disk Read/Write Test Results:

Protocol Test Type File Size Rsize (nfs) Throughput
SMB Write 256MB na 03.8 MB/s
SMB Read 256MB na 03.9 MB/s
NFS v3 Write 256MB na 05.4 MB/s
NFS v3 Read 256MB 16,384 09.3 MB/s
NFS v3 Read 256MB 32,768 10.8 MB/s

Next, a separate set of tests was done using iperf between two Ubuntu Linux machines.  The server resides on a large RAID array capable of reads and writes upwards of 500 MB/s, so disk speeds were not a limitation in this testing. 

In theory, the iperf test should remove most inefficiencies related to network protocol and show raw throughput. As it turns out, the results are very similar to the NFS testing above, which speaks well of the NFS protocol.

iperf results:

***ONE DIRECTIONAL TEST
------------------------------------------------------------
Client connecting to 192.168.1.111, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.182 port 42245 connected with 192.168.1.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  90.8 MBytes  75.9 Mbits/sec (9.49 MB/s)


***BIDIRECTIONAL (SIMULTAENOUS) TEST
------------------------------------------------------------
Client connecting to 192.168.1.111, TCP port 5001
TCP window size: 55.0 KByte (default)
------------------------------------------------------------
[  5] local 192.168.1.182 port 60753 connected with 192.168.1.111 port 5001
[  4] local 192.168.1.182 port 5001 connected with 192.168.1.111 port 59780
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  57.4 MBytes  48.0 Mbits/sec (6.00 MB/s)
[  4]  0.0-10.1 sec  43.0 MBytes  35.7 Mbits/sec (4.46 MB/s)

Combined => ~83.7 Mbits/sec (10.46 MB/s)



SMB vs. NFS Conclusion:
While this is probably no surprise to many, NFS is vastly more efficient.  Using the same commands and testing read speeds with a 32k rsize, nfs outperforms smb by ~280%

The difference was noticeable on XBMC performance as well.  When my network shares were mounted using SMB, I was unable to smoothly stream high bitrate uncompressed 1080p bluray rips (i.e. Avatar).  However, when using NFS, it played absolutely everything I could throw at it.

Netgear XAVB5001 Conclusion:

Bottom line, if your wiring is fairly modern and free of interference, while these adapters fall far short of their advertised 500 Mbps speeds, they should be more than capable of streaming your uncompressed HD content if you can serve your files via NFS.  Even if you are stuck using SMB (Windows network file sharing), most of your content will work without issue, but high bitrate uncompressed bluray will likely not play 100% smoothly.


Friday, January 28, 2011

Customized XBMC Transparency Skin for MythBox



I’ve been a long-time XBMC user and recently started using MythTV as well after dumping my satellite subscription.  The MythBox integration with XBMC is awesome, but one thing I felt was lacking was an XBMC skin with an easy front-page option to click directly into MythBox.

 

Most XBMC skins offer the ability to customize submenus one or more levels deep in their menu system, but if you’re trying to convince your wife or kids that MythBox is the best thing since sliced bread, you don’t want to frustrate them by asking them to remember a complicated set of button clicks to get into live TV (MythBox).  The way around that is to customize a skin to allow the MythBox script to be called from the home screen. (note that the Transparency skin does the option to add custom buttons to the front screen, but only if they call up a favorites.xml file, it does not allow a script to be called from the custom button, at least not currently)

 

Since a picture is worth a thousand words, below is what the customized Transparency skin looks like.  As it turns out, making this work only requires changes to one file and the addition of the mythtv background file. 

 

Download the required files here if you’d like to make these changes yourself.

 

You’ll need to unzip the files and copy them into your xbmc userdata folder.  Navigate to: userdata –> addons –> skin.transparency and then mimic the file structure in the zip file.  The file Includes_home.xml goes into the 720p folder and the mythtv.jpg file goes into the backgrounds folder. 

 

To enable the view from within Transparency, you need to have the following enabled: 

Settings –> Skin Settings –> Menu –> TV

 

“TV” in this case has been replaced by MythTV and will be called “Live TV” in the Menu.  If you want to name that something different, navigate to line 2037 in the Includes_Home.xml file and change “LIVE TV” to whatever you want it to be.

 

Enjoy!

 

xbmc-mythtv-mythbox-custom-transparency 

Technorati Tags: ,,

HDTV OTA Antennas: Terk HDTVa vs. Antennas Direct DB2



I’ve recently decided to cancel our $85 / month DirecTV subscription and will instead be utilizing a combination of MythTV (open-source DVR software ) for managing over the air (OTA) broadcasts and online streaming (Netflix, Hulu, etc.) as a replacement f or movies/TV series.  (which will undoubtedly be a topic in future posts).

 

During the process of setting this up, I had to choose an antenna for my attic to allow me to receive the OTA broadcasts in our area.  I purchased the two top-selling HDTV antennas from Amazon and compared the two. 

 

When dealing with antennas, I realize that everyone's situation is unique, but below is my experience.  I live in Austin, TX about 12 miles north of most of the main towers. The antenna is in my 2nd story attic. 

 

Most of the stations in my area broadcast on UHF, but one continues broadcasting in VHF so I initially thought I needed an antenna that did both UHF/VHF. For that reason, I picked up this Terk HDTVa (which does both UHF/VHF) first and was disappointed with its results, both on UHF & VHF. The signal was decent, but definitely not flawless.

 

In addition, I found two things about this design sub-par. First, with the VHF antennae extended, the unit takes up a lot of space, even in an attic. It's also very unstable when fully extended. Secondly, the stated need for it to be plugged in regardless of whether you are using the amplification or not makes for a slightly inconvenient installation. Finally, I noticed that the power adapter when plugged in was exceedingly hot, so hot that it was uncomfortable to hold in my hand. Long-term, that just does not seem safe to me.


To make a long story short, I tried the antenna for a day and then ordered the Antennas Direct DB2 Multi Directional HDTV Antenna to replace it. It immediately improved my reception on UHF and surprisingly, on the one VHF station as well. The antenna was placed in the exact same spot as the HDTVa.  Below are the signal comparison results for each antenna.


hdtva

Terk HDTVa antenna
Channel -> Signal Strength / Signal Quality
54.1 -> 61/83
42.1 -> 65/91
24.1 -> 80/100
18.1 -> 72/94
36.1 -> 78/95
7.1 -> 54/75 [VHF]

 

 

 

 

 

 

 


db2Antennas Direct DB2 antenna
Channel -> Signal Strength / Signal Quality
54.1 -> 91/100
42.1 -> 84/100
24.1 -> 94/100
18.1 -> 95/90
36.1 -> 96/95
7.1 -> 91/100 [VHF]


Conclusion: The DB2 was 15-30% better on every channel and also does not require power for amplification like the HDTVa does. A hands-down winner in my situation.

 

 

Configuring DD-WRT VPN client with HideIP VPN



HideIP VPN is a simple, fast VPN provider that I’ve used in the past to help navigate around poor BGP routing issues that were slowing down my backups to CrashPlan’s online backup service.  Specifically, the goal was to avoid any and all routing through Cogentco’s very slow route to CrashPlan Central (which increased my uploads by 350%).  I’m optimistic that at some point CrashPlan will fix this problem by talking to Cogentco about how it’s affecting their service, but despite it being reported several months ago by many users on their forums, their only response is that they don’t believe it’s their problem (bummer).

 

However, configuring the VPN on each machine in my network was a pain I didn’t want to go through, so instead I set up a VPN on my DD-WRT router (Asus RT-N16) to direct all CrashPlan Central traffic through the VPN (which is faster than the direct connection BGP chooses for me to CrashPlan’s central servers).

 

This is all done in DD-WRT version v24-sp2 (mega build).  In the DD-WRT GUI, choose Services –> VPN – PPTP Client –> Enable.

 

That brings you to this interface:

 

ddwrt-pptp-vpn

 

Now, let’s explain each of the options above.

 

Server IP or DNS Name: This is the address of the HideIP VPN server you wish to use.  At the time I was doing this, here is how the IP addresses map to locations.  The lines in bold are VPNs that actually do avoid Cogentco’s routing to central.crashplan.com, so if you’re trying to accomplish the same thing, I’d recommend you use one of those.

 

us1.hideipvpn.com resolves to: 74.55.175.26 - Houston, TX
us2.hideipvpn.com resolves to: 204.152.215.132 - Los Angeles, CA
us3.hideipvpn.com resolves to: 174.37.16.81 - Dallas, TX
us4.hideipvpn.com resolves to: 173.208.45.178 - NYC
us5.hideipvpn.com resolves to: 173.208.68.58 - Dallas, TX
us6.hideipvpn.com resolves to: 67.215.231.90 - Santa Ana, CA
us7.hideipvpn.com resolves to: 173.234.32.26 - Chicago, IL (the one I’m using above)
us8.hideipvpn.com resolves to: 70.32.45.178 – NYC

 

Remote Subnet:  This is the range of addresses belonging to Central.CrashPlan.com.  I chose a wide swath of addresses that includes the complete last octet of the IP address. 173.225.132.x, which is what the combination of remote subnet and subnet mask accomplishes above.  Use those same values if you are backing up to central.crashplan.com.  Realistically, this will cause some non-crashplan traffic to go over this VPN as well, but I don’t consider that a big problem.

 

MPPE Encryption: These options are necessary to connect to HideIPVPN’s service:

mppe required,no40,no56,stateless

 

MTU/MRU/NAT: leave as default.

Username/Password” as provided by HideIPVPN

 

Once you’ve configured these options click Save, then Apply Settings.   Then navigate to “Administration –> Reboot Router.”  The router will connect to the VPN upon reboot. Make sure to give it 2-3 minutes to fully reboot and connect, after which you can test success by opening a terminal window and doing a traceroute to central.crashplan.com.  In Windows, the output looks like this.  The key that you are looking for is for the 2nd line to read something like below, an internal IP address on HideIPVPN’s network.  After that, just check the route to make sure that it does not include CogentCo and you’re set.

 

traceroute

 

In my case, going through CogentCo maxes me out at a ~500 Kbps to central.crashplan.com.  When I establish the VPN through HideIP-VPN-NYC, I achieve ~1.8 Mbps (on a 3 Mbps uplink connection), a 350% improvement.

 

Friday, January 21, 2011

CrashPlan Online Backup: Maximizing Upload Speeds



Note that the problems related to dedupe/compression observed in this post were fixed by CrashPlan in the 3.03 software release in March of 2011.  As of March, 2011, the problems with CogentCo remain problematic (limits of 300-500 Kbps through CogentCo vs. 3 Mbps + through any other route).

 

The recent comments in a forum thread at CrashPlan re: compression / deduplication and CPU usage prompted me to run a series of tests with different configuration settings to see how/if it impacted upload speeds to CrashPlan Central (their online backup archive).  Below are the details on the tests I performed as well as the results.

 

Test file:  3GB ISO file of a DVD
Archive location: central5.crashplan.com

Online Archive Size: ~100k files, 2.4TB
Server OS: Windows Server 2008 R2
RAM: 12 GB
CPU: i7-860, 2.8GBz Quad Core with Hyperthreading
System Drive: 64GB SSD
Source File Drives: 10x2TB RAID 6
[Given this configuration, if this server ends up being resource constrained, you can imagine that almost any other machine would be as well given a similarly large backup set]

 

Between each change of the settings below, I restarted the CrashPlan service using the restart command in the GUI.  Speed results below are taken from observing Windows network monitoring for several minutes of sustained performance, not from the CPU GUI (which sometimes reports different throughput based on compression & de-dupe).  During testing, I was monitoring the WAN interface on my router during all tests to ensure there was no significant consumption of my bandwidth from anything else on my network.

 

Lastly, this Windows install is brand new as of yesterday with only a handful of applications installed other than CrashPlan, so interference from other software should be minimal.

 

With all that said... the results are:

 

Test 1: (Control case)
Data de-dupe: Minimal
Compression: Off
Results:  2.1 Mbps upload
CPU usage: ~4% average across 8 threads

 

Test 2: (test 1 with compression to ON)
Data de-dupe: Minimal
Compression: *On*
Results:   1.95 Mbps upload (essentially the same as test 1)
CPU usage: ~4% average across 8 threads

 

Test 3: (test 1 with compression to auto)
Data de-dupe: Minimal
Compression: *Auto*
Results:    2.1 Mbps Upload
CPU usage: ~5% average across 8 threads

 

Test 4: (test 1 with de-dupe to Full)
Data de-dupe: *Full*
Compression: Off
**Results:   530 Kbps Upload**
CPU usage: ~15% average across 8 threads (or effectively 100% of one thread)

 

Test 5: (test 1 with de-dupe to Auto)
Data de-dupe: *Auto*
Compression: Off
***Results:    550 Kbps Upload***
CPU usage: ~15% average across 8 threads (or effectively 100% of one thread)

 

Conclusion: De-dupe set to anything but minimal is an absolute throughput killer when backing up large compressed files and/or when backing up from a large backup set.  This could be due to the size of my backup set (100k files and 2.4 TB, which is a lot to de-dupe), but it appeared to be limited by the ability of the de-dupe process to use only 1 CPU (other than this, I observed low network usage, low overall CPU usage, low memory usage, no disk queues, etc.). 

 

Also, this was a head-slap moment for me as I've been struggling to figure out why my throughput to CrashPlan was so poor for some time now and was actually reimaging this server just for the purpose of starting fresh with CrashPlan and getting my speed issues fixed once and for all.  Duh...

 

Takeaway for me:  I'm turning compression to Auto and De-dupe to Minimal.  I can't tell you how happy I am to have finally figured out what was causing the performance bottleneck on this backup set...  This might not work for everyone, but if you're in  similar scenario to mine, I'd encourage you to try these settings.

 

I’ll close by saying I love CrashPlan.  It is the best backup software I’ve found out there and I’ve tried several.  Now that I have this figured out, it’s even better.  :-)  Hopefully this knowledge helps others out as well.

 

Finally, I’ll close by mentioning one other well-known issue affecting upload speeds to CrashPlan Central.  It involves routing through the cogentco network and is discussed here and here (and probably elsewhere) on CrashPlan’s forum.  I wrote up a related post discussing how this bottleneck can be avoided through use of a VPN to avoid the CogentCo network.

 

Advanced network configuration using vmware Player



Vmware-netcfgIn the past, the advanced network configuration utility for vmware was installed automatically with vmware player, but at some point they started making it more difficult to find.  As it turns out, vmnetcfg.exe is still included with the installer, but you have to manually extract and move it to your install directory.

The steps to extract it are fairly simple.  Download the standard vmware player installer and then do the following:

 

  • Open a command prompt in the same directory as the vmware installer.
  • Issue the command, VMware-player-3.<version>.exe /e .\netcfg
  • Navigate to the netcfg folder in the same directory, which now contains the extracted installer files
  • Open ‘network.cab’ with Winrar (or something similar) and you will see the vmnetcfg.exe file.
  • Copy that file to your install directory… Something like C:\Program Files\VMWare\VMWare Player\
  • Once copied, you can double-click the utility from that folder and begin the advanced configuration of your vmware network adapters.

 

If you’ve not used this utility before, it allows you to specify which virtual network adapters fulfill which function, define which subnets you want each virtual adapter to use, etc.  For example, if you have two network connections on your computer and want to bind different vmware network adapters to different physical network connections (and related VLANs), you can do so using this utility.

 

Thanks to The Advisers Group for the original tip.

 

Technorati Tags: ,,

Thursday, January 20, 2011

Enable Windows Snipping Tool in Server 2008 R2




64px-Snipping_Tool_Vista_IconStrangely, attempts to Google this solution led to many convoluted results that involved copying files from a Vista install to the Server 2008 directory or hacking the registry to make this work. As it turns out, in Server 2008 R2, it is simple as:

 

Server Manager > Features > Add Features > Desktop Experience.

 

This does include some other ‘junk’ that you might not want on your server like WMP, Desktop Themes, etc., but it will get you the Windows Snipping Tool as well.

 

Technorati Tags:

Followers

Facebook