Secure Subversion on the XAMPP Apache Server

Windows Development 2 Comments

To secure subversion, we just need to create a password file and configure apache to use it for subversion repository access.

  1. Open up the apache config file “C:\xampp\apache\conf\httpd.conf”. At the end, look for the subversion repository URL location and add the additional text below, starting with “AuthType”:
    <Location /repos>
        DAV svn
        SVNPath c:/svn_repos

        AuthType Basic
        AuthName "Subversion Repository"
        AuthUserFile conf/svn-password.pass
        Require valid-user
    </Location>
  2. Create the “svn-password.pass” file by launching the “Start->All Programs->Accessories->command Prompt” and running these commands:
    c:\xampp\apache\bin\htpasswd.exe -cm c:\xampp\apache\conf\svn-password.pass username1
    c:\xampp\apache\bin\htpasswd.exe -m c:\xampp\apache\conf\svn-password.pass username2
    c:\xampp\apache\bin\htpasswd.exe -m c:\xampp\apache\conf\svn-password.pass username3

    You will be prompted to input the password for each user. I’m suggesting that the full path to htpasswd.exe be used because other programs (such as PUTTY) may have incompatible versions.

  3. Restart the Apache server.
  4. Browse to http://localhost/repos/ and you will be asked to input a username and password.

Additionally, we can create an authorization file to assign permissions to users or groups of users.

  1. Open up the apache config file “C:\xampp\apache\conf\httpd.conf”. At the end, look for the subversion repository URL location and add the additional line called “AuthzSVNAccessFile”:
    <Location /repos>
        DAV svn
        SVNPath c:/svn_repos
       
        AuthType Basic
        AuthName "Subversion Repository"
        AuthUserFile conf/svn-password.pass
        AuthzSVNAccessFile conf/svn-authz.conf
        Require valid-user
    </Location>
  2. Create the “c:\xampp\apache\conf\svn-authz.pass” file. Here is some example content:
    [groups]
    devteam = username1,username2
    qateam = username3,username4,username5

    [/]
    @devteam = rw
    username6 = r

    [/project1]
    @devteam = rw
    @qateam = r
    username7 = rw
  3. Restart the Apache server.
  4. Browse to http://localhost/repos/ and you will be prompted to input a username and password.
  5. When using the subversion command line, you can use the –username flag to pass in your username the first time:
    svn --username username1 co http://localhost/repos/myproject

The info above was derived from How to Setup Subversion + Apache + WebSVN.

2 Comments

Using Eclipse to Debug PHP

Windows Development 1 Comment

With EasyEclipse and the PHP Debugger extension, you can debug and step through a PHP call.

Installing the PHP Debugger (DBG) Extension

  1. Download the PHP Debugger (DBG); specifically the DBG 2.15.5 dbg modules for windows.
  2. Unzip the dbg modules to a temporary directory.
  3. Copy the relevant php extension file, currently “x86\php_dbg.dll-5.2.x” in the archive (since XAMPP 1.6 comes with PHP 5.2.5), to the extension directory “c:\xampp\php\ext” and rename it to “php_dbg.dll”.
  4. Open up the “c:\xampp\apache\bin\php.ini” config file.
    • Add this to the end of php.ini:
      [debugger]
      extension=php_dbg.dll
      debugger.enabled=on
      debugger.profiler_enabled=on
      debugger.hosts_allow=localhost
      debugger.hosts_deny=ALL
      debugger.ports=7869, 10000/16
    • Search for the first instance of “zend_extension_ts” in php.ini (it should be right underneath the “[Zend]” section) and before it, add this line:
      [Zend]
      zend_extension_ts = "c:\xampp\php\ext\php_dbg.dll"
      ...
  5. Verify that the DBG Extension is loaded:
    • Stop and start the apache http server (a reload or restart won’t work).
    • Create a phpinfo.php file with this content:
      <html>
      <head>
      <title>PHP Test Script</title>
      </head>
      <body>
      <?php
      phpinfo();
      ?>
      </body>
      </html>
    • Install phpinfo.php under the apache root document directory and then browse to it. You will see the info on the PHP environment. If DBG is installed correctly, you should see this line:
      Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
          with DBG v2.15.5, (C) 2000,2007, by Dmitri Dmitrienko
    • Search on “dbg” and you will see the configuration section for dbg also.

Disabling PHP Debugging

  • Once you have installed the DBG extension, debugging will be enabled and for any PHP file that is browsed to, the PHP engine will attempt to debug and fail because we have not setup a PHP Debugger service yet.
  • To disable debugging permanently:
    1. Open up the “c:\xampp\apache\bin\php.ini” config file.
    2. Change the “debugger.enabled=on” to “debugger.enabled=off”.
    3. Restart the Apache HTTP server.
  • To disable debugging for the current active HTTP session:
    1. Browse to “http://localhost/myproject/index.php?DBGSESSID=-1”.
    2. This URL will remove the cookie added by the DBG extension to enable PHP debugging.

Starting the PHP Debugger Service

In order for PHP debugging to work, we must configure and start a PHP Debugger service using EasyEclipse:

  1. Run EasyEclipse and go to menu “Run->Debug…”. If you don’t see the Run->Debug menu, then open up the Debug perspective, switch back to PHP perspective, and try again.
  2. Select and highlight the “PHP DBG Script” and click on the New icon on the upper-left.
  3. Input whatever Name you want.
  4. Under the File tab, input “index.php” (or your main entrypoint PHP file). (Alternatively, you can Browse… and select it.)
  5. Under the Arguments tab, you should see “Use default working directory” checked and the “Working Directory” is “C:\Projects”.
  6. Under Environment tab:
    • Under the Interpreter sub-tab, click the New button and browse to “c:\xampp\php\php.exe”
    • Under the “Remote Debug” sub-tab, check “Remote Debug” and uncheck “Open with DBGSession URL in internal Browser”. Make sure that the Remote Sourcepath is “C:\Projects\myproject”.
    • Note: It is very important that the Remote Sourcepath (and all other paths) is set correctly; otherwise the PHP debug won’t work.
  7. Hit the Apply button.
  8. Start the PHP Debugger service by clicking on the Debug button below the Apply button. Take note of the port number (should be 10001) that the PHP Debugger service is listening on.

Debug Tracing PHP

  1. Open up “index.php” and set a breakpoint inside it (I suggest setting a breakpoint in the main entry routine).
    • You can set a breakpoint by clicking on the leftmost vertical blue bar in the file viewer. You should see a blue dot show up and an entry added to the Debug perspective’s Breakpoints window.
    • Alternatively, you can right-click on the leftmost vertical blue bar and select “Toggle PHP Breakpoint”.
  2. Browse to “http://localhost/myproject/index.php?DBGSESSID=1@localhost:10001” and the breakpoint should be hit. You can then step into, over, or out. Yippee!
    • Note: The 10001 in “localhost:10001” should match the port that the PHP Debugger is listening on in the EasyEclipse debug window.
    • Passing in the request parameter DBGSESSID will create a cookie (session based), so you no longer have to pass in the DBGSESSID http parameter in subsequent requests. Yipee again!
  3. Disable debug tracing for the current session by browsing to “http://localhost/myproject/index.php?DBGSESSID=-1”. Adding the parameter “DBGSESSID” with value “-1” will delete the debugging cookie.
  4. CAVEAT: Unfortunately, there is a bug with the PHP Debugger service in that after servicing the HTTP request, the PHP Debugger service will terminate.
    • You can quickly start the PHP Debugger service again by hitting F11 shortcut key. You will want to do this after each HTTP request is debugged.

Note: Some info above was derived from XOOPS Docman – Installing DBG.

1 Comment

Using Eclipse to Develop in PHP

Windows Development No Comments

easyeclipseThe EasyEclipse distribution is a quick and easy packaging of Eclipse with plugins for different purposes. I recommend using EasyEclipse for PHP which comes with PHP edit mode and useful plugins like SubEclipse for subversion source control integration.

  1. Download and install the latest version of EasyEclipse for PHP.
    • When you launch EasyEclipse, it will prompt you for the workspace directory. Input “c:\projects”. If EasyEclipse doesn’t prompt you for a workspace and instead uses a workspace from an old project, you can manually change the workspace by going to menu “File->Switch workspace”.
    • Note: It is not recommended for you to move or rename your workspace directory once created. The workspace properties have a reference to the original workspace directory.
  2. At this time, checkout the myproject codeline to c:\projects.
    • Go to the EasyEclipse menu “File->New->Project->SVN->Checkout Projects from SVN”.
    • Enter the Repository URL: http://localhost/repos/
    • Then select directory: /myproject
    • Change the project name to: myproject
  3. Edit PHPEclipse options to run XAMPP server:
    • Go to the php external tools settings “Windows->Preferences->PHPeclipse Web Development”. (The following steps will be referenced off of this location.)
    • Set the xampp start and stop buttons under “PHP External Tools->XAMPP”.
    • Set the DocumentRoot path under “Project Defaults” to: C:\projects
    • Add the following to the “Include Paths” under “Project Defaults” to: C:\projects\myproject
    • (Optional) Turn off php file browser preview. Under “Browser Preview Defaults”, uncheck “Refresh PHP browser view…” and “Show PHP browser view…” selections.
  4. (Optional) Configure the tab width to be 3:
    • Go to “Windows->Preferences->PHPeclipse Web Development->PHP”
    • Set “Displayed tab width” to 3.
  5. (Future) Synchronize your source to the latest SVN repository:
    • In the left-hand Navigator, right-click on “myproject”.
    • Select right-click menu “Team->Update”.
  6. Hint: If you are not in PHP edit mode, you can always go to it by going to menu “Windows->Open Perspective->PHP”

Note: If the subversion commands you issue using SubEclipse return errors, you may have an incompatible SVN install directory in the “path” environmental variable. To resolve this, you can remove the SVN install directory from the “path” environmental variable or you can re-install a compatible SVN version. For example, SVN 1.4.6 is compatible with SubEclipse 1.2.1 which comes with EasyEclipse 1.2.2.

No Comments

Add Subversion to the XAMPP Apache Server

Windows Development 9 Comments

subversionSubversion is a simple and popular source control system. It has the option of running as part of the Apache server. We’ll examine how to quickly get subversion up and running with the Apache server which comes with XAMPP.

  1. Download the latest subversion windows archive. Look for a file named “svn-win32-1.4.6.zip” (or with a later version number) at the bottom of the list.
  2. Unzip it to a local directory “c:\bin” and you will end up with “c:\bin\svn-win32-1.4.6”.
    • Add “c:\bin\svn-win32-1.4.6\bin” to the end of the Windows “path” environmental variable. This will allow us to run the command line subversion client and repository administration tool.
    • Warning: If you use Eclipse and the SubEclipse plugin, putting subversion in the “path” environmental variable may break the SubEclipse plugin. If this happens, just remove subversion from the “path” environmental variable and use Eclipse to perform subversion functions exclusively. Or ensure that your installed SVN version is compatible with SubEclipse; for example, SVN 1.4.6 is compatible with SubEclipse 1.2.1.
    • Also set “SVN_EDITOR=notepad.exe” to identify the default text editor to be used by the subversion command line tools.
  3. Create an initial blank repository called “svn_repos” by launching the command prompt window and running:
    svnadmin create c:\svn_repos
  4. Configure apache to use the subversion module
    • Stop the Apache server
    • Copy the apache module “c:\bin\svn-win32-1.4.6\bin\mod_dav_svn.so” into the apache modules directory “C:\xampp\apache\modules”. Go ahead and overwrite the existing “mod_dav_svn.so” file as it is from an older version of subversion.
    • Open up the apache config file “C:\xampp\apache\conf\httpd.conf”
    • Enable the subversion module by adding “LoadModule dav_svn_module modules/mod_dav_svn.so” anywhere after the existing “LoadModule dav_module modules/mod_dav.so” line. The best thing would be to add it as the last LoadModule line.
    • At the end, add a URL location (what you would enter into the browser or your subversion client) for your repository:
      <Location /repos>
          DAV svn
          SVNPath c:/svn_repos
      </Location>
    • Start the Apache server
    • You can access the repository using the URL “http://localhost/repos”. You should see a “Revision 0” header with no other data. If you get an error, check the “C:\xampp\apache\logs\error.log” file for details on the problem.
    • Note that you will need to add a location block like the above for each repository that you create with “svnadmin create”. (You shouldn’t need to do this because a repository can contain as many projects as you want.)
    • Note: we have not secured the repository so anyone can read or write to it. To secure subversion, see my followup post.
  5. Import an existing project (source code directory) into the “svn_repos” repository
    • If you have source code in a project directory like “c:\oldprojects\myoldproject”, you can import the contents of “myoldproject” into the subversion “svn_repos” repository by running this command:
      svn import c:\oldprojects\myoldproject http://localhost/repos/myproject -m "Initial import"
    • The comment “-m “Initial import”” is optional. If you don’t input it, the default subversion editor (“notepad.exe”) will appear. Add your import comment and save the file. If you close the file without making any changes, you will need to input “c” into the command prompt window to continue the import.
    • Browse to “http://localhost/repos/myproject” and you should see “Revision 1” header with the list of folders and files that have been imported.
  6. You can create a new empty project by creating an empty folder and importing it (see above).
  7. Checkout the initial “myproject” project to create a local working copy
    • Open the command line and go to the “c:\projects” directory.
    • Check out “myproject” by issuing the following command:
      svn co http://localhost/repos/myproject myproject
    • You will see a “c:\projects\myproject” directory with your imported folders or files (if you did the import step above). If you didn’t import, you will see an empty directory except for a “.svn” subdirectory; don’t touch the “.svn” subdirectory as it is used by the subversion client for bookkeeping.
  8. Inside your local working copy “c:\projects\myproject”, you can issue the following common subversion commands:
    • To update your local working copy with the latest revision from the repository:
      svn update
    • To add a new file:
      svn add newfile.cpp
    • To remove a file:
      svn rm oldfile.cpp
    • To move and/or rename a file:
      svn mv oldfile.cpp subdir/newoldfile.cpp
    • List the changes (aka status) that you have made to your working copy:
      svn st
    • To commit (aka checkin) the local working changes to the repository:
      svn ci -m "my checkin comments"

      If you don’t input the comment using the “-m” flag, the default editor (“notepad.exe”) will open to allow you to input the comment.

  9. To ignore certain subdirectories or files that you do not wish to check into subversion
    • Go to the directory containing the subdirectories or files to be ignored and run this command:
      svn propedit svn:ignore .
    • The default editor (“notepad.exe”) will open. Input the subdirectory and file names, one per line.
    • Instead of putting the exact filename, you can use a file pattern like “*.obj”.
    • Save and close the file.
    • To make this change permanent, issue a “svn ci” command.
    • You can also create a text file “ignore.txt” containing the subdirectory, filenames, and file patterns that you wish subversion to ignore, and apply it recursively to the current and all subfolders by running:
      svn propset -R svn:ignore . -F ignore.txt

Some info above derive from Ned Batchelder: Subversion on Windows quick start.

9 Comments

Setting Up an Apache, MySQL, PHP Development Environment with XAMPP

Windows Development 2 Comments

Rather than installing Apache, MySQL, and PHP individually, XAMPP is an integrated development enironment that combines all three (and more) into one easy installation.

  • Download XAMPP and install it into the default “c:\xampp” directory. Set apache and mysql to start as services.
  • Start XAMPP and browse to “http://localhost/” and you will see a welcome message.
  • If you don’t want to use the default “c:\xampp\htdocs” directory as your document root, you can update the DocumentRoot “C:/xampp/htdocs” property and <Directory “C:/xampp/htdocs”> element in the “c:\xampp\apache\conf\httpd.conf” file. The result should look like:
    DocumentRoot "c:/projects"
    <Directory />
    ... do not change this / default root block...
    </Directory>
    <Directory "c:/projects">
    ...
  • Likewise, if you use SSL, update DocumentRoot “C:/xampp/htdocs”
    in “c:\xampp\apache\conf\extra\http-ssl.conf”.
  • Note: XAMPP uses the “c:\xampp\apache\bin\php.ini” when running PHP. There is a second “c:\xampp\php\php.ini” which is used if you call “c:\xampp\apache\bin\php.exe” from the command line.
  • Edit “c:\xampp\apache\bin\php.ini” to enable logging by setting the following properties:
    error_reporting = E_ALL
    log_errors = On
    error_log = "C:\xampp\apache\logs\phperror.log"
  • To enable access to the original XAMPP htdocs directory, we can create a subpath redirect to it using the alias function.
    1. Edit “c:\xampp\apache\conf\httpd.conf” and locate the <Directory “C:/projects”> block (or whatever path you had selected as document root).
    2. Immediately after the whole <Directory> block above (look for the matching end tag </Directory>), add the following text:
      # Allow access to the original xampp htdocs
      Alias /htdocs C:/xampp/htdocs
      <Directory "C:/xampp/htdocs">
          Options Indexes FollowSymLinks Includes ExecCGI
          AllowOverride All
          Order allow,deny
          Allow from all
      </Directory>
    3. Adjust the htdocs files to account for the base URL change:
      • Edit “c:\xampp\htdocs\index.php”, locate header(‘Location: ‘.$uri.’/xampp/’);, and change that to be header(‘Location: ‘.$uri.’/htdocs/xampp/’);.
      • Repeat the above for “c:\xampp\htdocs\xammp\index.php” and “c:\xampp\htdocs\xampp\lang.php”.
      • Edit “c:\xampp\htdocs\xampp\splash.php”, locate
        <a href=”/xampp/lang.php?’.$key.'”>
        , and change that to be <a href=”/htdocs/xampp/lang.php?’.$key.'”>.
    4. Start XAMPP, open your browser to “http://localhost/htdocs” and you will see the original xampp htdocs directory.
  • If you attempt to access Apache from another machine in your network and your brower returns an access error, you may need to adjust your Windows Firewall. The latest version of Windows Firewall forces you to manually open up ports for services.
    1. Go to Start, Control Panel, Windows Firewall and click on the Advanced tab.
    2. Make sure that “Local Area Connection” is checked and hit the Settings button to the right of it.
    3. Under Services tab, make sure that “Web Server (HTTP)” is checked.
2 Comments

Backing Up Windows Desktop Locally and Remotely

Windows 4 Comments

To prevent data loss on my Windows desktop, I’ve created automated scripts to backup the data locally and remotely.

For both sections below, we will pretend that I want to backup two directories:

  • C:\Documents and Settings\username\My Documents\Tax Files
  • C:\Documents and Settings\username\My Documents\Photos

Backing Up Windows Desktop Locally

The best way to backup your data locally is to have a second hard drive and copy your data to that second hard drive. This way, if the primary hard drive dies, you still have a copy of your data on the second hard drive.

To do the copy, you could just use the DOS “xcopy” command, but that is very inefficient if you have a large amount of data to copy; each time, it will copy everything again (even if nothing changes). The better way to copy is to apply only the differences between the original data and the copy on the second hard drive. I recommend using Beyond Compare for this purpose (unfortunately, it costs $30).

Warning: I tried using rsync to do local copies on Windows, but it had problems with long path names and would sometimes remove access permissions from copied directories and files. As a result, I wasn’t able to read the copied directories and files or delete them. With administrative rights, I was able to recursively give myself full rights to the copied directories and files again. In short, I don’t recommend using rsync for copies where the destination target is Windows. If you are still interested, here’s the local-to-local rsync command:

cd "\Documents and Settings\username"
rsync.exe -vrtc --progress "My Documents" "/cygdrive/D/Backups/Daily/"

Tip: For remote Unix targets, you can recursively adjust the permissions by using ssh to issue a chmod:

ssh user@hostname 'chmod -R 755 /directory'

Here is the Beyond Compare script file, named “daily_backup.bc2”, which I used:

# Turn on logging to a file
log normal "C:\temp\bc_synclog_%date%.txt"

# Load the source and destination folder
load "C:\Documents and Settings\username\My Documents" "D:\Backups\Daily\My Documents"

# Mirror only these directories (the endings slash indicates directory)
filter "Tax Files\;Photos"

# One-way Mirror from source to destination
sync create-empty mirror:lt->rt

Here is the DOS batch file, named “daily_backup.bat”, which is used to launch Beyond Compare:

C:\Program Files\Tools\Beyond Compare 2\bc2.exe" @"C:\Scripts\daily_backup.bc2"
move C:\Temp\bc_synclog_*.txt "D:\Backups\Daily"

Now to schedule Windows to run the DOS batch file once a day:

  1. Go to menu “Start->Control Panel->Scheduled Tasks->Add Scheduled Task”. Click Next.
  2. Click Browse button and select the DOS batch file “daily_backup.bat”.
  3. Select “Daily” and click Next.
  4. Select a start time and click Next.
  5. Input your password. Click Next. Click Finish.

Backing Up Windows Desktop Remotely

The issue with a local backup is that if there is an accident (like a fire), both hard drives may be destroyed. The best backup is to another server which is not physically located in the same location as your desktop.

If you have a remote server which allows FTP access, you can use Beyond Compare to do the copy. Here is the Beyond Compare script to do so:

# FTP prompts if overwriting, set auto confirm yes
option confirm:yes-to-all

# Turn on logging to a file
log normal "C:\temp\bc_synclog_remote_%date%.txt"

# Load the source and destination folder (all on one line)
load "C:\Documents and Settings\username\My Documents" "ftp://username:password@ftp.hostname/backup"

# Mirror only these files and directories
filter "Tax Files\;Photos"

# Mirror
sync create-empty mirror:lt->rt

If the Beyond Compare FTP hangs, you may need to enable passive mode.

  1. Launch Beyond Compare
  2. Go to menu “Tools->Options->FTP->Firewall / Proxy”
  3. Check the “Passive Mode” box

Beyond Compare FTP does not support delta copies (just copying the differences in the files); so for efficiency, you may wish to use the free rsync Windows port called DeltaCopy if your remote server supports rsync.

Here is the DOS batch script to issue the rsync command:

cd "\Documents and Settings\username\My Documents"
C:\Scripts\rsync\rsync.exe -vrt --delete "Tax Files" username@hostname:~/backup/
C:\Scripts\rsync\rsync.exe -vrt --delete Photos username@hostname:~/backup/

Unless you establish trust with the remote server (the DeltaCopy website has instructions on how to install the ssh-keygen.exe utility or you can just download PuTTYgen), rsync will prompt you for the password. This will make automating the remote backup rather difficult since it won’t be able to run unattended. In the case where your remote server doesn’t support establishing trust, you will need to use a scripting tool like Expect, which comes as a part of ActiveState Tcl.

Warning: Other programs may come with their own ssh.exe and ssh-keygen.exe executables which are not compatible with rsync. To avoid issues, you will want to make sure that the rsync directory is the first one in the “%PATH%” environmental variable.

After installing the ActiveState Tcl above (pick the standard distribution of ActiveTcl which is free), you will need to run the following command to install Expect:

C:\Scripts\Tcl\teacup.exe install Expect

Here is the Tcl/Expect script, named “expect_rsync.tcl”, that will run RSYNC, wait for the password prompt, and input the password for you:

# Load the expect extension
package require Expect

# Enable logging

log_user 1

# Set timeout for password wait to 10 secs

set timeout 10

# Input arguments

array set OPTS {
   host   ""
   user   ""
   passwd  ""
   backup  ""
}

# Usage info

proc usage {code} {
   global OPTS
   puts [expr {$code ? "stderr" : "stdout"}] \
   "$::argv0 -host hostname -user username -passwd password -backup location ?options?
   -help         (print out this message)"

   exit $code
}

# Parse the arguments

proc parseargs {argc argv} {
   global OPTS
   foreach {key val} $argv {
      switch -exact -- $key {
         "-host"   { set OPTS(host)   $val }
         "-user"   { set OPTS(user)   $val }
         "-passwd" { set OPTS(passwd) $val }
         "-backup" { set OPTS(backup) $val }
         "-help"   { usage 0 }
      }
   }
}

parseargs $argc $argv

# Make sure we are not mssing any input arguments

if {$OPTS(host) == "" || $OPTS(user) == "" || $OPTS(passwd) == "" || $OPTS(backup) == ""} {
   usage 1
}

# Spawn an rsync process

spawn rsync -vrt --delete $OPTS(backup) $OPTS(user)@$OPTS(host):~/backup/
match_max 10000

set id $spawn_id

# Look for passwod prompt

expect -i $id timeout {
   puts "timed out"
   exit -1
} eof {
   puts "spawn failed with eof"
   exit -1
} "*?assword:*" {
   send -i $id -- "$OPTS(passwd)\r"
}

# send blank line make sure we continue session

send -i $id -- "\r"

# Close session and wait for spawn return

wait -i $id

Here is the DOS batch script to issue the Tcl rsync command:

cd "\Documents and Settings\username\My Documents"
C:\Scripts\Tcl\tclsh85.exe C:\Scripts\rsync\expect_rsync.tcl -host hostname -user username -passwd password -backup "Tax Files"
C:\Scripts\Tcl\tclsh85.exe C:\Scripts\rsync\expect_rsync.tcl -host hostname -user username -passwd password -backup Photos

Encrypting Your Remote Backup Files

If your remote server is not 100% secure (it may be your web hosting service), you may wish to encrypt your backups before copying them to the remote server. I use Winzip for this purpose. Winzip 10 or later comes with a free command line interface which we can script.

Here is the DOS batch script I use to zip, encrypt, and copy to the remote server:

"C:\Program Files\Tools\WinZip\wzzip.exe" -s<password> -ez -p -r -u -ycAES256 -ybc -yu "C:\Documents and Settings\personal\My Documents\zip_archives\tax_files.zip" "C:\Documents and Settings\personal\My Documents\Tax Files\*"
"C:\Program Files\Tools\WinZip\wzzip.exe" -s<password> -ez -p -r -u -ycAES256 -ybc -yu "C:\Documents and Settings\personal\My Documents\zip_archives\photos.zip" "C:\Documents and Settings\personal\My Documents\Photos\*"
C:\Scripts\rsync\rsync.exe -vrt --delete zip_archives username@hostname:~/backup/

The “-u” flag given to Winzip tells Winzip to update the archive file. This will add new files to or update existing files in the zip archive which have changed. Using this flag will drastically reduce subsequent executions of Winzip at the expense of increasing archive file size. If you wish to reduce the size of the archive file, just delete the archive file and a subsequent run of Winzip will regenerate it.

RSync over SSH

In the worse case, your remote server may support SSH but not rsync. You can still run rsync over the SSH connection. Here is the updated rsync command to use:

C:\Scripts\rsync\rsync.exe -vrt --delete -e "ssh -p <port> -l <username> -v" zip_archives hostname:~/backup/

You may omit “-p <port>” if your remote server uses the default SSH port of 22.

Unfortunately, if you cannot establish trust with the remote server and need to input the password, automation using the Tcl/Expect script is not possible. There is an issue in the current version of ActiveState Expect that prevents it from working with rsync over SSH.

DOS Batch Tools

I had to create two command line tools to enhance the DOS batch functionality. The first tool is called WinClose.exe and it is used to close running programs. I used it to close Microsoft Outlook to unlock the storage file so that I could copy it. The second tool is called Wait.exe and it is used to pause the batch execution to allow Microsoft Outlook enough time to completely close (5 seconds) before the copy occurs. You can find these two tools in my Open Source page.To prevent data loss on my Windows desktop, I’ve created automated scripts to backup the data locally and remotely.

4 Comments

Automated Daily Backup of the mySQL Database

Linux 8 Comments

Now that I have my tikiwiki running on a Linux AS4 server and having felt the sting of having to recover it, I have incentive to create an automated daily backup of the mySQL database. I have decided to do the backup using a simple shell script and schedule it using the Linux crontab.

Here is the final version of the shell script (saved to “/root/sql_backup.sh”). This script will do a mySQL dump of one database to a gzip archive file in a backup directory, keep only the 10 most recent archive files (pruning the rest), and rsync the backup directory to a remote server.

#!/bin/sh
# MySQL backup script - do a backup and rsync to remote server

# mySQL
SQLROOTPASS=mysqlrootpassword
SQLDB=mysqldatabasename

# Backup Dirs
LOCALDIR=/home/backup/mysql
REMOTEHOST=myremoteserver
REMOTEDIR=/home/backup/mysql
NUMBACKUPS=10

# Create local dir
mkdir -p $LOCALDIR

# Do a mySQL backup
NOW=$(date +"%Y-%m-%d-%H-%M-%S")
echo "Doing a database backup to $LOCALDIR/$SQLDB-$NOW.sql.gz"
# Note: The following mysqldump command should be one line only!
mysqldump -u root -h localhost -p$SQLROOTPASS $SQLDB | gzip -9 > $LOCALDIR/$SQLDB-$NOW.sql.gz

# Prune old backups
FILES=$(ls -t $LOCALDIR/*.gz)
for FILE in $FILES
do
if [ $NUMBACKUPS -gt 0 ]
then
   echo Keeping $FILE
else
   echo Deleting $FILE
   rm $FILE
fi
NUMBACKUPS=$(expr $NUMBACKUPS - 1)
done

# Create remote dir
ssh $REMOTEHOST mkdir -p $REMOTEDIR

# Sync from local dir to remote dir
echo "RSyncing to $REMOTEHOST.."
rsync -rt --delete $LOCALDIR/ $REMOTEHOST:$REMOTEDIR/

Rather than explain the whole script, I’ll just point out some parts that may require more attention:

  • Unfortunately, this script does contain the mySQL root password in the clear. I haven’t figured out how to do a mysqldump as a non-root SQL user yet.
  • To avoid the need to include the remote server’s root password, this script will require that you have already established trust with the remote server. This avoids the need to input a password when using RSync, SSH, or SCP. See below for instructions on how to establish this trust.
  • When doing an assignment to a variable in the script, don’t insert spaces after the variable name. For example, “TEMPVAR=Hello” works but “TEMPVAR =Hello” won’t.
  • The format “TEMPVAR=$(…)” tells the script to execute the command inside and assign the result to the variable. This is equivalent to the format “TEMPVAR=`…`” using backward single quotes.
  • In order to configure RSync to delete files on the remote system (otherwise we’ll use up all the hard drive space), we must use the -r flag (recurse into subdirectories), –delete flag, and use a directory (“$LOCALDIR/” ending with a slash) as the source. The -t flag (persist file modification time) is necessary for RSync to do delta updates.

To establish trust with a remote server, do the following:

  1. Log into your server as the root user.
  2. Look for this file “/root/.ssh/id_rsa.pub”.
  3. If the file or directory doesn’t exist, then you will need to generate the RSA public/private key:
    ssh-keygen -t rsa

    Just take the default inputs and your RSA keys will be generated.

  4. Copy the public key “id_rsa.pub” to the remote server’s “/root/.ssh” directory.
    scp /root/.ssh/id_rsa.pub myremoteserver:/root/.ssh/my_id_rsa.pub

    If that directory does not exist, you may wish to generate the RSA public/private keys there also.

  5. You will need to add the contents of “my_id_rsa.pub” to the end of the remote server’s “/root/.ssh/authorized_keys” file (it contains keys from other servers which the remote server should trust). If the file doesn’t exist, then create it as an empty file.
    cp authorized_keys authorized_keys.original
    cat authorized_keys my_id_rsa.pub > authorized_keys
    rm my_id_rsa.pub
  6. You can test it out by making an SSH connection to the remote server.
    ssh root@myremoteserver

    If you are not prompted for the password, then the trust was successfully established.

Finally, let us schedule the mySQL backup to run every day at 3am using the Linux crontab:

crontab -e
0 3 * * * sh /root/sql_backup.sh > /tmp/sql_backup.output 2>&1

Some hints about using the crontab:

  • “crontab -e” will allow you to edit the crontab contents.
  • The crontab line format is:
    <minute=0-59> <hour=0-23> <day of month=1-31> <month=1-12> <day of week=0-6 (sun-sat)> <command>
  • Precede the crontab line with a # (pound) character to indicate a comment which crontab will ignore.
  • The “2>&1” will redirect STDERR to STDOUT, which in turn is redirected to “/tmp/sql_backup.output” above. (0=STDIN, 1=STDOUT, 2=STDERR)
  • “crontab -l” will list the crontab contents.
8 Comments

Restoring Tikiwiki 1.8.5 on Red Hat Linux AS4

Linux No Comments

redhatRecently, I had to rebuild a tikiwiki server (on a Linux machine) with only a mysql database backup image. I quickly found that the backup image of a tikiwiki does not contain the version of the tikiwiki software, which is very necessary; otherwise, the database schema won’t match. Fortunately, I managed to find that the tikiwiki version was 1.8.5 from some old notes.

Here is the resulting software stack after I finished.

  • Red Hat Enterprise Linux AS Release 4
  • mySQL Server 4.1.20
  • Apache HTTP Server 2.0.52
  • PHP 4.3.9
  • Tikiwiki 1.8.5 Polaris

Here’s how I went about restoring the tikiwiki (assuming that the machine already has Redhat AS4 installed):

  1. Install PHP, HTTP Daemon (httpd), and supporting libraries:
    yum list | grep -i php
    yum install php.i386
  2. Install mySQL server (mysqld) and supporting libraries:
    yum list | grep -i mysql
    yum install mysql-server.i386
  3. Install mySQL client and the PHP-mySQL libraries:
    yum list | grep -i php-mysql
    yum install php-mysql.i386
  4. Start both mysqld and httpd and configure them to start on reboot:
    chkconfig mysqld on
    service mysqld start
    chkconfig httpd on
    service httpd start
  5. Configure mysqld and setup database:
    (Set mySQL root password)
    mysqladmin -u root password 'mypassword'

    (Create a database)
    mysqladmin -u root -p create mydatabase

    (Login as mySQL root user)
    mysql -u root -p

    (Use main mySQL database)
    mysql> use mysql;

    (Give localhost permission to access all databases)
    mysql> insert into host(host, db, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv) values('localhost', '%', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y');

    (Create a user)
    mysql> insert into user (host, user, password) values('localhost', 'myuser', password('mypassword'));

    (Give that user access to the created database from localhost)
    mysql> insert into db (host, db, user, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv) values ('localhost', 'mydatabase', 'myuser', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y');

    (Quit from mySQL)
    mysql> quit;

    (Reload mySQL server)
    mysqladmin -u root -p reload
  6. Reload the HTTP Server just in case:
    service httpd reload
  7. Make sure that PHP is working by copying this testphp.php file to “/var/www/html” and browsing to “http://localhost/testphp.php”:
    <html>
    <head>
    <title>PHP Info Script</title>
    </head>
    <body>
    <?php
    phpinfo();
    ?>
    </body>
    </html>
  8. Make sure that PHP-mySQL is working by copying this testmysql.php file to “/var/www/html”, replacing the text ”mysql_root_password” with your real password, and browsing to “http://localhost/testmysql.php”:
    <html>
    <title>mySQL Test Script</title>
    <body>
    <?php
    // connect and select a database
    $link = mysql_connect("localhost:3306", "root", "mysql_root_password")
    or die ("Couldn&#39;t connect:  Check to make sure that:<br>" .
            "<ul><li>your MySQL server is running</li>" .
            "<li>you used the correct hostname (<tt>vergil/ovid</tt>)</li>" .      
            "<li>you added a colon with your port number after the hostname</li>" .
            "<li>you used the username &#39;root&#39;</li>" .      
            "<li>you used the correct root password</li>" .
            "<li>you didn&#39;t forget to close a set of quotation marks</li><br><br>");
    print "Connected successfully.<br>";
    $db = "mysql";
    mysql_select_db($db) or die("Could not select the database '" . $db . "'.  Are you sure it exists?");

    // perform an SQL query
    $query = "SELECT * FROM user";
    $result = mysql_query($query) or die("Query failed");

    // print the result of the first row (row counting starts at zero)
    printf("Host: %s<br>\n", mysql_result($result, 0, "Host"));                  
    printf("User: %s<br>\n", mysql_result($result, 0, "User"));
    printf("Grant privilege: %s<br>\n", mysql_result($result, 0, "Grant_priv"));

    // free result set
    mysql_free_result($result);
    // close the connection
    mysql_close($link);
    ?>
    </body>
    </html>
  9. Unzip the tikiwiki 1.8.5 archive to the “/var/www/html” directory and run the setup script:
    chmod u+x setup.sh
    ./setup.sh
    chmod u-x setup.sh
  10. Configure the tikiwiki by browsing to ”http://localhost/tiki-install.php” and inputting the tikiwiki database name, user, and password.
  11. Restore the database from the backup image:
    mysql -u root -p
    mysql> use tikiwiki;
    mysql> source tiki_backup.sql
  12. And we are done!

Some info above was provided by Linux Help – mySQL setup Guide.

No Comments

The Sunroof From Heck

Hardware No Comments

jettafronttubewater.jpgThree years ago, I went on vacation. I left my Volkswagen Jetta, not yet one year old at the time, at my parents for a week. During that time, there were heavy rains. When I came back, the left-front driver foot well was under three inches of water. I was sure that I had locked the doors securely before leaving… and all the windows, including sunroof, were closed tightly. What could be the matter? Maybe it was a leak somewhere on the driver side?

A wet vac, a lot of effort, and the problem was mostly solved, except for the unpleasant wet carpet smell which lingered before disappearing. Being a procrastinator, I never did take the car to the dealer to find the leak.

Since then, rains have come and gone. Unless the rains were heavy, I didn’t see any moisture in the cabin. But when the heavy rains came, the water appeared in the front driver foot well. I checked the driver-side door but didn’t see any wet trail. I couldn’t figure it out.

Recently, during some heavy rains, I noticed that the front right passenger foot well was also accumulating water. What the heck? Did my Jetta just spring another leak? I resolved to get down to the root of the problem; I went to the Internet and found complaints about the sunroofs from all cars including Acura, Mercedes, and Volkswagen (bingo!).

jettafronttube.jpgEvidently, the sunroofs are not water-tight. Water does get into and accumulate in the well of the sunroof. The water is usually drained away by tubes. If the tubes get clogged or disconnected, then the water builds up, eventually floods into the roof structure, and appears in the main cabin in several places. Supposedly once a year, you should check that sunroof drainage tubes and clear any blockage.

I cracked open the trusty Jetta owner’s manual; it didn’t mentioned anything about the above. What the heck? How many drivers out there are wondering why their brand new car with sunroof starts leaking after a year or two?

On the Jetta, there are four drainage tubes, one at each corner of the sunroof. The front two tubes come out in the middle of both sides, near the front door hinges. The rear two tubes comes out underneath both sides near the rear bumper. Now, why the front two doesn’t come out underneath, I have no idea. Worse, there are nipples at the end of each of the four tubes… I guess to promote clogging?

jettareartube.jpgOthers recommend cutting the nipples off, but I took the less severe method of squeezing the nipples to make them open wide. Pouring water into the sunroof and then squeezing the nipples caused all sorts of particles, some big and some small, to come out, followed by a torrent of freed water. Yippee! Now, I just need to remember to do this once or twice a year.

If you don’t have time to do the above, another quick fix is to use painter’s tape (aka masking tape) to seal all four edges of the sunroof.

Update: A couple of years later, Volkswagen issued a safety recall concerning this issue. The solution was to cut the rubber nipples off.

No Comments

Getting T-Mobile EDGE to Work on Your iPhone 1.1.2

Mobile Devices 4 Comments

iphone1gThe following steps will configure your iPhone 1.1.1 or 1.1.2 to work with T-Mobile’s $5.99 Internet Access plan (aka T-Zone or T-MobileWeb). You will need to have a basic knowledge of how to use the command line.

  1. First, jailbreak and unlock your iPhone.
  2. Disable the Auto-Lock (to prevent the iPhone from going to sleep) by pressing the home button and tapping on Settings->General->Auto-Lock->Never.
  3. If you have not already installed the BSD Subsystem, you should install it now:
    • Press the home button and tap the Installer icon. Update the installer if necessary.
    • Tap the Install tab, scroll to the System category, and install the BSD Subsystem. If you don’t see BSD Subsystem listed, then it is already installed.
  4. Go back to the Categories list by tapping the Install tab, scroll to System category, and install OpenSSH.
  5. Find the IP Address of the iPhone:
    • Press the home button and tap on Settings->Wi-Fi.
    • Locate your wireless network and click on the blue arrow icon to the far right.
    • Look for the IP Address line. Your IP Address should look something like 192.168.0.21.
  6. On your computer, launch an SSH client and connect to your iPhone.
  7. On Windows, do the following:
    • If you are running Windows and need a free SSH client, download putty.zip from the PuTTY website.
    • Uncompress putty.zip to a folder (ex: c:\temp\putty) and run Putty.exe.
    • Input the IP address of the iPhone into the Host Name (or IP address) field.
    • Click on the Open button at bottom.
  8. On Mac OS X, do the following:
    • Launch the Terminal program.
    • Type ssh root@192.168.0.21. (Please replace 192.168.0.21 with the IP address of your iPhone.)
  9. It will take 30-40 seconds to connect to the iPhone the first time because the iPhone will be generating SSH security keys.
  10. Once the iPhone is done, you will get a prompt on your SSH client to accept the iPhone’s SSH security keys. Answer yes. (Subsequent connections will be faster and will skip this security prompt.)
  11. Input the username root and the password alpine.
  12. Once you have successfully logged into the iPhone, you may wish to change the root password to prevent unauthorized access:
    • Type passwd into the SSH window and hit Enter.
    • Input the new root password twice. (The next time you connect with SSH, you will need to use this new password to login.)
  13. Backup the two system files that we will modify (if they exist):
    • Type cd /private/var/root to go to that directory.
    • Type ls to list the contents of the directory.
    • If you see a file called proxy.pac (most likely, you won’t), make a backup copy of it by typing this command: cp proxy.pac proxy.pac.original
    • Type cd /private/var/root/Library/Preferences/SystemConfiguration and then type ls
    • You should see a file named preferences.plist. Make a copy of it using this command: cp preferences.plist preferences.plist.original
  14. Create a custom version of proxy.pac and upload it to the iPhone.
    • On Windows, use the Notepad text editor. On Mac OS X, use TextEdit text editor.
    • Input the following text into the editor:
      function FindProxyForURL(url, host)
      {
          if (isInNet(myIpAddress(), "10.0.0.0", "255.0.0.0"))
              return "PROXY 216.155.165.50:8080";
          else
              return "DIRECT";
      }
    • Save this file as proxy.pac.
      • On Windows Notepad, make sure you select menu File->Save as… and All Files in the Save as type input. This will prevent Notepad from appending a .txt to create an erroneously named proxy.pac.txt file. Save it to a directory like c:\temp.
      • On Mac OS X TextPad, save it to a directoy like /temp.
    • Upload the file to the iPhone using Windows Command Prompt:
      • Launch the Command Prompt by selecting menu start->run and typing cmd.
      • Go to the directory where PuTTY was uncompressed to by typing a command like cd c:\temp\putty.
      • Secure copy the proxy.pac file to the iPhone with the command: pscp c:\temp\proxy.pac root@192.168.0.21:/private/var/root
    • Upload the file to the iPhone using Mac OS X Terminal: scp /temp/proxy.pac root@192.168.0.21:/private/var/root
  15. Modify the preferences.plist and upload it to the iPhone. Some websites will say to totally replace the preferences.plist file but this is wrong; each iPhone comes with a unique preferences.plist that needs to be custom modified.
    • Copy the existing preferences.plist from the iPhone.
      • On Windows Command Prompt: type cd c:\temp\putty and pscp root@192.168.0.21:/private/var/root/Library/Preferences/ SystemConfiguration/preferences.plist . (Don’t forget to input the last period!)
      • On Mac OS X Terminal: type cd /temp and scp root@192.168.0.21:/private/var/root/Library/Preferences/ SystemConfiguration/preferences.plist .
      • Note: In the commands above, there is no space or break between Preferences/ and SystemConfiguration.
    • Open the preferences.plist. On windows, Notepad will not be adequate for this task; instead, use Wordpad by typing in the Command Prompt: write preferences.plist
    • Locate the following section:
      <dict>
      <key>Interface</key>
      <dict>
      <key>DeviceName</key>
      <string>ip1</string>
      <key>Hardware</key>
      <string>com.apple.CommCenter</string>
      <key>Type</key>
      <string>com.apple.CommCenter</string>
      </dict>
    • Immediately after, add this new section:
      <key>Proxies</key>
      <dict>
      <key>ProxyAutoConfigEnable</key>
      <integer>1</integer>
      <key>ProxyAutoConfigURLString</key>
      <string>file:///private/var/root/proxy.pac</string>
      </dict>
    • Save the file, close the text editor, and copy the file back to the iPhone:
      • On Windows Command Prompt: pscp preferences.plist root@192.168.0.21:/private/var/root/Library/Preferences/ SystemConfiguration
      • On Mac OS X Terminal: scp preferences.plist root@192.168.0.21:/private/var/root/Library/Preferences/ SystemConfiguration
      • Note: In the commands above, there is no space or break between Preferences/ and SystemConfiguration.
  16. Reboot your iPhone by holding down the home and power buttons until you see the startup screen with the white apple logo.
  17. Click the home button and tap on Settings->General->Network->Edge. Make sure that the APN is set to wap.voicestream.com, the Username is blank, and the Password is blank.
  18. Turn Off the Settings->Wi-Fi to force the iPhone to use the EDGE network.
  19. See if you can still use Safari to browse the internet. You should see a blue E appear near the top left while you are using the T-Mobile EDGE network. If you can browse the internet, congratulations! (If the EDGE connectivity doesn’t work, see troubleshooting tips at the end of this post.)
  20. Turn the Settings->Wi-Fi back On . Make sure that Settings->Wi-Fi->Ask to join networks is On to allow the iPhone to use Wi-Fi or EDGE (if Wi-Fi is not available).
  21. Re-enable Auto-Lock by going to Settings->General->Auto-Lock. The default is one minute.

If you need a way to explicitly turn off the EDGE network support (and the SSH access service), you can install the Services application:

  1. Press the home button and tap the Installer icon. Update the installer if necessary.
  2. Tap the Install tab, scroll to the System category, and install the Services application.
  3. Launch Services to toggle the EDGE, Wi-Fi, Bluetooth, or SSH support off or on.

Some advance troubleshooting steps which I read about but have never tried (as the steps above worked perfectly for me):

  • Power off the iPhone (hold the power button until you see slide to power off), tap on the slide to power off, wait a minute, and then power it back on. Try Safari.
  • Double-check the Settings->General->Network->Edge APN configuration. If the APN is not wap.voicestream.com, change it, save, and reboot. Try Safari.
  • Double-check the APN again. If it is not wap.voicestream.com, then the save did not work. Instead, modify the preferences.plist directly, reboot the iPhone, try Safari.
    <dict>
    <key>apn</key>
    <string>wap.voicestream.com</string>
    <key>password</key>
    <string></string>
    <key>username</key>
    <string></string>
    </dict>
  • Try using APN wap.voicestream.com with guest for both Username and Password.
  • Try using ApN internet2.voicestream.com with blank Username and Password.
  • Try using ApN internet2.voicestream.com with guest for both Username and Password.
  • Try using this proxy.pac instead:
    function FindProxyForURL(url, host)
    {
        if (isInNet(myIpAddress(), "10.0.0.0", "255.0.0.0"))
            return "PROXY 10.23.8.3:8080";
        else
            return "DIRECT";
    }

The information above is consolidated from Complete Guide: t-mobile on the iPhone, How to Unlock Your iPhone and Use The $5.99 T-Zones Plan, and T-Mobile EDGE and iPhone.

4 Comments

« Previous Entries