Make a Dumb TV Smart Using Roku and Plex

Audio Visual No Comments

My brother-in-law gave me his huge 65 inch LCD TV. Score! But it was a dumb TV with no Internet capability or apps support. And I didn’t have cable or a digital OTA (Over the Air) tuner. Fortunately, there were many ways to turn a dumb TV into a smart TV. I chose the Roku 2 XS 1080p Streaming Player. Why? Because someone was selling one for $20 on craigslist.

Once I owned the Roku 2 XS, I started to see why it was the best media player to purchase (mere-exposure effect or familiarity principle). You see, the Roku 2 XS has the best features of the Roku 3 (USB port, Bluetooth remote, SD card) while keeping the backward compatibility of supporting a composite AV (audio/video) output, in addition to HDMI. The USB port allows me to directly connect a USB flash drive or portable hard drive full of videos to the Roku. The Bluetooth remote means I can hide the Roku behind the TV and still be able to control it. Finally, I think the old TV only does 1080p, so the 1080p supported by Roku 2 XS is perfectly fine.

Though the Roku supported wireless Internet, I hooked it up using a wired LAN connection to avoid any possible lag in throughput (can’t hurt when streaming 1080p HD video). Once I finished attaching the Roku to the TV with a HDMI cable, it was time to power it on. Because I did a factory reset on the Roku, I had to pair the remote again by pressing the little purple button inside the remote’s battery compartment for 3 seconds. I created an account with Roku and added channels (Roku’s name for apps); for example, the YouTube, Netflix, and Amazon Video channels are the most popular if you have kids. (If you want to play videos from the SD card or USB port, you will need to install the Roku Media Player channel.)

The Roku has no off switch, so it is immediately ready to use (instead of booting up each time). I read that it will go to sleep after 30 minutes of inactivity. Make sure to always stop playing before turning off the TV though. Or just unplug the Roku.

Cast to Roku

Because Roku supports the DIAL (Discovery and Launch) protocol used by Chromecast, you can cast to it from YouTube on your Windows desktop, Macbook laptop, or smartphone. Just click on the cast icon in the YouTube player and select the Roku.

The YouTube cast function and YouTube Channel are both temperamental. Sometimes when I try to cast, I have to cast twice before the video would actually play on the Roku. Sometimes the YouTube player says it is casting but it isn’t, so I have to manually stop casting and redo the cast. Worse, sometimes the YouTube Channel crashes and I am knocked back to the Roku’s main home page.

On Windows, my YouTube player did not show the cast icon. The problem was caused by the new YouTube layout. To fix it, I had to go to back to the old layout by clicking on my YouTube user icon and selecting the “Restore old YouTube” option. The old layout correctly showed the cast icon. Strangely, when I switched back to the new layout (by browsing to YouTube.com/new), the new layout now showed the cast icon. I saw this behavior using the Chrome browser, so other browsers may not have this issue.

Plex Stream From Desktop

If you have kids, you might prefer to avoid swapping movie DVDs by ripping them all to video files (I recommend using the open source HandBrake utility). Putting the files on a portable USB hard drive and connecting it to the Roku will give you a library of entertainment to choose from.

In my case, I didn’t have a portable USB hard drive, so I decided to just stream the movies from my desktop. The Plex Media Server was the free and simple DIY solution. I downloaded it, install it, launch it, created an account, added a Movies library, and selected the folder where my movie files were. The Plex Media Server scanned the folder (and its sub-folders), added the movie files, and downloaded nice preview images and summaries for each movie. (If you add and remove movie files, you can tell Plex Media Server to rescan by clicking on the options “…” to the right of the Movies library and selecting the “Scan Library Files” command.)

The Plex Media Server runs as a local web server which your browser can connect to. The Plex Media Server’s user interface will appear in the browser when you double-click on the Plex icon in the Windows’ system tray or the Mac’s status menu. In addition, the local Plex Media Server integrates with the Plex website online to allow you to access your videos from anywhere. I didn’t see a need for global access and thought it was a security risk, so I disabled the “Remote Access” option under Settings.

On the Roku, I added the Plex Channel, and signed in. The Roku displayed a 4 character alphanumeric code and asked me to input it into the Plex Media Server’s Link Account page at plex.tv/link. Once that was accomplished, the Plex Channel listed my movies.

Note: When playing .mkv movie files, the Plex Channel may crash randomly. Plex Channel is solid when playing .mp4 or .avi files.

Plex supports subtitle files, such as .srt files. To see the subtitles on your TV, you will need to make two changes:

  • Configure the movie to use subtitles. On the Plex Media Server interface, click on the movie to see its details, and select the subtitle file to use.
  • Configure the Roku to display subtitles. On the Roku, go to main menu Settings, Captions, Captions mode, and select “On always” (the other choices are Off and “On replay”). This is a global setting affecting all movies. Unfortunately, there is no easy way to turn subtitles on and off while the movie is playing.

Strangely, the Plex Channel interface provides an option to delete the movie. If you select the delete command, the actual movie file on the desktop is deleted (the corresponding subtitle file is not deleted). Thankfully, if you delete by mistake, you can find the movie file in the the desktop’s Trash folder.

Automobile Roku

I thought about setting up the Roku in my sister’s minivan. Instead of dragging all the movie DVDs around and swapping them in and out, my sister could use the Roku and a portable USB hard drive. I could connect the Roku to the car’s composite AV input and power it from the provided outlet.

Unfortunately, I found that the Roku took 45-60 seconds to boot up and about 5-6 button clicks to play a movie file. When you have kids, you can’t afford to take a minute to load the movie. You just want to pop in the DVD, hit play, and start driving. Better yet, start driving and the DVD will automatically continue playing where it left off. The Roku did not have such a hands-off continue playing feature. So it was a no go.

No Comments

Securing WordPress

Linux No Comments

Recently, I received a couple of “Password Reset” request emails from my blog. I surmised that someone had tried to log into my WordPress administrative account, was unsuccessfully, and had resorted to using the “Lost your password?” link on the login page. Disconcertingly, the email indicated that my admin username was being used. I checked my blog and thankfully, it looked like nothing was wrong.

It was time to make my WordPress installation more secured.

Disable Password Reset

I decided to remove the “Lost your password?” link on the WordPress login page. I doubted that I would forget my admin password and even if I did, I could always go directly to the MySQL database to change it. Because I didn’t need to support more than one user (my admin user), I decided to use the simpler manual method from How to Remove the Password Reset / Change option from WordPress, instead of the Plainview Protect Passwords plugin.

Here are the steps to globally disable the password reset option:

  1. Create a file named “disable-password-reset.php” with the following content:
    <?php
    /*
     * Plugin Name: Disable Password Reset
     * Description: Disable password reset functionality. Only users with administrator role will be able to change passwords from inside admin area.
     * Version: 1.0
     * Author: WPBeginner
     * Author URI: http://wpbeginner.com
     */

     
    class Password_Reset_Removed
    {
     
      function __construct()
      {
        add_filter( 'show_password_fields', array( $this, 'disable' ) );
        add_filter( 'allow_password_reset', array( $this, 'disable' ) );
        add_filter( 'gettext',              array( $this, 'remove' ) );
      }
     
      function disable()
      {
        if ( is_admin() ) {
          $userdata = wp_get_current_user();
          $user = new WP_User($userdata->ID);
          if ( !empty( $user->roles ) && is_array( $user->roles ) && $user->roles[0] == 'administrator' )
            return true;
        }
        return false;
      }
     
      function remove($text)
      {
        return str_replace( array('Lost your password?', 'Lost your password'), '', trim($text, '?') );
      }
    }
     
    $pass_reset_removed = new Password_Reset_Removed();
    ?>
  2. Upload this file to your server’s WordPress plugins directory (“/var/www/wordpress/wp-content/plugins”).
  3. Log into WordPress, go to Plugins, and activate the “Disable Password Reset” plugin. The password reset option will now be disabled for all users, including administrators.

As a precaution, I also changed the admin username because it had been compromised. This was accomplished by creating a new admin user in WordPress, logging into the new user, and then deleting the old user.

Force Login to Require Secure HTTPS

When I configured my VPS (Virtual Private Server), I allowed the WordPress login page and administrative areas to be accessible by unsecured HTTP. So anyone sniffing data packets on the Internet could see, in plain text, the info my browser was sending to my server. This may explain how my admin username was known to whomever was attempting to access my WordPress site.

To prevent the above, I decided to force the WordPress login and administration pages to require secure HTTPS access. I found the simplest method at WordPress SSL Settings and How to Resolve Mixed Content Warnings, which was to enable the “FORCE_SSL_ADMIN” WordPress option.

Edit the WordPress configuration file (“/var/www/wordpress/wp-config.php”) and add the text below before the “/* That’s all, stop editing! Happy blogging. */” statement at the bottom of the file:

/**
 * Secure (force HTTPS) entire wp-admin area
 * Note: Includes FORCE_SSL_LOGIN which secures login.php script
 */

define('FORCE_SSL_ADMIN', true);

/* That's all, stop editing! Happy blogging. */

After the change above, even if I were to access the login page using unsecured HTTP, it would automatically redirect me to the secure HTTPS login page.

Your Connection Is Not Fully Secure

Normally, when accessing a website using secure HTTPS, the browser would display a padlock next to the URL to indicate that the connection is secure. (Chrome shows a yellow padlock before the URL. Internet Explorer shows a grey padlock after the URL.) However, when I accessed my WordPress site using secure HTTPS, the padlock was not shown. Under Chrome, when I clicked on the info icon next to the URL, a message “Your connection to this site is not fully secure” was displayed.

I tested my site using Why No Padlock? and it found two issues:

  1. Server supports SSLv3, may be vulnerable to POODLE attack. It is suggested to disable the SSLv3 protocol.
  2. Number of insecure items: 15. (A.k.a. the mixed content issue. All 15 items were unsecured HTTP image links.)

The first issue can be resolved by disabling the SSL v3 protocol:

  1. Edit the Nginx server block file (“/etc/nginx/sites-available/default”). Locate and add/modify the “ssl_protocols” directive like below (the first line is commented out):
    #       ssl_protocols SSLv3 TLSv1;
            ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
  2. Restart the Nginx service so the changes above will take effect:
    sudo service nginx restart

Note: When I restarted Nginx, I saw warnings in the Nginx error log (“/var/log/nginx/error.log”) that looked like the following:

2017/09/02 01:15:24 [warn] 20631#0: duplicate value "TLSv1.2" in /etc/nginx/sites-enabled/mydomain:128
2017/09/02 01:15:24 [warn] 20631#0: duplicate value "TLSv1.1" in /etc/nginx/sites-enabled/mydomain:128
2017/09/02 01:15:24 [warn] 20631#0: duplicate value "TLSv1" in /etc/nginx/sites-enabled/mydomain:128

It might be caused by adding the “ssl_protocols” directive to more than one Nginx server block file. The warnings didn’t seem to affect the server adversely, so I ignored them.

If you have more than one domain (resulting in more than one Nginx server block file), you can add the “ssl_protocols” directive to the Nginx configuration file (“/etc/nginx/nginx.conf”), instead of to all the server block files. Directives in the Nginx configuration file will apply to all server block files.

The second issue, “insecure items” or mixed content, was caused by unsecured image link URLs in each blog post’s content. Fixing it would require editing all image links to use the secured URL instead. I plan to replace the unsecured image link URLs with secured URLs over time. Eventually, I hope to see the padlock on secured HTTPS access to my WordPress site. (By default, WordPress inserts an image link using an unsecured URL. So you will need to edit the pasted image link manually to use the secured URL.)

Note: I was concerned that browsers would not cache images downloaded by secure HTTPS. However, after some research, I found out that modern browsers (as of 2010) will cache HTTPS content by default.

WordPress Debug Mode

If you haven’t already done so, you might consider temporarily enabling the WordPress debug mode to see if there are issues with the theme or plugins. Setting debug mode will return a wealth of log messages (all errors, notices, and warnings) that might point out potential issues with your WordPress installation.

To set debug mode, edit the WordPress configuration file (“/var/www/wordpress/wp-config.php”) and add the following directives before the “/* That’s all, stop editing! Happy blogging. */” statement:

// Enable WP_DEBUG mode
define( 'WP_DEBUG', true );

// Enable Debug logging to the /wp-content/debug.log file
define( 'WP_DEBUG_LOG', true );

// Disable display of errors and warnings inside the page HTML
define( 'WP_DEBUG_DISPLAY', false );
@ini_set( 'display_errors', 0 );

// Use dev versions of core JS and CSS files (only if modifying them)
define( 'SCRIPT_DEBUG', true );

/* That's all, stop editing! Happy blogging. */

Once you are satisfied, don’t forget to disable the debug mode because it may adversely affect the performance of your WordPress site.

// Enable WP_DEBUG mode
define( 'WP_DEBUG', false );

I found only notices and PHP strict warnings concerning some plugins, but thankfully, no errors.

No Comments

How to Sell a Used Computer Like a Car Salesman

Hardware 2 Comments

A normal Craigslist transaction is quick. You establish your interest and confirm the price, place, and time to meet. You meet. The seller hands you the item. If it is electronics, you take a few minutes to test out the functions. You hand over the cash. Shake hands and then leave. 5-10 minutes maximum.

In early May, my sister asked me to buy a desktop computer for her. Her budget was $100, which is
totally doable for a used Core i5 desktop.

I found one, a Dell Optiplex 790 with Core i5, on craigslist advertised for $115. I negotiated it down to $100 by text messages. The ad didn’t say anything about a business so I figured it was a private seller.

The address I was given was mapped to the All Green Recycling Center in Tustin, CA. Some sellers do arrange meetings at their work location so I thought nothing of it. When I showed up, it turned out that the business was selling it.

The sales person who met me in the unoccupied reception area was Maria. She told me that someone showed up 5 minutes before me and purchased the computer I wanted. However, they did have another computer with the same specs, but with a 1TB hard drive instead of a 500GB one. The price was $150.

I told her that $150 was quite a stretch from $100 just for a bigger used hard drive. She said she would talk to her manager and disappeared behind the interior office door. 10 minutes later, she appeared. This initiated the trend where she would claim to need to do something and then disappear for 10-15 minutes at a time. Similar to how car salesmen would disappear for long stretches of time in order to wear the customer down.

Once Maria came back, she said that the lowest price they could do was $130… she paused for a few seconds and then said $120.

I’m not sure what that was about. Was she trying to anchor me at $130 so $120 would sound like a great deal? Or did her manager tell her to start at $130 and only go down to $120 at the lowest? Was she my best friend or my worst enemy?

All I heard was $120. I told her I didn’t really want the bigger hard drive, but $120 was doable if they would throw in the used wireless card that she had indicated were available in the text messages. She said she would ask her manager and disappeared once more.

10 minutes later, Maria appeared and said that they didn’t have a wireless card. I asked her if they could just change the hard drive to a smaller one. She replied that they didn’t do customization, so changing the hard drive was impossible. That the lowest price was $120 and that was it.

I told her, okay, let’s do it. She disappeared for 15 minutes and came back with the desktop. Then she said the price was $131 with tax. I gave her my credit card and ID and then she disappeared again. 10 minutes passed. I thought, how long does it take to charge a credit card and print a receipt? Was she 3D printing a copy of my credit card?

I had time to think and it occurred to me that I was overpaying. $131 was 30% above my sister’s $100 budget. I could have purchased the same desktop off of ebay for $100-$110 with free shipping, and not have had to deal with this very slow, horrible bait-n-switch experience.

I opened the interior office door, flagged down an employee, and asked him to tell Maria to abort the transaction. We didn’t want to buy after all.

I think Maria’s manager overheard because he came out of his office and asked what was going on. I told him that we had agreed by text on the price of $100, but that when I came, they didn’t have the computer I wanted. Instead they had another computer which was more expensive. And now, I decided not to do the deal after all. The manager was concerned and asked me to confirm twice that an agreement was reached before I showed up. He asked me to wait while he went to find Maria.

Within a couple of minutes, Maria appeared and said that they would swap out the 1TB hard drive for a 500GB drive and charge the originally agreed upon price of $100. Looks like you do do customization, I thought. She gave me my credit card and ID back. She promised it would only take a few minutes to swap the drive (which I doubted) and then took the desktop back in.

By this time, I had been there for almost an hour. I think the slow, horrible service is Maria. I don’t know what her deal is but she seems very incompetent at her job. Awfully slow for no apparent reason that I could see. The other employees seemed more friendly and competent because when they saw me waiter, they all immediately asked if I was being helped.

So I waited 15 minutes and still no Maria. Surely it wouldn’t take more than 5 minutes to swap a hard drive; were they also re-installing Windows 10 or something? Finally, another employee came and asked if I was being helped. I told him Maria was helping me, that I had come to buy a computer, and that it had been more than an hour. He looked surprised and said he would find Maria to see what was going on.

A couple of minutes later, Maria appeared with the desktop. Was she just hiding somewhere with the desktop and only came because a coworker found her? Like a golem.

She asked if I got an invoice? Ugh, I told her that I didn’t. I asked her if she had charged my card? She said no. I wanted to ask why she gave me back my credit card and than waited until after the hard drive was swapped before asking if she had given me an invoice. What if I had said yes and left without paying? At this point, I decided that she was not all together there. I asked her if she needed my credit card and ID, she said yes, and I gave them to her.

She left. I waited 10 minutes. Still no sign of Maria. Again, how long does it take to charge and print an invoice? Evidently for Maria, longer than 10 minutes. Finally, she appeared with the invoice.
Thankfully, the torture was over.

The business did not offer an option to boot up the desktop. However, they did offer a 90 day warranty. When I got to my sister’s, I tested the desktop. It booted up to Windows 10 fine. The only problem was the DVD drive didn’t open. Sighed. My sister said she would take it back for a drive replacement or repair. I warned her to avoid Maria; anyone else would be preferable.

That was a horrible experience. My subsequent craigslist transactions have involved filtering out all businesses. Even then, I needed to double-check because some businesses masqueraded as private sellers.

If you want to find out how buying a used computer can be as horrible as buying a used car, look up Maria.

2 Comments

Windows Explorer Has Stopped Working

Windows No Comments

My brother-in-law has a problem with his Windows 7 laptop. He continually sees the pop-up error message, “Windows Explorer has stopped working”, and the desktop would flicker (killing all file explorer windows) as Windows restarts the process. He cannot do anything productive with the malfunctioning machine. He asks me to take a look.

When diagnosing and fixing a misbehaving Windows system, I do the following:

  1. Clean virus or spyware (aka malware) infection. Virus and spyware could cause Windows to act strangely by damaging critical system files.
  2. Prevent strange programs from running on startup.
  3. Fix file system issues and check Windows file system integrity.
  4. Disable and delete unknown browser plugins. Reset the homepage to a blank tab.
  5. Resolve Windows registry inconsistencies.
  6. Pull the latest Windows updates.

Note: Most of the instructions below are applicable to Windows 8 and Windows 10 with some minor differences.

Death to Viruses and Spyware

Before running the virus and spyware scans, I recommend deleting temporary files to speed up the whole process by reducing the number of files to scan. Launch “Disk Cleanup”, select the primary drive, click on the “Clean up system files” option, select the primary drive again (if prompted to), check all temporary files found, and delete them.

Note: Under the Disk Cleanup’s “More Options” tab, you can delete “System Restore and Shadow Copies”. I don’t recommend deleting System Restore images because that will remove the ability to restore Windows to an earlier point in time. Only do so if your Windows computer is working without any problems and you really want to reduce the virus scan time. (Virus scanners doing a full, not quick, scan will take significantly more time to scan through the System Restore files.)

I run Microsoft Security Essentials (known as Windows Defender in most versions of Windows), update it, and start a quick scan. Updating it successfully is a good sign because some viruses will cripple virus scanners pre-emptively. Security Essentials find no infection. (If you don’t already have a virus scanner installed, I recommend downloading the free Windows Security Essentials.)

I then launch Malwarebytes Anti-Malware, update it, and initiate the scan. Malwarebytes finds and cleans several malware infections. I don’t know whether those infections are serious or not; I’m just glad they are gone. The worst is knowing that even if you successfully clean a virus or spyware infection, it may not solve all the problems because they tend to leave damage behind.

Note: Malwarebytes Anti-Malware is free and requires that you manually run it. The paid premium version provides real-time, constant surveillance.

Because I want to make certain that the laptop is clean, I also run ComboFix. ComboFix is a powerful spyware scanner which I’ve used successfully in the past with Windows XP. ComboFix should only be used at your own risk, because it could potentially damage Windows further. If you decide to use ComboFix, download it from bleepingcomputer.com, not from combofix.com or combofix.org. The latter will ironically give you a spyware-infected ComboFix version!

Thankfully, ComboFix completes running and does not destroy Windows 7. Unfortunately, the ComboFix log file is cryptic so while I’m fairly certain that it fixed something, I’m not exactly sure what.

The good news is that after a reboot, Windows no longer displays the “Windows Explorer has stopped working” popup message. So it looks very likely that spyware or malware caused the initial issue.

Say No to Startup Programs

Note: The System Configuration tool (msconfig) was removed from Windows 10. Its functions are incorporated into the Task Manager’s Startup and Services tabs.

Execute “msconfig” on Windows 7 to run the System Configuration tool and look for the Startup tab, which will list the programs launched at boot time. Google any program you don’t recognize. You may see remnant programs, with non-sensical names like “BDsad32Zm”, left by the virus or spyware — just uncheck them. (To reduce the startup time, I recommend unchecking any unnecessary programs like QuickTime and Adobe Reader. They supposedly speed up the launching of these programs, but at the cost of increasing startup time.)

Note: To reduce the startup time further, you can look at the Services tab and uncheck any unnecessary services, like “Distributed Link Tracking Client” (useful only if you link shared files across the network). Alternatively, instead of using “msconfig”, you could launch “services.msc” and disable the service or modify it to start manually, instead of automatically. Windows will start a manual service if necessary; for example, if an automatic service depends on it.

I do not find any strange startup programs on my brother-in-law’s laptop. Malwarebytes or ComboFix may have gotten rid of them already.

System Integrity Or Else

Corrupted files on the hard drive may cause Windows or programs to behave strangely. Thankfully, Windows provides two tools to diagnose this issue: a Check Disk tool (chkdsk) to fix general file system problems and a System File Checker (sfc) to verify Windows system files. (For more info on the System File Checker, see Use the System File Checker tool to repair missing or corrupted system files.)

To use the Check Disk tool:

  1. Click on the Windows start menu icon, input “cmd”, right-click on the “cmd.exe” or “Command Prompt” result, and select “Run as administrator” to launch the command prompt window with administrative privileges.
  2. Execute the following command (first line without the “>”):
    > chkdsk /f

    The type of the file system is NTFS.
    Cannot lock current drive.

    Chkdsk cannot run because the volume is in use by another
    process.  Would you like to schedule this volume to be
    checked the next time the system restarts? (Y/N)
  3. Input “Y” for Yes and reboot the laptop. The startup process will scan the hard drive for errors before running Windows.

Note: If you have a second hard drive, you can check it without rebooting; for example, by running the command “chkdsk /f d:” if you have a “D:\” drive.

To use the System File Checker:

  1. Run a “Command Prompt” as administrator (same steps as above).
  2. Execute this command:
    > sfc /scannow

    Beginning system scan.  This process will take some time.

    Beginning verification phase of system scan.
    Verification 100% complete.
    Windows Resource Protection found corrupt files but was unable to fix some of them.
    Details are included in the CBS.Log windir\Logs\CBS\CBS.log. For example
    C:\Windows\Logs\CBS\CBS.log

The Check Disk tool finds some file inconsistencies and fixes them. Unfortunately, though the System File Checker finds issues, it is not able to repair them all. This means that my brother-in-law’s laptop has some Windows system file corruption, which is bad.

The “CBS.log” file is large and dense. Microsoft recommends filtering it by running this command:

findstr /c:"[SR]" %windir%\Logs\CBS\CBS.log >"%userprofile%\Desktop\sfcdetails.txt"

The resulting “sfcdetails.txt” is not too helpful; it doesn’t show the filenames. Here is an excerpt:

2017-05-08 15:15:49, Info                  CSI    00000578 [SR] Verifying 100 (0x00000064) components
2017-05-08 15:15:49, Info                  CSI    00000579 [SR] Beginning Verify and Repair transaction
2017-05-08 15:15:49, Info                  CSI    0000057b [SR] Verify complete
2017-05-08 15:15:49, Info                  CSI    0000057c [SR] Verifying 100 (0x00000064) components
2017-05-08 15:15:49, Info                  CSI    0000057d [SR] Beginning Verify and Repair transaction
2017-05-08 15:15:49, Info                  CSI    00000580 [SR] Cannot verify component files for a3ba03adb219630fa0874057b9609115, Version = 6.1.7601.23418, pA = PROCESSOR_ARCHITECTURE_INTEL (0), Culture neutral, VersionScope = 1 nonSxS, PublicKeyToken = {l:8 b:31bf3856ad364e35}, Type neutral, TypeName neutral, PublicKey neutral, manifest is damaged (TRUE)
2017-05-08 15:15:49, Info                  CSI    00000582 [SR] Verify complete

Note: Unfortunately, I had used the Disk Cleanup tool to delete the System Restore images earlier. As a result, I am unable to revert to earlier Windows 7 versions which might not have had the system file corruption. I’m not sure, but the System File Checker might have been able to use files from the System Restore images to correct the issues above. Duh.

I’m hopeful that the Windows Update, which we will do later, will fix these corrupted files.

System File Checker To The Max

You may consider running the System File Checker in Windows safe mode. On boot up, hold the F8 key until you see the Advanced Boot Options menu and then select the “Safe Mode with Command Prompt” option. Doing so may allow the System File Checker to repair files that might be in use during a normal boot up.

Additionally, before running System File Checker, you may wish to ensure that temporary files belonging to it (under “PendingRenames” and “PendingDeletes”) are deleted. Those files are protected so you’ll also need to take ownership before you can delete them. Run the Command Prompt as administrator and issue these commands:

cd %windir%\winsxs\Temp\PendingRenames

# Take ownership of files
takeown /f *.*

# Grant file permissions to administrators (assuming you are one)
icacls *.* /GRANT ADMINISTRATORS:F

# Delete the files
del *.*

# Repeat to delete files under PendingDeletes (which is a hidden directory)
cd %windir%\winsxs\Temp\PendingRenames

Note: The Advanced Boot Options menu also has a “Repair Your Computer” item; however, when I select it, I get an error, “The boot selection failed because a required device is inaccessible”. I try inserting both a Windows 7 installation DVD and USB flash drive, but they are not accepted. I think this option depends on having a special repair partition, which is missing from the laptop.

Browser Plugins Be Gone

As a general matter of computer hygiene, I check both the Internet Explorer and Chrome browsers on the laptop. Specifically, I am looking for plugins, extensions, or add-ons that shouldn’t be there. When I find them, I disable and delete/uninstall them.

I also double-check that the default search provider and homepage have not been overwritten. For example, the unwanted SmartSearch plugin loves to set the search provider to Yahoo and the homepage location to its own search website.

I do not locate any unwanted browser plugins on the laptop.

Windows Registry Be Consistent

I install the free CCleaner tool, select the Registry tool, scan for issues, and fix them all. I recommend choosing the option to backup the registry before making changes. That way if it goes horribly wrong (but Windows still works), you’ll have a way to undo the action. Always reboot afterwards to check that the registry changes are okay.

The Windows registry serves as Window’s memory bank. Inconsistencies in it will cause Windows to misbehave. Corruption in it could cause Windows to stop running. As with ComboFix, use CCleaner at your own risk.

Note: CCleaner also offers a tool to find and delete temporary and unnecessary files. I usually use it in addition to the Window’s “Disk Cleanup” tool.

To The Latest And Greatest

I launch “Windows Update”, see that some updates are pending, and start their installation. Then I wait and wait. The progress bar shows zero progress. When I hover the mouse over the Windows Update icon in the system tray, the tooltip message “Windows is downloading updates (0% complete)” keeps appearing. I give up after 30 minutes.

After some research, I find that Microsoft had significantly changed how Windows Update worked on October 11, 2016. If the Windows Update is unable to update itself, then the “0% complete” issue could occur. I download and install the KB3172605 package according to Windows 7 Update solution.

I re-run the Windows Update and this time, the updates download and install successfully, except for one. The single failed update is the “2017-05 Security Monthly Quality Rollup for Windows 7 for x86-based System (KB4019264)” package. The error code is 80073712, which means that the Windows component store is corrupt.

Unfortunately, Windows Update couldn’t repair the corrupt Windows system files.

Ready To Repair

I find suggestions that the standalone System Update Readiness tool could be used to repair corrupt Windows system files. I download the version for 32-bit Windows 7 and run it. It is able to repair some files, but not all.

Note: Windows 7 comes with a built-in DISM (Deployment Image Servicing and Management) tool which contains some of the System Update Readiness tool’s functions. Though the Windows 7 DISM is not as powerful as the Windows 8 or 10 version, you can run it with “DISM /Online /Cleanup-Image /Scanhealth” (more powerful parameters like “/Restorehealth” will not work under Windows 7). I still decid to use the standalone System Update Readiness tool instead.

Thankfully, the System Update Readiness tool’s log file, “%windir%\Logs\CheckSUR.log”, is very readable and lists the un-repairable files at the end.

Unavailable repair files:
    winsxs\manifests\x86_a3ba03adb219630fa0874057b9609115_31bf3856ad364e35_6.1.7601.23418_none_c3113e25b89565d3.manifest
    winsxs\manifests\x86_microsoft-windows-t..platform-comruntime_31bf3856ad364e35_6.1.7601.23375_none_cce17c0fae9f1772.manifest
    winsxs\manifests\x86_microsoft-windows-ie-ieetwcollector_31bf3856ad364e35_11.2.9600.17501_none_4942df405f80bb4b.manifest
    winsxs\manifests\x86_microsoft-windows-t..icesframework-msctf_31bf3856ad364e35_6.1.7601.23572_none_7811a73e66577d81.manifest
    servicing\packages\Package_37_for_KB3138612~31bf3856ad364e35~x86~~6.1.1.1.mum
    servicing\packages\Package_7_for_KB3138612~31bf3856ad364e35~x86~~6.1.1.1.mum
    servicing\packages\Package_for_KB3138612_SP1~31bf3856ad364e35~x86~~6.1.1.1.mum
    servicing\packages\Package_for_KB3156017~31bf3856ad364e35~x86~~6.1.1.0.mum
    servicing\packages\Package_37_for_KB3138612~31bf3856ad364e35~x86~~6.1.1.1.cat
    servicing\packages\Package_for_KB3138612_SP1~31bf3856ad364e35~x86~~6.1.1.1.cat
    servicing\packages\Package_for_KB3156017~31bf3856ad364e35~x86~~6.1.1.0.cat

Following instructions from How to fix errors found in the CheckSUR.log, I download the packages for KB3138612 (Windows6.1-KB3138612-x86.msu) and KB3156017 (Windows6.1-KB3156017-x86.msu), and place them in the newly-created “%WinDir%\Temp\CheckSUR\Packages” folder.

I re-run the System Update Readiness tool. It repairs the corrupted files belonging to those packages. The resulting log now only shows the corrupted manifest files:

Unavailable repair files:
    winsxs\manifests\x86_a3ba03adb219630fa0874057b9609115_31bf3856ad364e35_6.1.7601.23418_none_c3113e25b89565d3.manifest
    winsxs\manifests\x86_microsoft-windows-t..platform-comruntime_31bf3856ad364e35_6.1.7601.23375_none_cce17c0fae9f1772.manifest
    winsxs\manifests\x86_microsoft-windows-ie-ieetwcollector_31bf3856ad364e35_11.2.9600.17501_none_4942df405f80bb4b.manifest
    winsxs\manifests\x86_microsoft-windows-t..icesframework-msctf_31bf3856ad364e35_6.1.7601.23572_none_7811a73e66577d81.manifest

Unfortunately, I am unable to find a source for the manifest files online or otherwise. My friend has a Windows 7 desktop, but his installation is 64-bit and I need the 32-bit versions.

Upgrade To Windows 7 For The Win

When the System File Checker failed, I was afraid I would have to do it. I fought it, but in the end, it looks like I need to do an in-place upgrade of Windows 7. Effectively, re-install Windows 7 on top of itself. This “upgrade” should replace the corrupted files, while preserving everything else.

Note: I thought that I might avoid re-installing Windows 7 by just re-installing the Service Pack 1 (in the hope that it would be sufficient to replace the corrupted files). Unfortunately, when I attempt to install the Service Pack 1, it fails with a cryptic zero file termination error.

To perform an in-place upgrade, do the following:

  1. Make sure you have the Windows 7 product key!
  2. Run Windows 7 as normal. Log in with an administrator account.
  3. Disable any virus or spyware scanner. To disable Microsoft Security Essentials, go to Settings, Real-time protection, uncheck the “Turn on real-time protection (recommended)” option, and click on the “Save changes” button.
  4. Insert a bootable USB flash drive containing the Windows 7 Professional with SP1 32-bit installer (see instructions on how to create one). Or if you prefer, insert the Windows 7 install DVD.
  5. Run the “setup.exe” on the USB flash drive (or DVD) and click on the “Install now” button.
  6. Select the “Go online to get the latest updates for installation (recommended)” option. (If you don’t have Internet access, you’ll need to choose the second option, “Do not get the latest updates for installation”.)
  7. Accept the license terms and select the “Upgrade” option.
  8. Sit back, relax, and wait. My brother-in-law’s laptop takes almost an hour to do the in-place upgrade. I observed the following: 20 minutes, reboot, 20 minutes, reboot, 5 minutes, reboot, chkdsk, and reboot.
  9. After the final reboot, you will be prompted to enter the Windows 7 product key. Input the product key.
  10. Select the recommended defaults in the following two screens. You’ll then see the normal Windows 7 login screen.
  11. Log in, activate the Windows 7 product key, and re-enable the virus scanner.

After the “upgrade”, I run “Windows Update”. After a couple of minutes, it says “133 important updates are available”. I was afraid of that — looks like I need to install all the updates since Service Pack 1. Two hours later (after 133 updates, reboot, 59 updates, reboot, 3 updates, reboot, 1 update, and reboot), all the updates are installed successfully.

Back to the Past

Over the next hour, two more updates are found and installed. Then an “Internet Explorer 11 for Windows 7” update appears which fails to install. What?!

I run the Check Disk (chkdsk) tool. Unexpectedly, it finds some file index errors and correct them all. I run the System Update Readiness tool. It finds some issues and fixes them all. I run the System File Checker (sfc). It finds problems and complains “Windows Resource Protection found corrupt files but was unable to fix some of them”. Uh oh, it’s the same problem.

I filter the System File Checker’s log file and one log statement says that the “mvc80JPN.dll” file is corrupt. That file belongs to Microsoft Visual Studio 2005. I download and install the Microsoft Visual Studio 2005 runtime, but it does not correct the file. I am not able to find another source online. I am stuck at the same dead end.

Having no choice, I hide the “Internet Explorer 11” update. The “Internet Explorer 10” update appears, it fails to install, and I hide it also. Looks like all Internet Explorer versions depend on the corrupt “mvc80JPN.dll” file.

Over the next day, two dozen updates show up, including the “2017-05 Security Monthly Quality Rollup for Windows 7 for x86-based System (KB4019264)” update. All of them download and install successfully. Eventually, no new updates appear.

For now, my brother-in-law can live without the latest Windows Explorer version. He can use the existing Internet Explorer 8 or the latest Chrome browser. If he uses Internet Explorer 8, he should be okay if he avoids Japanese websites; he doesn’t read Japanese so I don’t think that it will be a problem.

Note: If and when my brother-in-law next encounters a major issue with his Windows 7 laptop, I will probably just do a fresh installation of Windows 10 on it.

And we are done. I hope the above will help you to solve your “Windows Explorer has stopped working” error or other Windows problems.

No Comments

Install macOS Sierra Using Bootable USB Flash Drive

Mac OS X No Comments

To perform a clean installation of macOS Sierra (basically, Mac OS X 10.12), I recommend using a bootable USB flash drive containing the macOS Sierra installer. Below are the steps I took to create the bootable USB flash drive and how I used it to install macOS Sierra.

Note: The macOS Sierra Disk Utility and installer appears to be more buggy and much slower than previous versions. The best advice for installing macOS Sierra is to try again and be very patient (if you expect an operation to complete in 5 minutes, then give it at least 50 minutes).

Download macOS Sierra Installer

The macOS Sierra installer is available from the Mac App Store. Run the “App Store” application, search for “macOS Sierra”, and download it. It will save the installer as an “/Applications/Install macOS Sierra.app” file (about 4.97GB in size).

Note: If you run the macOS Sierra installer to upgrade your Mac, the downloaded file will be deleted automatically after the upgrade is completed. To keep that file, you will want to move it out of the Applications folder so it won’t be deleted after an upgrade. Launch the “Terminal” application and run this command to move the downloaded installer to your user’s “Downloads” folder:

sudo mv /Applications/Install\ macOS\ Sierra.app/ ~/Downloads/

If you are paranoid (doesn’t hurt), you can verify that the installer file was downloaded correctly by verify its checksum. Run the “Terminal” application and this command:

hdiutil verify /Applications/Install\ macOS\ Sierra.app/Contents/SharedSupport/InstallESD.dmg

# If successful, the last output line should read:
# hdiutil: verify: checksum of "/Applications/Install macOS Sierra.app/Contents/SharedSupport/InstallESD.dmg" is VALID

Format USB Flash Drive

The macOS Sierra installer takes up 5.1GB of space on the USB flash drive, so you will need a flash drive with a capacity of 8GB or greater.

Note: If the flash drive is mounted under “/Volumes” successfully when you plug it in, you can skip the following steps to reformat the flash drive. This is because the script we run to create the bootable drive will reformat the flash drive as an initial step. Because I am paranoid, I recommend reformatting the USB flash drive manually anyhow.

Format the USB flash drive using these steps:

  1. Plug in the USB flash drive to your Mac.
  2. Launch the “Disk Utility” application.
  3. On the left-hand pane, select the USB drive (not the partition under it, if any).
  4. Click on the “Erase” tab (or button at the top).
    1. Input a name like “Sierra” (this name will be overwritten later).
    2. Select “Mac OS Extended (Journaled)” for “Format”.
    3. Select “Master Boot Record” for “Scheme”.
    4. Click the “Erase…” button at the bottom. Click the “Erase” button in the warning popup dialog if you get one.
      • The format operation may take several minutes to complete. (USB 2.0 and large capacity drives will take longer.) After the format completes, the partition will be mounted under “/Volumes/Sierra” (or whatever name you selected above).
      • Note: Under macOS Sierra, the Erase function will fail if the USB drive’s partition is mounted. You can manually unmount the partition before running Erase. Or you can run Erase twice; the first time will unmount the partition and fail, and the second time will actually do the format (which will succeed).
  5. Close the “Disk Utility” application.

Create Bootable USB Flash Drive Installer

To create the bootable USB macOS Sierra installer, run the “Terminal” application and this command:

# The --volume value is the mounted USB flash drive partition; in this case, named /Volumes/Sierra

sudo /Applications/Install\ macOS\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Sierra --applicationpath /Applications/Install\ macOS\ Sierra.app --nointeraction

# You will be prompted for your user's administrative password.

The “createinstallmedia” program will erase the USB flash drive, create a new partition named “Install macOS Sierra”, and copy the installation files to that partition. The output will look like:

Erasing Disk: 0%... 10%... 20%... 30%...100%...
Copying installer files to disk...
Copy complete.
Making disk bootable...
Copying boot files...
Copy complete.
Done.

The program will pause at the “Copying installer files to disk…” output line above. This step took 20-30 minutes with my Kingston 16GB USB 2.0 flash drive. Yours may take a shorter or longer time. I recommend giving it at least an hour, maybe two, before giving up.

Note: Mac hardware is very finicky about USB flash drives. Initially, I used a Corsair 32GB USB 3.0 drive; however, when I held down the Option key to try to boot with it, the Mac would freeze with a black startup screen. The Kingston 16GB USB 2.0 drive did not have this problem. So if you enounter issues (when erasing and copying) or weirdness (when booting), consider changing to another brand of USB flash drive. If you don’t have another drive, consider at least testing the flash drive to make sure it is not bad or corrupted (“First Aid” in “Disk Utility” is the minimum; google for more powerful tools).

Boot With USB Flash Drive

Note: I recommending connecting the Mac to its AC power adapter before beginning the macOS Sierra installation. The installation may take a long time (an hour or more) and you don’t want the battery to die in the middle.

To boot a Mac with the USB flash drive:

  1. Shutdown the Mac.
  2. Insert the USB flash drive.
  3. While holding the “option/alt” key down, turn on the Mac to display the boot Startup Manager.
  4. You should see one or more icons, one of which should be called “Install macOS Sierra” for the USB flash drive. (The internal hard drive may not be visible if it does not have a valid, bootable partition installed.)
    • Note: If you don’t see the USB flash drive’s “Install macOS Sierra”, try removing and re-inserting the USB flash drive while viewing the Startup Manager screen. The USB flash drive should then appear after a few seconds.
  5. Select the “Install macOS Sierra” (with left/right arrow keys) and hit the “return/enter” key to boot from the USB flash drive.

It may take 5-10 minutes or longer to load the installer from the USB flash drive. Sometimes the progress bar may appear to be frozen… just be patient. I would give it at least 30-60 minutes to load before giving up.

Format the Hard Drive

When the installer finishes loading, you will see a “macOS Utilities” window appear. Do the following to format the internal hard drive:

  1. Click on the “Disk Utility” option and click the “Continue” button on the bottom to launch the “Disk Utility” application.
  2. On the left-hand pane, select the hard drive (not the partition under it, if any).
  3. Click on the “Erase” button at the top.
    1. Input a name like “macOS”.
    2. Select “Mac OS Extended (Journaled)” for “Format”.
    3. Select “GUID Partition Map” for “Scheme”.
    4. Click the “Erase…” button at the bottom.
      • For SSD (Solid State Drive), the format operation may take less than a minute to complete. For mechanical hard drive, it may several minutes to hours, depending upon the size, speed, and condition of your hard drive.
      • Note: Again, the Erase function will fail if the hard drive’s partition is mounted. You can manually unmount the partition before running Erase. Or you can run Erase twice; the first time will unmount the partition and fail, and the second time will actually do the format (which will succeed).
  4. Close the “Disk Utility” application.

Note: Now and then, I noticed the output of the Erase seems to erroneously double the size of the hard drive. For a 128GB hard drive, the graph shows 120.88GB macOS (in blue) and 120.37 GB Unformatted (in red). I think it is just a user interface bug because when I close Disk Utility and re-open it, the graph then only shows the 120.88GB macOS (in blue).

Install macOS Sierra

Back at the “macOS Utilities” window, do the following to begin the macOS Sierra installation process:

  1. Click on the “Install macOS” option and click the “Continue” button.
  2. The “macOS Sierra” installer’s splash screen will appear. Click the “Continue” button.
  3. Click on the “Agree” button to agree to the license. A popup confirmation window will appear; click on the popup’s “Agree” button.
  4. Select the hard drive and click the “Install” button.

Note: You may encounter strange hardware behavior. On my 13 inch Macbook Pro Retina, the macOS Sierra installer turned the fan on to maximum for the whole duration of the installation. Thankfully, once it finished and rebooted, the fan turned off and stayed off.

The macOS Sierra installer tries to be helpful by telling you how long it will take. Unfortunately, it lies. You should take whatever remaining time it tells you and multiple by 10 (for minutes) or 100 (for seconds). If it says “6 minutes remaining”, that could mean 60 minutes or one hour remaining. Worse, if it says “6 seconds remaining”, you may be staring at that message for 600 seconds or one hour.

The best solution is to be patient. Go grab a bite to eat and watch a movie. Take a long nap or better yet, sleep your 8 hours. I would wait at least 4 hours before giving up.

Note: You can display the installer’s log window (using the menu or pressing Cmd+L). I didn’t find this helpful at all. Even for a successful install, numerous errors are logged; I don’t know what is a critical or non-critical error. And often, you won’t see a progress/status log output for a long time, easily 20-30 minutes. Not seeing any new log statements does not mean that the installer froze. So the logs didn’t do anything for me.

What Does Giving Up Mean?

Giving up means you have accepted defeat. The next step is to retreat and try again. Some suggestions on how to proceed:

  • Reset your Mac by doing the following:
    1. Reset the SMC (see step 3 under the “Reset the SMC on Mac notebook computers” section).
    2. Reset the NVRAM (aka PRAM).
    3. Run the Apple Hardware Diagnostic or Test to make sure you don’t have a hardware failure.
    4. Finally, retry the macOS Sierra install.
  • Use a different USB port on the Mac.
  • Use another brand of USB flash drive.
  • Delete and re-download the macOS Sierra installer (especially if you downloaded it a long while ago). Even if the checksum is okay, you may want to re-download in case there is a newer version of the installer with a bug fix for your very problem.
  • Download an older Mac OS X version, say Mac OS X 10.11 El Capitan, install that, and then upgrade to macOS Sierra. If you know the Mac OS X version which came with your Mac originally, consider downloading (if you still have access) and installing that version first.
  • Use the Mac Recovery System to download and install the original OS version that came with your Mac. Then upgrade from that to macOS Sierra.
  • Buy a more recent model Mac (at most a couple of years old). It may be that your current Mac is too old or slow to support macOS Sierra. It’s okay to keep running an old Mac OS version. (For example, if I had a Core 2 Duo Mac, I would not run anything later than Mac OS X 10.9 Mavericks on it.)

Hopefully, this post will help you to do a fresh installation of macOS Sierra.

Some info above taken from:

No Comments

Outdoor Home Video Surveillance With CleverLoop

Hardware No Comments

candace_flynnMy Dad was concerned. He thought he heard noises from the backyard and maybe saw some movement there at night. My Dad grows fruit trees and vegetables, which might attract intruders. His neighborhood is not the safest place; his house had recently been burglarized. He wanted a way to check that the perimeter was safe before leaving the house. The obvious answer, besides moving away, was an outdoor video surveillance system.

After days of research, I settled on the CleverLoop Smart Home Security System. CleverLoop is expensive. I could get video systems (especially coaxial ones) with more cameras for significantly less. However, my primary concern is usability. My Dad is going to use it so it has to be very easy to use.

CleverLoop’s selling point is usability, implemented by their mobile user interface and smart backend (a base station device is included with the cameras). They admit that their cameras are the same generic ones made in China as the other video surveillance systems use. Hardware is not what CleverLoop is competing on, it is software. CleverLoop’s mobile phone app (for iPhone or Android) looked to be more user friendly and feature rich than the competitors’ versions.

Here is how CleverLoop did against my selection criteria, in prioritized order:

  1. User-friendly interface – CleverLoop mobile phone app looks easy to use
  2. Accessible from anywhere over Internet – Dad wanted to check the video feeds when he is away from the house
  3. Decent video quality – CleverLoop can support 720p HD (High Definition)
  4. Night vision mode – automatic IR (infrared) night vision
  5. Motion detection and alerts – CleverLoop’s base station handles this
  6. No monthly fee – free CleverLoop plan includes multiple users and backs up alerts to the cloud for 7 days
  7. Wireless – all CleverLoop cameras are wireless, but we ended up not using this function

When I was researching, CleverLoop was selling a kit containing 3 outdoor cameras. At the time, the base station only supported 3 cameras. This was a problem because a house is usually square shaped and I needed 4 cameras to cover the 4 sides. Thankfully, when I was ready to buy, CleverLoop offered a kit containing 4 outdoor cameras. A camera at each corner of the house looking down each side will cover the whole perimeter of the house, albeit with blind spots under each camera. I supposed that they had updated the base station software to support 4 cameras, instead of 3.

What to Buy

Before purchasing an outdoor video surveillance system, you must decide on the type of wiring because that is the hardest part of the installation. I had decided that the most future proof wiring is PoE (Power over Ethernet) Cat6 (Category 6) Ethernet cable. PoE reduces installation to running just one cable conveying both power and data. The Cat6 Ethernet cable will support higher data rates for when more powerful cameras become available. If I use a coaxial or non-PoE Ethernet cable, I would need to run a second cable for the power. Trust me on using a PoE cable, because you don’t want to have to drill a second extra hole through two or more plywood layers (common for exterior walls).

While CleverLoop does not support PoE, there is a way to simulate it. Basically, use adapters, passive PoE injectors and splitters, to feed both data (non-PoE Ethernet cable) and power through the “PoE” Ethernet cable. To protect the PoE adapters from the elements, you can use an outdoor electrical outlet cover box. This very helpful Youtube video, Installing CleverLoop Outdoor IP Camera, was my installation bible. CleverLoop has a support page, Powering an outdoor camera using Power-over-Ethernet (PoE), that references the video along with useful pictures of the PoE injector and splitter wiring.

In addition to ordering the CleverLoop Security System with 4 Outdoor Cameras, I purchased the following for the installation:

  • 500 ft of Cat6 Ethernet Cable (from Amazon). I think I used 100-150 feet at best.
  • 100 Cat6 RJ45 cable connectors (from Amazon). Maybe used a dozen.
  • RJ45 Crimp, Cut, and Strip Tool (from Amazon). Used to cut the Cat6 Ethernet cable and crimp the RJ45 connectors onto the cable ends.
  • 4 Passive PoE Injector and Splitter Kits with 5.5×2.1mm DC Connector (from Amazon). Double-check that the splitters and injectors have 5.5×2.1mm DC connectors because that is what CleverLoop cameras use.
  • 4 Weatherproof Outlet Covers (from Home Depot; Bell Outdoor Weatherproof In-Use Cover for $8.44 each). Get the 2 inches thick cover, not the 1 inch thick cover; you’ll need the extra space to hold the PoE splitter, Cat6 Ethernet cable, and CleverLoop Ethernet data and power cables.

cleverloop_outlet_coverI ordered way more Cat6 Ethernet cable and RJ45 connectors than I needed. You can safely order much less, depending upon the size of your house.

If I had to do the installation again, I would also purchase a Network Cable RJ45 Tester (like this one from Amazon). The tester can determine whether the RJ45 connectors are crimped onto the ends of the Ethernet cable correctly or not. Instead, I used an Internet-enabled router and my Macbook to test the cable, which was very inconvenient.

Besides the above, you will probably need a power drill with a long drill bit (one foot or longer). The external frame on my Dad’s house under the roof line (where we placed the cameras) consisted of two 2×6 planks with an inch spacing between. The long drill bit was necessary to punch an Ethernet cable-sized hole all the way through.

Drill, Baby, Drill

After everything arrived, I reserved a weekend to complete the installation. The idea was to run the Cat6 Ethernet cables in the attic and drill holes from the attic to the outside at each corner of the house. I didn’t look forward to working in the hot and dirty attic. Thankfully, my older brother volunteered to do the cabling and drilling.

It took my brother the better part of the first day to drill the necessary holes and to run the cables, though with a long break in the middle. He had to drive home to get his 1.5 feet long drill bit and disposable coveralls (for getting into tight, dirty corners filled with insulation) once he realized both were necessary.

I crimped the RJ45 connectors onto the ends of the Cat6 Ethernet cables and tested them; I only had to redo 5 connectors. When installing the RJ45 connectors, the sequence of color wires needs to be the same at both ends. I found this page, Making Ethernet Cables – Tricks of the Trade, helpful and followed the sequence of colors that he used to make a straight through Ethernet cable.

On the second day, my brother mounted the cameras and outlet covers (containing the PoE splitters, assorted Ethernet cables, and CleverLoop power cables). Because the drilled holes, cameras and outlet covers were located under the eaves, he did not bother to waterproof the drilled holes (the Ethernet cables fit the holes quite snuggly). He also didn’t bother arranging drip loops for all the cables coming to and leaving from the outlet covers.

In the attic, the Cat6 Ethernet cables from each camera met at a point above where my Dad’s Internet router was located. We tried to run a Cat6 Ethernet cable down the wall to the router, but a horizonal wood frame member prevented it. Makes no sense; who would put a solid, horizontal plank of wood in the middle of an interior wall? (Even fish tape would not have helped in this scenario!) We ended up drilling a hole in the ceiling close to the wall and running the Cat6 Ethernet cable down the wall to the router. It’s a little ugly, but the alternative, ripping out the dry wall, is even more undesirable.

Good thing I had a spare 5-port Ethernet switch; otherwise, I would have had to purchase one. In the attic, I connected all the camera’s Cat6 Ethernet cables to the switch and then the switch to the Internet router. Thankfully the attic had an outlet sockets nearby because I needed to plug in all the CleverLoop cameras’ power bricks (I used two power strips) and connect them to the PoE injectors, and then the PoE injectors to the Cat6 Ethernet cables. In the end, I ended up with a rat’s nest of cables, PoE injectors, and power bricks. I put the whole pile on a sheet of wood to prevent any contact with the attic’s insulation material.

Tip: I recommend running all the Cat6 Ethernet cables down into your air conditioned house. During the summer, the attic becomes so hot that I’m concerned that the Ethernet switch and power bricks may not last long.

Unfortunately, the cameras did not have any power indicators. (Though, if it was at night, I might have been able to see the IR LEDs light up.) Even worse, the camera took up to 5 minutes to come online (for it to be accessible by IP address) and to start streaming video. (The CleverLoop manual says it takes 90 seconds to boot up the camera.) Initially, I thought that the passive PoE injectors and splitters didn’t work because the Cat6 Ethernet cables were too long (one ran over 50 feet in length). I re-tested the longest Cat6 Ethernet cable and double-checked that there were no loose connections. Thankfully, one camera’s IP address appeared on my router’s client list (minutes had past during my debugging) and slowly the rest came online. Everything was fine; I just needed to be patient.

CleverLoop App

I connected the CleverLoop base station directly to the router. I installed the CleverLoop app onto my iPhone, created an account, and scanned the QR code at the bottom of the base station. In the CleverLoop app, I added the four cameras and started seeing their video streams (have to view each camera and click the play icon).

Now that we could see what the cameras were seeing, my brother made final adjustments to the cameras’ positions with my Dad’s input. Be careful when adjusting the positioning screws! Because the camera case is plastic, too much force could easily strip the screw holes. My brother did strip one of the positioning screw holes. He fixed it by screwing in an even larger screw.

When it got dark, the cameras automatically switched to black-and-white IR night vision mode. I was impressed by the night vision; I was able to see pretty far and the video was clearer than expected.

In the morning, one camera had an “Access Denied” error. And a second camera was still stuck in black-and-white IR mode. Rebooting the two cameras, removing and re-adding them did not fix the problem. As common with software, it turned out the cameras needed firmware updates to eliminate these and other bugs.

Tip: To reboot a camera, I can open up the camera’s outlet cover and disconnect the camera’s power cord from the PoE splitter. This is a lot easier than going into the attic, locating the particular camera’s Cat6 Ethernet cable and disconnecting its power plug from the PoE injector.

Per the CleverLoop support page, Check your camera firmware version (For X series indoor camera and outdoor camera only), I upgraded all the cameras’ firmwares to the latest recommended version. Note that you cannot upgrade directly to the latest version, you must upgrade incrementally. For example, I upgraded from version 00.10.01.0037 to 00.10.01.0037P1, and then from 00.10.01.0037P1 to 00.10.01.0039cl (latest version).

Upgrading to the latest firmware fixed both the “Access Denied” and stuck in black-and-white IR mode errors. And then 10 minutes after the upgrades, all cameras went offline (inaccessible by the CleverLoop app or browser). I had to power all cameras off and on to get them back. So, I recommend that you always manually reboot the camera after a firmware update.

Tip: You can browse to the camera’s IP address and view its video feed in the browser. The default username and password is “admin” and “123456”. On my Chrome browser running on Windows 7, I had to select “view video -Mode 2” (which uses Adobe Flash) to see the streaming video feed. Selecting “view video -Mode 1” (which uses “application/x-hyplayer” and QuickTime Player) did not work.

Video Quality

By default, the cameras are set to stream SD (Standard Definition 480p or 640×480 pixels) video quality. On a smartphone’s small screen, SD video looks pretty sharp. I did enable the 720p HD video on the cameras and only saw a slight improvement in quality on my iPhone 5’s tiny screen (it could be my imagination). I was more concerned about overwhelming the Internet connection’s upload bandwidth (for streaming over the Internet) so I changed it back to SD quality. So far, no complaints about video quality from Dad, who uses an iPhone 6 Plus.

Multiple Users

When I first installed the CleverLoop app, CleverLoop allowed multiple users to log in using the same account. So both my Dad and my sister (who lives with our father and has an Android phone) used the same account. A month later, that was no longer allowed. Logging into an account would log out anyone currently logged into that account. My sister had to create a new Cleverloop account and then add the base station (by scanning the QR code at the bottom of the base station). Thankfully, she didn’t need to re-add the cameras.

I think that CleverLoop disallowed account sharing in order to better support the Geo-Fencing function (automatically arms the motion detection system when your smartphone leaves the home location). However, because my Dad wanted alerts regardless of whether he was home or not, I had disabled the Geo-Fencing feature.

Alerts vs Movements

At first, I was confused by the movement and alert notifications. After reading up on it, I learned that we needed to train the base station to distinguished between harmless movements (like a branch waving in the wind) and important alerts (an intruder at the door). We are expected to view the video clip attached to each notification. And if we don’t agree with the classification (movement or alert), we can inform the system so by clicking on the “This should be an alert/movement” text at the bottom of the video feed.

cleverloop_motion_detectionTo eliminate the clutter caused by many notifications, you can manually delete the alerts and movements. There is a delete button that you can press when viewing the movement or alert. Additionally, there is an batch action (the top-right pencil icon when viewing a particular camera) that allows you to select multiple notifications for removal.

You can fine-tune the motion detection by marking areas of the video feed for analysis. Go to camera settings (the top-right gear icon when viewing a particular camera) and select “Fine-tune Smart Detection”. You can create up to 3 rectangular areas (a.k.a. hotspots) for motion detection and analysis. The parts of the video outside of the hotspots are not analyzed.

One major problem I see with the motion detection video clips is that the first few seconds are not shown. So, if you have someone who enters the video feed and exits quickly, the video clip won’t show that person. For example, when someone comes to the front door, drops off a package, and leaves, all I see in the video clip is a part of his back or his shadow as he is leaving. This is a known problem with a known solution called video pre-buffering. Basically, the system records continuously and when motion is detected, the generated video clip includes the previous 5-10 seconds. Hopefully, the CleverLoop base station will be updated to do pre-buffering soon.

Wireless Not Needed

When my Dad first asked about video surveillance for the backyard, he mentioned putting a camera on the detached garage and pointing it at the back of the house. Because I didn’t want to run an Ethernet cable from the house to the detached garage, I decided that a wireless camera was a necessary requirement.

Given a choice, I would prefer to avoid a wireless solution. Having a camera constantly streaming video across a wireless connection does not sound like a good idea, especially if it is 720p HD video. I can foresee complaints about Youtube and Netflix being flaky and having to reboot the wireless router frequently. Besides, if you’re wiring for power (which the camera needs regardless), you might as well wire for data too.

Thankfully, we managed to avoid using the camera’s wireless function. I asked my Dad to wait, to use the existing four wired cameras first. If he still wishes to have a camera on the detached garage, we can buy another CleverLoop base station with one or more outdoor cameras. So far, no requests for additional cameras from Dad.

Progress Report

After a month of operation, the CleverLoop base station stopped working (no alerts sent and CleverLoop app showed blank video feeds) and had to be rebooted. This is normal. I remember when cable modems and wireless routers first came out, I had to reboot them once a week. Then as they were improved, once a month. Now, several months go by before a reboot is necessary. I expect the same progress for the CleverLoop base station.

Because the CleverLoop app showed blank video feeds, I thought the cameras had stopped working. But it was the base station that had gone kaput. The base station pulls video from the cameras, analyzes the video for motion, and sets alert notifications. The CleverLoop app, in turn, pulls video from the base station. So, if the base station stops working, you get no video and no alerts in the CleverLoop app. Browsing directly to the cameras show all four video feeds working. Surprisingly, all cameras continue to function flawlessly without requiring any reboots.

Tip: The base station is responsible for doing the motion detection and sending the alert notifications. If you don’t get motion detection alerts after arming the system, check that the base station is powered on and connected (blinking LEDs in the back right on top of the ethernet jack).

Feature Requests

Below are my improvement and feature requests for CleverLoop:

  • The alert and movement video clips to include the previous 5 seconds. The video clips are missing the first few seconds immediately after the motion was detected. (See “pre-buffering” reference above.)
  • CleverLoop app to have a user-only mode where administrative functions like add/remove base station, add/remove camera, and camera settings are not accessible. In the current user interface, my Dad could easily remove the base station, remove a camera, or modify a camera’s settings with some accidental touches.
  • Camera list screen to update snapshot images of all four cameras once per second (or even once per 5-10 seconds). Currently, outdated snapshot images are shown.
  • Like the above, camera screen to update camera’s snapshot image once per second.
  • CleverLoop app to disallow adding a 5th camera to a base station. Even though I have added the maximum 4 cameras, the camera list screen still shows the “Add a New Camera” option.
  • Install CleverLoop app on an iPad. Currently, the iPad’s App Store does not list the CleverLoop app. It would be nice to see a bigger video feed on the iPad.

One feature that I think my Dad might like is to display all four video feeds on his large LED TV. If he ever asks for it, I think I can create an HTML page that shows the video feeds from all four cameras (using Adobe Flash). Run it on a laptop (or tiny desktop) attached to his TV and voila, a security guard’s dream come true!

No Comments

Clone a Big Hard Drive to a Smaller One

Windows No Comments

I had tried out Windows 10 by installing it on a second, bigger 500GB SSD (Solid State Drive) than my existing Windows 7’s 240GB SSD. Having determined that I wanted to permanently move to Windows 10, I decided to move Windows 10 to the smaller drive, overwriting Windows 7.

026GarfieldFirst, to clone from a bigger to smaller drive requires that the bigger drive not contain more data than can fit into the smaller drive. Second, the bigger drive must not have data stored at a location beyond the maximum supported location on the smaller drive. The safest way to satisfy both requirements is to shrink the source partition to ensure that it will fit 100% onto the hard drive.

Disable BitLocker

Before doing anything, I decided to decrypt the drive by turning Bitlocker off. I had tested cloning a Bitlocker-protected Windows 7 drive but it failed with a blue screen on startup, after getting past the annoying Bitlocker recovery procedure (because the hard drive signature had changed). So, I decided that it would be best to decrypt, clone, and then re-encrypt. Turning Bitlocker off didn’t take too long (about 20 minutes) because my Windows 10 was a fresh install with just Office and some other apps (about 35GB in size).

To turn BitLocker off, run “Manage BitLocker” and select the “Turn off BitLocker” option.

Resize Source Partition

So, here’s how to resize the Windows 10 source partition:

  1. Run Window 10’s “Create and format hard disk partitions” application (aka “Disk Management”).
  2. Right-click on the Windows 10 partition and select “Shrink Volume…”.
  3. Adjust the “Enter the amount of space to shrink in MB” until the “Total size after shrink in MB” is significantly smaller than the target hard drive size. (Because my target hard drive is 240GB, I tried to get the partition below 200GB to be on the safe side. Make sure to account for Windows 10’s two system partitions, a 300MB recovery partition and a 500MB EFI partition.)

There may be an upper limit to how much you can shrink the volume. You will see a text, “You cannot shrink a volume beyond the point where any unmovable files are located”, that explains why. Disk Management cannot move files used by system hibernation, paging, and protection (aka system restore) so it cannot shrink the volume past the furthest located of these files.

Note: If you have a non-SSD hard drive, you will want to run “Disk Defrag” (aka “Defragment and Optimize Drives”) first to consolidate the file locations to the head of the hard drive, before shrinking the partition.

Disable System Services

The solution to allow you to shrink the volume further is to disable system hibernation, paging, and protection first.

  • Disable Hibernation
    1. Click on Start, type “Command Prompt”, right-click on it and select “Run as administrator”.
    2. In the Command Prompt, type “powercfg /h off” to turn Hibernation off.
  • Disable Paging
    1. Run “View advanced system settings” to open the “System Properties” dialog, make sure the Advanced tab is selected, and click on the “Settings” button in the Performance section.
    2. Under the Advanced tab in the “Performance Options” dialog, click on the “Change…” button.
    3. Select “No paging file” and click the Set button. (We will need to reboot for this change to take effect.)
  • Disable System Protection
    1. Run “View advanced system settings” to open the “System Properties” dialog and select the “System Protection” tab.
    2. Select the C:\ drive and click on the “Configure…” button.
    3. Check the “Disable system protection” box and click OK. Answer Yes.

Once you have disabled the system services above, reboot (so the paging change can take effect), and repeat the Disk Management steps above to shrink the Windows 10 partition. You should be able to shrink the volume smaller than the destination drive’s size. (If you have more data on the source drive than can be contained by the target drive, you will need to uninstall and/or delete things from the source drive.)

CloneZilla the Drives

We have to use CloneZilla in expert mode (instead of beginner mode) in order to configure it to allow cloning from a bigger to a smaller drive.

Follow the first set of instructions at Clone a Hard Drive Using Clonezilla Live to create a bootable USB flash drive containing the latest version of Clonezilla Live.

Then follow the revised instructions below to clone the drives. (Steps 1 thru 7 are the same. In Step 8, we select “Expert mode” instead of “Beginner mode”.)

  1. Attach the destination drive to the same machine containing the source drive.
  2. Start the machine and boot from the USB flash drive. You may need to press a particular function key to load the boot menu (F12 on my Lenovo desktop) or you may need to adjust the BIOS setup to boot from a USB drive before the hard drive. (If you get offered Legacy or UEFI bootup options for the USB flash drive, choose UEFI.)
  3. On Clonezilla Live’s startup screen, keep the default “Clonezilla live (Default settings, VGA 800×600)” and press Enter.
  4. Press Enter to accept the pre-selected language, “en_US.UTF-8 English”.
  5. Keep the default “Don’t touch keymap” and press Enter.
  6. Make sure “Start_Clonezilla” is selected and press Enter to start.
  7. Because I am copying from one hard drive to another, I select the “device-device work directly from a disk or partition to a disk or partition” option. Press Enter.
  8. Change to “Expert mode” option and press Enter.clonezilla_expert
  9. Keep the first “disk_to_local_disk” option and press Enter.
  10. Select the source drive and press Enter.
  11. Select the target destination drive and press Enter.
  12. Check the “-icds Skip checking destination disk size before creating partition table” flag and press Enter.clonezilla_icds
  13. Keep the default “Skip checking/repairing source file system” selection and press Enter.
  14. Select the “-k1 Create partition table proportionally” flag and press Enter.clonezilla_k1
  15. Type “y” and press Enter to acknowledge the warning that all data on the destination hard drive will be destroyed.
  16. Type “y” and press Enter a second time to indicate that you are really sure.
  17. In answer to the question “do you want to clone the boot loader”, type uppercase “Y” and press Enter. (I need to clone the boot loader so the destination hard drive will be bootable like the source hard drive.)
  18. The hard drive cloning will occur.
  19. When the cloning completes, press Enter to continue.
  20. Select “poweroff” to shut down the machine.
  21. Once the machine is off, remove the source drive and boot from the destination drive. (Or use the boot menu to select the destination drive.)

Thankfully, CloneZilla automatically increase the size of the Windows 10 partition on the destination drive to take up the remaining available free space. (If CloneZilla didn’t increase the partition size for you, you can use the “Extend Volume…” function in “Disk Management” to grow the partition size manually.)

Re-enable System Services

Once you are certain that Windows 10 is working successfully off the smaller drive, you can re-enable the system hibernation, paging, and protection.

  • Enable Hibernation
    1. Click on Start, type “Command Prompt”, right-click on it and select “Run as administrator”.
    2. In the Command Prompt, type “powercfg /h on” to turn Hibernation on.
  • Enable Paging
    1. Run “View advanced system settings” to open the “System Properties” dialog, make sure the Advanced tab is selected, and click on the “Settings” button in the Performance section.
    2. Under the Advanced tab in the “Performance Options” dialog, click on the “Change…” button.
    3. Select “Automatically manage paging file size for all drives” at the top and click the OK button. (We will need to reboot for this change to take effect.)
  • Enable System Protection
    1. Run “View advanced system settings” to open the “System Properties” dialog and select the “System Protection” tab.
    2. Select the C:\ drive and click on the “Configure…” button.
    3. Check the “Turn on system protection” box and click OK.

Re-enable BitLocker

If you want to, re-encrypt the hard drive by turning Bitlocker on. Run “Manage BitLocker” and select the “Turn on BitLocker” option. (I don’t recommend choosing the option to encrypt the entire drive, instead of the used disk space only, unless you want to make sure that no one can recover deleted files. Encrypting the entire drive takes significantly more time, depending upon the amount of free disk space.)

If BitLocker didn’t already ask you to reboot, do a reboot to ensure that the paging change above takes effect.

Note: If you leave the source drive attached, it won’t show up in Windows 10’s File Explorer. Run “Disk Management” and you will see that the source drive’s status is “Offline (The disk is offline because it has a signature collision with another disk that is online)”. To make the source drive visible and accessible, right-click on the source drive’s label (“Disk 1” in my case) and select Online.

CloneZilla Didn’t Work!

I tried using CloneZilla to clone my laptop’s HDD (Hard Disk Drive) to a smaller SSD. Unfortunately, CloneZilla threw an error, “Write block error: no space left on device”. Even though it then completed the cloning process, my laptop was not able to boot off the resulting SSD.

Instead, I attached the laptop HDD and SSD to my desktop and ran the free version of EaseUS Partition Master on my desktop to successfully clone from the laptop HDD to the SSD. Here is what I did:

  1. Install EaseUS Partition Master Free Edition, run it, and click “Launch Application”.
  2. Select the source disk (in the right-hand panel listing all the disks), right-click and choose “Copy disk”. (Alternatively, you can run menu Wizard, “Clone disk wizard”, and select the source disk.)
  3. After the wizard finishes analyzing the source disk, click Next.
  4. Select the destination disk. Next.
  5. Choose the “Delete partitions on the destination hard disk” option. Next.
  6. The wizard should select the same sizes for the destination partitions as the source partition’s (except for the last partition, which should be the Windows partition, if the hard drive sizes are different).
    • On a MBR (Master Boot Record) disk, you should have two partitions, a tiny System partition and a large Windows partition. On a GPT (GUID Partition Table) disk, you will have one or two other tiny system or reserved partitions.
    • Note: The wizard had a problem selecting the destination partition sizes for my laptop’s SSD. It increased the 1GB System partition to 100GB. I had to drag to resize the System partition to a value closed to 1GB (couldn’t get it exactly the same) and increase the Windows partition size accordingly (to eliminate the Unallocated space).
    • If you drag the partition to a small enough size, you won’t be able to see the text inside showing the size. Just rest your mouse pointer on the partition and a popup text will appear with the size info.
  7. Once you are satisfied with the sizes, click Next and Finish. The Partition Master’s disk info will change to reflect the changes you made.
  8. Click the top-left Apply button to make those changes take effect.

If you don’t have a second Windows machine to do the above, you can do a self-migration of Windows 10 from the current disk to another disk using the EaseUS Todo Backup Free Edition. Run it and click the top-right Clone button. More instructions can be found at How to Migrate Windows 10 from HDD to SSD?

Most info above derived from:

No Comments

Node.js Express with Nginx Reverse Proxy and Cache

Mac OS X No Comments

As a web developer, have you ever asked yourself whether to use a period or a plus sign to concatenate two strings? The former is used by PHP and the latter by Javascript. If you switch between the frontend Javascript and the backend PHP languages often, you’ll find yourself asking this and other syntax questions.

nodejs-logoNode.js eliminates that problem by supporting the use of Javascript in the backend. It is a Javascript runtime that replaces the PHP backend. However, using Javascript for both frontend and backend is not the best reason to migrate to Node.js. You will want to use Node.js because of the Node Package Manager (NPM). NPM makes finding, sharing, and reusing Javascript code packages a breeze. When implementing a backend function, the first step to take is to search for an NPM package that provides that function already.

Node.js Express is a simple but powerful Node.js web application framework. We’ll use it as the backend to serve web pages. While Express can serve static files, Nginx is much faster at that task and provides many other benefits.

Using Nginx as a reverse proxy (browsers query Nginx which then calls Express) and cache for Express provides the following benefits:

  • Nginx is built as a high performance server with many optimizations. It can handle many concurrent connections and can perform load balancing. Nginx can serve (or cache and serve) static files like HTML, javascript, images, and CSS more efficiently than Node.js.
  • As the point of entry, Nginx provides better security because it is an older, proven web server solution. Nginx supports both the older SSL/TLS and newer HTTP/2 protocols.
  • Nginx can be used for port 80 and 433 binding to avoid having to run Node.js as a root user, which is bad practice and possible vulnerability. (Under Unix, the first 1024 ports require root privileges to bind to.)

Below are the steps I took to setup Node.js and configure a Nginx reverse proxy and cache. Though I’m doing the steps below on Mac OS X 10 Yosemite, the Node.js code and Nginx configuration should be applicable to other platforms.

Install Node.js

We will install Node.js using MacPorts. Launch the Terminal app and run these commands:

# Node.js uses python scripts; python 2.7 is included with Mac OS X since v10.8
python --version

# Install Node.js
sudo port install nodejs
node -v

# Install Node Package Manager
sudo port install npm
npm -v

# Create a project directory
mkdir myproject
cd myproject
mkdir static

# Create a NPM project package.json file
npm init

# Install Express package
npm install express --save

The “npm init” command will prompt for the project name, version, and description in order to generate a “package.json” file. You can accept the defaults and edit the “package.json” file later.

The “npm install express –save” command will download the Express package to a node_modules subdirectory. The optional “–save” flag will add the Express package and its version as a dependency in the “package.json” file. (The alternative “–save-dev” flag will save the package as a development dependency instead.)

The benefit of the above is that you can give the “package.json” file to another developer and they can run the “npm install” command to install the specific versions of all dependent packages (as listed in the “package.json”).

Express Server

Create a a file named “server.js” with the following content:

// Require Express package
var express = require('express');

// Create an Express app
var app = express();

// Serve static files from static dir
app.use(express.static('static'));

// Handle get on root / request
app.get('/', function (req, res) {
  res.send('Hello World!');
});

// Bind to port 8080
var server = app.listen(8080, function () {
  var port = server.address().port;
  console.log('Listening on port %s', port);
});

To test, do the following:

  1. Put an image file, say “earth.gif”, under the static subdirectory.
  2. In a Terminal window, launch the server with this command:
    node server.js
  3. Browse to http://localhost:8080/ and you will see a “Hello World!” response.
  4. Browse to http://localhost:8080/earth.gif to see the static image file.
  5. In the Terminal window running Node.js, press Ctrl-C to quit.

Instead of using the “node server.js” command, you can use the “npm start” command. If you look inside the “package.json” file, you will see that there exists a script target named “start” which runs the “node server.js” command.

Install Nginx

Install Nginx with this command:

sudo port install nginx

The Nginx configuration file is located at “/opt/local/etc/nginx/nginx.conf”. The default document root is set to “share/nginx/html”, which maps to the “/opt/local/share/nginx/html” directory.

You can start, reload, and stop Nginx using these commands:

# Start Nginx
sudo port load nginx

# Reload the config (actually restarts)
sudo nginx -s reload

# Stop Nginx
sudo port unload nginx

# Check to see if Nginx is running
ps -e | grep nginx

While Nginx is running, browse to http://localhost/ to see the Nginx welcome page.

If you need to troubleshoot, the Nginx error log file is located at “/opt/local/var/log/nginx/error.log”.

Nginx Reverse Proxy

When Nginx proxies a request to Node.js, it will optimized the request headers it receives from the client:

  • Nginx gets rid of any empty headers from the proxied request.
  • Nginx considers any header names with underscores as invalid. It will remove them. If you wish to preserve these headers, set the Nginx “underscores_in_headers” directive to “on”.
  • The “Host” header is re-written to the value defined by the $proxy_host variable. This will be the IP address or hostname and port number of the upstream, as defined by the “proxy_pass” directive.
  • The “Connection” header is changed to “close” to indicate to the upstream server that this connection will be closed once the original request is responded to.

Configure the Nginx reverse proxy by running “sudo nano /opt/local/etc/nginx/nginx.conf” and making the following changes to the root location:

    server {
        # ...

        # Helpful headers to pass to Node.js
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme; #http pr https
        proxy_set_header X-Real-IP $remote_addr; #client IP address
        # List of IP addresses client has been proxied through until now
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        location / {
            #root   share/nginx/html;
            #index  index.html index.htm;

            proxy_pass http://localhost:8080;
        }

Reload the Nginx config with the “sudo nginx -s reload” command. Browsing to http://localhost/ will now show the Node.js “Hello World!” message, instead of the Nginx welcome page.

By default, the “Host” header is set to “$proxy_host”, which is the upstream hostname or IP address and port defined in the “proxy_pass” definition. In the above, we have overridden it to be “$host” which is set to, in order of preference: the hostname from the request line itself, the optional “Host” header from the client request, or the server name matching the request. This is a best practice for Nginx and ensures that the “Host” header passed to the proxied server is as accurate as possible.

Note: The headers sent by the client are available in Nginx as variables. The variables start with an “$http_” prefix, followed by the header name in lowercase, and with any dashes replaced by underscores. So the client “Host” header is available in Nginx as “$http_host”.

If you wish to reverse proxy a non-root path location, use this Nginx location configuration:

        location /proxy/ {
            # Ending slash prevents passing /proxy/ path to Node.js
            proxy_pass http://localhost:8080/; # Ending slash required!
        }

Browse to http://localhost/proxy/ to test.

Nginx Serves Static Files

If Nginx has access to the Node.js project directory (for example, “/Users/myuser/myproject”), it is best to configure Nginx to serve the static files directly.

Run “sudo nano /opt/local/etc/nginx/nginx.conf” and add the following “/static/” location:

        location /static/ {
            root /Users/myuser/myproject; # Node.js project location
            expires 30d; # Cache-Control: client cache valid for 30 days
        }

Browse to http://localhost/static/earth.gif to test.

The “expires 30d” command adds a “Cache-Control” response header telling the browser client to only cache the static resource for 30 days maximum. Install Curl and dump the response header to see the “Cache-Control” value:

# Install Curl
sudo port install curl

# Run Curl to dump response headers
curl -X GET -I http://localhost/static/earth.gif
    # Cache-Control: max-age=2592000

Unfortunately, the above solution requires the URL to contain the “/static/” path. One workaround to avoid the “/static/” path is to have Nginx serve static files with specific extensions like so:

        # location ~* means to use case-insensitive match
        location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|bmp|js|html|htm)$ {
            root /Users/chvuong/nodejs/static;
            expires 30d;
        }

Because Node.js returns “Cache-Control” with an immediate expiration, we can verify that Nginx is serving the static file by doing the following:

# Get static image file from Nginx
curl -X GET -I http://localhost/earth.gif
    # Cache-Control: max-age=2592000

# Get static image file directly from Node.js
curl -X GET -I http://localhost:8080/earth.gif
    # Cache-Control: public, max-age=0

If we use Node.js to provide APIs only, we can configure Nginx to return static files only if they exist. Because API paths don’t usually correspond to existing directories and/or static files, Nginx will pass the API calls to Node.js. Here’s the Nginx configuration to use:

        location / {
            root /Users/chvuong/nodejs/static;
            expires 30d;

            # Return static file if exist; otherwise pass to Node.js            
            try_files $uri @nodejs;
        }

        location @nodejs {
            proxy_pass http://localhost:8080;
        }

Please remove or comment out the previous “location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|bmp|js|html|htm)$” block because it will interfere with the above configuration.

Test using the Curl commands above to get the “earth.gif” static file. You can also edit “server.js” to remove or comment out the line below to prevent Node.js from serving static files:

// Serve static files from static dir
//app.use(express.static('static'));

Nginx Cache

If Nginx does not have access to the static files (for example, Nginx is running on another server), you can configure Nginx to cache the static files returned by Node.js. If the browser client requests a static file which is already cached, Nginx will return it without having to request that file from Node.js again.

Run “sudo nano /opt/local/etc/nginx/nginx.conf” and add the following cache configuration:

http {
    # ...

    # Cache for static files
    proxy_cache_path /tmp/nginx-cache levels=1:2 keys_zone=staticcache:8m max_size=100m inactive=60m use_temp_path=off;
        # keyzone size 8MB, cache size 100MB, inactive delete 60min
    proxy_cache_key "$scheme$request_method$host$request_uri";
    proxy_cache_valid 200 302 60m; # cache successful responses for 60 minutes
    proxy_cache_valid 404 1m; # expire 404 responses 1 minute

    server {
        # ...

        location / {
            proxy_pass http://localhost:8080;
        }

        # Only cache static files; don't cache the dynamic API response!
        location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|bmp|js|html|htm)$ {
            proxy_cache staticcache; # Use "staticcache" we defined above
            proxy_cache_bypass $http_cache_control; # Support client "Cache-Control: no-cache" directive
            add_header X-Proxy-Cache $upstream_cache_status; # Hit or Miss

            # Nginx cache to ignore Node.js default "Cache-Control: public, max-age=0"
            # and don't pass it on to clients
            proxy_ignore_headers Cache-Control;
            proxy_hide_header Cache-Control;
            expires 60m; # "Cache-Control: max-age=3600" tells client to cache for 60 minutes

            proxy_pass http://localhost:8080;
       }

Caveats:

  • Double-check that “server.js” is configured to serve static files:
    // Serve static files from static dir
    app.use(express.static('static'));
  • Nginx will create the “/tmp/nginx-cache” directory because “/tmp” allows write access to everyone. If you change the cache directory location, please make sure to manually create the necessary directories. You will see an error in the Nginx error log if Nginx can’t create or access the cache directory.

You can test the cache by using Curl and taking note of the “X-Proxy-Cache” response header:

# First try will result in cache miss
curl -X GET -I http://localhost/earth.gif
    # X-Proxy-Cache: MISS

# Second try will result in cache hit
curl -X GET -I http://localhost/earth.gif
    # X-Proxy-Cache: HIT

If you wish to cache a non-root path location, use this Nginx location configuration:

        location /proxy/ {
            proxy_pass http://localhost:8080/; # Ending slash required!        
        }

        # Match string changed to capture path after /proxy/
        location ~* ^/proxy/(.+\.(jpg|jpeg|gif|png|ico|css|bmp|js|html|htm))$ {
            proxy_cache staticcache;
            proxy_cache_bypass $http_cache_control;
            add_header X-Proxy-Cache $upstream_cache_status;

            proxy_ignore_headers Cache-Control;
            proxy_hide_header Cache-Control;
            expires 60m;

            # Pass captured path string without /proxy/ to Node.js
            proxy_pass http://localhost:8080/$1;
        }

Nginx Load Balancing

Once you have Nginx reverse proxy working, load balancing is very simple. Here’s the Nginx load balancing configuration:

http {
    # ...

    # Node.js servers for load balancing
    upstream nodejs-backend {
        least_conn; # Give new connection to backend with least active connections
        localhost:8080;
        nodejs2.example.com; # default port 80
        nodejs2.example.com:8080;
    }

    server {
        # ...

        location / {
            # Pass to the upstream name, instead of the specific nodejs hostname
            proxy_pass http://nodejs-backend;
        }

I only tested the above with one Node.js server in the upstream definition (because I only have one server). It worked for one server and it should work for more.

Most info derived from:

No Comments

Free SSL Certificate from Let’s Encrypt for Nginx

Linux No Comments

See my previous post in the unmanaged VPS (virtual private server) series, Automate Remote Backup of WordPress Database, on how to create and schedule a Windows batch script to backup the WordPress database.

Update: Let’s Encrypt has made changes that have broken my pre-existing certificate renewal. First, the new certbot package has replaced the original letsencrypt package. Second, the certbot package does not recognize the pre-existing certificates in the “/etc/letsencrypt” directory (generated by the old letsencrypt package). If you have the old letsencrypt package, I recommend deleting it, the “~/.local” directory, and the “/etc/letsencrypt” directory before installing the new certbot package. I’ve updated the instructions below to use the new certbot package.

I’ve been meaning to enable HTTPS/SSL access to this WordPress site since I heard that Google had started giving ranking boosts to secure HTTPS/SSL websites; however, the thing stopping me was the expensive and yearly cost of a SSL certificate. (Unfortunately, self-signed SSL certificates wouldn’t work because browsers would throw security warnings when encountering them.) But now, there is a new certificate authority, Let’s Encrypt, which provides free SSL certificates.

The only catch is that the SSL certificates would expire in 90 days. But that’s okay because Let’s Encrypt provides a command line client which can create and renew the certificates. Some elbow grease and a weekly Cron job should automatically renew any expiring SSL certificates.

Note: StartSSL provides a free, non-commercial SSL certificate which can be manually renewed after a year. I learned about it at the same time as Let’s Encrypt, but decided to go with Let’s Encrypt because of the possibility of automation and no restrictions on usage.

Below are the steps I took to create and install the SSL certificates on the Nginx server running on my unmanaged DigitalOcean VPS.

Create SSL Certificate

Ubuntu didn’t have the certbot package, so we will need to build it from source. Secure shell into your VPS server and run these commands:

# Install Git version control (alternative to Subversion)
sudo apt-get install git

# Double-check that Git is installed by getting version
git --version

# Download the certbot client source code (to the home directory)
cd
git clone https://github.com/certbot/certbot

# Install dependencies, update client code, build it, and run it using sudo
cd certbot
./certbot-auto --help
# Input the sudo password if requested to

# Get a SSL certificate for mydomain.com
./certbot-auto certonly --webroot -w /var/www/wordpress -d mydomain.com -d www.mydomain.com
# Input your email address (for urgent notices and lost key recovery)

# Get another SSL certificate for mydomain2.com
./certbot-auto certonly --webroot -w /var/www/mydomain2 -d mydomain2.com -d www.mydomain2.com

Note: The Let’s Encrypt Ubuntu Nginx install instructions suggest using the wget command to get the latest available certbot version. I think “git clone” is a better method because it provides a more powerful way to update the certbot package, as we will see later.

Running the “certbot-auto” script will do the following:

  1. Install any missing dependencies including the GCC compiler and a ton of libraries.
  2. Update the certbot client source code.
  3. If necessary, build or update the certbot client, located at “~/.local/share/letsencrypt/bin/letsencrypt”. (The name switch from letsencrypt to certbot is not complete and thus a little confusing.)
  4. Run the certbot client using sudo; thus, you may be prompted to input the sudo password.

Note: If you want to speed it up by avoiding the extra update steps, you can just run the “sudo ~/.local/share/letsencrypt/bin/letsencrypt” command directly, instead of the “~/certbot/certbot-auto” script.

When running the “certbot-auto certonly –webroot” certificate generation option, the following (with some guesses on my part) occurs:

  1. The certbot client will create a challenge response file under the domain’s root directory (indicated by the “-w /var/www/wordpress” parameter); for example, “/var/www/wordpress/.well-known/acme-challenge/Y8a_KDalabGwur3bJaLfznDr5vYyJQChmQDbVxl-1ro”. (The sudo access is required to write to the domain’s root web directory.)
  2. The certbot client will then call the letsencrypt.org ACME server, passing in necessary credential request info such as the domain name (indicated by the “-d mydomain.com -d www.mydomain.com” parameters).
  3. The letsencrypt.org ACME server will attempt to get the challenge response file; for example, by browsing to “http://mydomain.com/.well-known/acme-challenge/Y8a_KDalabGwur3bJaLfznDr5vYyJQChmQDbVxl-1ro”. This verifies that the domain has valid DNS records and that you have control of the domain.
  4. The letsencrypt.org ACME server passes the generated SSL certificate back to the certbot client.
  5. The certbot client writes the SSL certificate to the “/etc/letsencrypt” directory, including the private key. If you can only backup one thing, it should be the contents of this directory.
  6. The certbot client deletes the contents of the “.well-known” directory; for example, leaving an empty “/var/www/wordpress/.well-known” directory once done. You can manually delete the “.well-known” directory.

Note: It is possible to create a multi-domain certificate containing more than one domain, but I recommend keeping it simple. Multi-domain certificates are bigger to download and may be confusing to the user when viewed.

Configure Nginx

The directory where the SSL certificates are located under, “/etc/letsencrypt/live”, require root user access, so we will need to copy the certificate files to a directory which Nginx can read.

# Copy out the SSL certificate files
sudo cp /etc/letsencrypt/live/mydomain.com/fullchain.pem /etc/nginx/ssl/mydomain-fullchain.pem
sudo cp /etc/letsencrypt/live/mydomain.com/privkey.pem /etc/nginx/ssl/mydomain-privkey.pem
sudo cp /etc/letsencrypt/live/mydomain2.com/fullchain.pem /etc/nginx/ssl/mydomain2-fullchain.pem
sudo cp /etc/letsencrypt/live/mydomain2.com/privkey.pem /etc/nginx/ssl/mydomain2-privkey.pem

# Double-check that the files exist and are readable by group and others
ls -l /etc/nginx/ssl

Because our website will behave the same under HTTPS as under HTTP, we will only need to make minimal changes to the existing HTTP server configuration.

Edit Nginx’s server block file for mydomain (“sudo nano /etc/nginx/sites-available/wordpress”) and add the “#Support both HTTP and HTTPS” block to the “server” section:

server {
        #listen   80; ## listen for ipv4; this line is default and implied
        #listen   [::]:80 default ipv6only=on; ## listen for ipv6

        #Support both HTTP and HTTPS
        listen 80;
        listen 443 ssl;
        ssl_certificate /etc/nginx/ssl/mydomain-fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/mydomain-privkey.pem;

        root /var/www/wordpress;
        #index index.php index.html index.htm;
        index index.php;

        # Make site accessible from http://localhost/
        #server_name localhost;
        server_name mydomain.com www.mydomain.com;

If you wish to have this server block be the default to serve (for both HTTP and HTTPS) when your server is accessed by IP address, change the “listen” directives to the following:

server {
        #Support both HTTP and HTTPS
        listen 80 default_server; # default_server replaces older default
        listen 443 ssl default_server;
}

Note: Make sure that only one of your server block files is set to be the default for IP address access. Also, when you browse to the IP address using HTTPS, you will still get a security warning because the IP address won’t match the domain name.

Repeat the above modifications for any other domain’s server block file.

Note: If you want HTTPS to behave differently than HTTP, leave the HTTP server section alone, uncomment the “# HTTPS server” section (at bottom of the server block file) and make your updates there.

Once you are done updating the server block files, tell Nginx to reload its configuration:

# Reload Nginx service
sudo service nginx reload

# If Nginx throws an error, look at the error log for clues
sudo tail /var/log/nginx/error.log

To test, browse to your server using the HTTPS protocol; for example, “https://mydomain.com/”.

Renew SSL Certificate

To renew all your SSL certificates, run this command 30 days before the expiration date:

~/certbot/certbot-auto renew

Note: If you run it earlier than 30 days before the expiration date, no action will be taken.

Cron Job To Renew

To automate the SSL certificate renewal, we will use a Cron job that runs weekly under the root user. Running the job weekly is sufficient to guarantee a certificate renewal within the 30 days before expiration window.

First, create a script by running “nano ~/certbot_cron.sh” and inputting the content below (make sure to replace “mynewuser” with your actual username):

#!/bin/bash

# Run this script from root user's crontab

# Log file
LOGFILE="/tmp/certbot-renew.log"

# Print the current time
echo $(date)

# Try to renew certificates and capture the output
#/home/mynewuser/certbot/certbot-auto renew --no-self-upgrade > $LOGFILE 2>&1
/home/mynewuser/certbot/certbot-auto renew > $LOGFILE 2>&1

# Check if any certs were renewed
if grep -q 'The following certs have been renewed:' $LOGFILE; then
  # Copy SSL certs for Nginx usage
  cp /etc/letsencrypt/live/mydomain.com/fullchain.pem /etc/nginx/ssl/mydomain-fullchain.pem
  cp /etc/letsencrypt/live/mydomain.com/privkey.pem /etc/nginx/ssl/mydomain-privkey.pem
  cp /etc/letsencrypt/live/mydomain2.com/fullchain.pem /etc/nginx/ssl/mydomain2-fullchain.pem
  cp /etc/letsencrypt/live/mydomain2.com/privkey.pem /etc/nginx/ssl/mydomain2-privkey.pem

  # Reload Nginx configuration
  service nginx reload
fi

Notes about the script:

  • To test the script, run this command:
    sudo sh ~/certbot_cron.sh

    If none of your certificates are renewed, you won’t get a “Reloading nginx configuration” message (outputted by the “service nginx reload” command).

  • The “–no-self-upgrade” argument flag can be passed to certbot to prevent certbot from upgrading itself. At first, because we will be running the script under the root user, I hesitated to allow certbot to update with root permissions automatically. Avoiding an update seemed more secure and definitely faster to execute. However, without the update, by the time the three months expired, the certbot is hopelessly outdated and won’t successfully renew certificates. So, I had to allow certbot to automatic self-upgrade with root privileges to avoid having to do manual updates.
  • To simulate certificate renewals, use the “–dry-run” argument flag to simulate a successful renewal. Change the “certbot-auto” command in the script to the following:
    /home/mynewuser/certbot/certbox-auto renew --dry-run > $LOGFILE

    When you re-run the “certbot_cron.sh”, you will get the “Reloading nginx configuration” message. Don’t forget to remove this change from the script once you are done testing.

  • The script copies out all the SSL certificates, instead of checking for and only copying certificates which have been modified. I don’t think the effort to do the latter is worth it.

Add the script to the root user’s Crontab (Cron table) by running these commands:

# Edit the root user's crontab
sudo crontab -e
  # Insert this line at the end of the file
  @weekly sh /home/mynewuser/certbot_cron.sh > /tmp/certbot-cron.log 2>&1

# List content of root user's Crontab
sudo crontab -l

# Find out when @weekly will run; look for cron.weekly entry
cat /etc/crontab

Note: Instead of “@weekly”, you may wish to set a specific time that works best for your situation. Refer to these Cron examples for info on how to set the time format.

If you want to test the Cron job, do the following:

# Delete the script's output log file
sudo rm /tmp/certbot-renew.log

# Change "@weekly" to "@hourly" in the Crontab
sudo crontab -e
  # Edit this line at the end of the file
  @hourly sh /home/mynewuser/certbot_cron.sh > /tmp/certbot-cron.log 2>&1

# Wait more than an hour

# Check if output log files were generated
cat /tmp/certbot-renew.log
cat /tmp/certbot-cron.log

# Change "@hourly" back to "@weekly" in the Crontab
sudo crontab -e
  # Edit this line at the end of the file
  @weekly sh /home/mynewuser/certbot_cron.sh > /tmp/certbot-cron.log 2>&1

Update Certbot

If you have chosen to disable the certbot self update in the cron script (using the “–no-self-upgrade” argument flag), I recommend manually running the “certbot-auto” command (without any arguments) once a month to make sure that certbot is up-to-date.

If you find that the “certbot-auto” command is unable to self-update or doing the self-update doesn’t solve an issue, you can try to update using the “git pull” command.

# Update the certbot source code using git
cd certbot
git pull

# See status of certbot source code and version
git status

Backup SSL Certs & Keys

Note: Re-issuing the SSL certificates (because of the switch from the letsencrypt to the certbot package) proved to be painless and fast. Thus, I’ve realized that backing up the “/etc/letsencrypt” directory is not necessary. If something goes wrong, just re-issue the SSL certificates.

In Automate Remote Backup of WordPress Database, we created a Windows batch file to download MySQL dump files and other files from the server. Let us add an additional command to that Windows batch script to download a zip archive of the “/etc/letsencrypt” directory.

Originally, I added an ssh command to the Windows batch file to zip up the “/etc/letsencrypt” directory. Unfortunately, accessing that directory requires sudo privileges which causes the script to prompt for the sudo password. I looked at two solutions to running sudo over SSH without interruption. The first involved just echo’ing the sudo password (in plaintext) to the ssh command. The second involved updating the sudoers file to allow running a particular file without requiring the password. I didn’t actually test the two solutions, but they didn’t look secure so I decided go with a very simple solution: run the zip command in the “~/certbot_cron.sh” script.

First, edit the Cron script (“nano ~/certbot_cron.sh”) and add the tar zip command after reloading the Nginx server:

# Check if any certs were renewed
if grep -q 'The following certs have been renewed:' $LOGFILE; then
  ...

  # Reload Nginx configuration
  service nginx reload

  # Zip up the /etc/letsencrypt directory
  tar -zcvf /tmp/letsencrypt.tar.gz /etc/letsencrypt
fi

Note: We are using the tar command, instead of gzip, because gzip doesn’t handle the symbolic links under the “/etc/letsencrypt/live” directory correctly.

There is a security issue because “/tmp/letsencrypt.tar.gz” is readable by others; if this is a concern, you can adjust access permissions by adding the following commands to the “~/letsencrypt_cron.sh” script:

  ...

  # Zip up the /etc/letsencrypt directory
  tar -zcvf /tmp/letsencrypt.tar.gz /etc/letsencrypt

  # Change owner and restrict access to owner
  chown mynewuser /tmp/letsencrypt.tar.gz
  chmod 600 /tmp/letsencrypt.tar.gz
fi

Second, edit the Windows batch script file “C:\home\myuser\backups\backup_wordpress.bat” and add the following to the end:

REM Download the /etc/letsencrypt tar file

mkdir \home\myuser\backups\letsencrypt
cd \home\myuser\backups\letsencrypt
rsync.exe -vrt --progress -e "ssh -p 3333 -l mynewuser -v" mydomain.com:/tmp/letsencrypt.tar.gz %date:~10,4%.%date:~4,2%.%date:~7,2%-letsencrypt.tar.gz

And we are done with the backup. In the future, if you ever need to restore the contents of “/etc/letsencrypt”, upload the tar archive to the server’s “tmp” directory and run the following on the server:

# Unzip the tar file
cd /tmp
tar -xvzf letsencrypt.tar.gz
# Will uncompress everything to /tmp/etc/letsencrypt

# Copy the contents to its original location
sudo cp -r /tmp/etc/letsencrypt /etc/

Redirect HTTPS to HTTP

If you have a domain which you don’t wish to provide HTTPS access to (i.e., go through the trouble of creating a SSL certificate for), you can configure Nginx to redirect HTTPS requests to HTTP. Uncomment the “# HTTPS server” section in the domain’s server block file and add a redirect statement:

# HTTPS server
#
server {
        listen 443 ssl;
        server_name monieer.com www.monieer.com;

        #To redirect, use return instead of no longer recommended rewrite:
        #rewrite ^(.*) http://$host$request_uri;
        return 302 http://$host$request_uri;
}

Because we did not set the “ssl_certificate” and ssl_certificate_Key” values in the server_block above, Nginx will use the default_server’s SSL certificate instead. Unfortunately, the browser will show a security warning because the domain name in the default_server’s SSL certificate won’t match the requested domain name. If the user agrees to proceed, the redirection to non-secure HTTP access would correctly take place.

Info above derived from:

No Comments

Nginx with PHP and MySQL on Windows 7

Windows Development 8 Comments

In the past, whenever I needed a web server on Windows, I would install the XAMPP distribution (comes with Apache, PHP, and MySQL) and call it a day. This time, I wanted to use Nginx instead of Apache as the web server. Below are the steps I took to install Nginx, PHP and MySQL separately on Windows 7.

Install Nginx

Nginx is pretty easy to install on Windows. Just do the following:

  1. Download the latest Nginx for windows version. (Currently, only 32-bit versions are available.)
  2. Unzip to a directory like “c:\nginx”.
  3. Create two subdirectories which “nginx.exe” expects to exist:
    mkdir c:\nginx\logs
    mkdir c:\nginx\temp
  4. If you want to change the document root from the default “c:\nginx\html” and/or enable directory listing, edit the “c:\nginx\conf\nginx.conf” file and adjust the global “location /” declaration like so:
            location / {
                #root   html; # comment out default root at "nginx_install_dir\html"
                root   /www;  # use new root "c:\www" (assuming nginx is install on c: drive)
                index  index.html index.htm;
                autoindex on; # Add this to enable directory listing
            }
  5. To run the Nginx web server, launch the “Command Prompt” and issue these commands:
    # Go to Nginx installation directory
    c:
    cd \nginx

    # Start Nginx
    start nginx.exe

    Notes:

    • Running “c:\nginx\nginx.exe” without changing to the “c:\nginx” directory will fail because Nginx will look for the log and temp subdirectories, which won’t exist under another directory.
    • The “start” command will launch Nginx in a separate window; otherwise, Nginx would take control of the current “Command Prompt” window. That separate window will appear and quickly disappear.
    • It is not necessary to run the “Command Prompt” as an administrator.
  6. Browse to http://localhost/ . You should see a “Welcome to nginx!” page.
  7. To quit Nginx, in the “Command Prompt” window, do the following:
    # Go to Nginx installation directory
    c:
    cd \nginx

    # Quit Nginx
    nginx.exe -s quit

    If you have started Nginx more than once, the quit command above will only kill the last Nginx process started. To kill all Nginx processes, run the following:

    taskkill /F /IM nginx.exe

To avoid launching multiple instances of Nginx, I created the following “intelligent” Nginx start and stop batch script files.

Create a file named “start_nginx.bat” with the content below:

@ECHO OFF

REM Start Nginx
tasklist /FI "IMAGENAME eq nginx.exe" 2>NUL | find /I /N "nginx.exe">NUL
IF NOT "%ERRORLEVEL%"=="0" (
   REM Nginx is NOT running, so start it
   c:
   cd \nginx
   start nginx.exe
   ECHO Nginx started.
) else (
   ECHO Nginx is already running.
)

And create a file named “stop_nginx.bat” with this content:

@ECHO OFF

REM Stop Nginx
tasklist /FI "IMAGENAME eq nginx.exe" 2>NUL | find /I /N "nginx.exe">NUL
IF "%ERRORLEVEL%"=="0" (
   REM Nginx is currently running, so quit it
   c:
   cd \nginx
   nginx.exe -s quit
   ECHO Nginx quit issued.
) else (
   ECHO Nginx is not currently running.
)

Install and Configure PHP

To install PHP on Windows:

  1. Browse to php.net, click on the “Windows downloads” link, and download the latest thread safe version. Either 32-bit or 64-bit versions will work.
  2. Unzip to a directory, like “c:\nginx\php”.
  3. Select a PHP configuration (I recommend the development version):
    copy c:\nginx\php\php.ini-development c:\nginx\php\php.ini

We will run the “c:\nginx\php\php-cgi.exe” server to allow Nginx to execute PHP scripts using the FastCGI protocol.

  1. Edit “c:\nginx\conf\nginx.conf” to uncomment the FastCGI section and update the fastcgi_param entries like so:
            # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
            #
            location ~ \.php$ {
                root           html;
                fastcgi_pass   127.0.0.1:9000;
                fastcgi_index  index.php;
                #fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
                fastcgi_param  REQUEST_METHOD $request_method;
                fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include        fastcgi_params;
            }

    Notes:

    • Don’t forget to update the “root” location value if you are not using the default “c:\nginx\html” directory.
    • The fastcgi_param values are recommended extra parameters passed to PHP scripts for their optional use.
  2. Add the following to the bottom of the “start_nginx.bat” file:
    REM Start php-cgi
    tasklist /FI "IMAGENAME eq php-cgi.exe" 2>NUL | find /I /N "php-cgi.exe">NUL
    IF NOT "%ERRORLEVEL%"=="0" (
       REM php-cgi is NOT running, so start it
       start /min c:\nginx\php\php-cgi.exe -b localhost:9000 -c c:\nginx\php\php.ini
       ECHO php-cgi started.
    ) else (
       ECHO php-cgi is already running.
    )

    Notes:

    • Use localhost instead of 127.0.0.1 to support both IPv4 and IPv6 addressing (if it is enabled). 127.0.0.1 was not resolvable on my IPv6-enabled Windows installation. (Strangely, using 127.0.0.1 in the nginx.conf’s FastCGI section above is okay though.)
    • Unfortunately, “start php-cgi.exe” will show a separate “Command Prompt” window which will remain visible; the “/min” parameter flag is used to minimize that window. If you really want to prevent that window from appearing, you’ll need to use a VBScript to execute the batch file.
    • The order in which Nginx and php-cgi are launched does not matter.
    • The PHP distribution has a “php-win.exe” file which supposedly is the same as “php-cgi.exe” but without throwing up a “Command Prompt” window; however, I could not get “php-win.exe” to run as a server.
  3. Add the following to the bottom of the “stop_nginx.bat” file:
    REM Stop php-cgi
    tasklist /FI "IMAGENAME eq php-cgi.exe" 2>NUL | find /I /N "php-cgi.exe">NUL
    IF "%ERRORLEVEL%"=="0" (
       REM php-cgi is currently running, so quit it
       taskkill /f /IM php-cgi.exe
       ECHO php-cgi killed.
    ) else (
       ECHO php-cgi is not currently running.
    )
  4. Create a PHP test script at “c:\nginx\html\info.php”with the following content:
    <?php
    phpinfo();
    ?>
  5. Run “start_nginx.bat” to launch Nginx and php-cgi. Browse to http://localhost/info.php and you should see information about the PHP installation.

Install MySQL

Let’s get MySQL up and running:

  1. Download the latest MySQL Community Server. I suggest the “ZIP Archive” distributions, either the 32-bit or 64-bit version. Click the Download button. You don’t need to log in to download, just click the “No thanks, just start my download” link at the bottom of the page.
  2. Unzip to a directory like “c:\nginx\mysql”.
  3. Select the default MySQL configuration:
    copy c:\nginx\mysql\my-default.ini c:\nginx\mysql\my.ini
  4. Initialize MySQL by running the “Command Prompt” as an administrator (so Windows registry keys and service can be created) and these commands:
    # Current directory must be the MySQL installation directory
    cd c:\nginx\mysql

    # Initialize database with root user and blank password
    bin\mysqld --initialize-insecure

    # Install MySQL as a Windows service
    bin\mysqld --install-manual

    # Start MySQL Server service
    net start mysql
  5. Test by running these commands (administrator privileges not required):
    # Run MySQL client, skipping password input since blank
    c:\nginx\mysql\bin\mysql.exe -u root --skip-password

    # Run some commands and a query
    mysql> SHOW databases;
    mysql> USE mysql;
    mysql> SHOW tables;
    mysql> DESC user;
    mysql> SELECT * FROM user;

    # Assign new root password
    mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'new_password';

    # Quit MySQL client
    mysql> quit;

    # Run MySQL client with password prompt
    c:\nginx\mysql\bin\mysql.exe -u root -p
    # Input the new_password
  6. Enable PHP mysqli extension by uncommenting the line below in “c:\nginx\php\php.ini”:
    extension=php_mysqli.dll
  7. Create a test script named mysql.php with the following content:
    <?php
    // HTML response header
    header('Content-type: text/plain');

    // Database connection parameters
    $DB_HOST = 'localhost';
    $DB_PORT = 3306; // 3306 is default MySQL port
    $DB_USER = 'root';
    $DB_PASS = ''; // blank or new_password
    $DB_NAME = 'mysql'; // database instance name

    // Open connection (all args can be optional or NULL!)
    $mysqli = new mysqli($DB_HOST, $DB_USER, $DB_PASS, $DB_NAME, $DB_PORT);
    if ($mysqli->connect_error) {
      echo 'Connect Error (' . $mysqli->connect_errno . ') ' . $mysqli->connect_error . PHP_EOL;
    } else {
      // Query users
      if ($result = $mysqli->query('SELECT User FROM user')) {
        echo 'Database users are:' . PHP_EOL;
        for ($i = 0; $i < $result->num_rows; $i++) {
          $result->data_seek($i);
          $row = $result->fetch_assoc();
          echo $row['User'] . PHP_EOL;
        }
        $result->close();
      } else {
        echo 'Query failed' . PHP_EOL;
      }
    }

    // Close connection
    $mysqli->close();
    ?>
  8. Run “stop_nginx.bat” followed by “start_nginx.bat” to restart Nginx and php-cgi processes. Browse to http://localhost/mysql.php and you should see a listing of the MySQL database users.
  9. You can stop and/or uninstall the MySQL Server service by running “Command Prompt” as an administrator and issuing these commands:
    # Stop MySQL Server service
    net stop mysql

    # Uninstall MySQL Server service
    sc delete mysql

You don’t have to run MySQL Server as a Windows service. Instead, you can run MySQL Server from the “Command Prompt” (administrator privileges not required) like so:

# Start MySQL Server
c:\nginx\mysql\bin\mysqld.exe

# Stop MySQL Server
c:\nginx\mysql\bin\mysqladmin.exe -u root shutdown

Unfortunately, the “mysqld.exe” will take control of the “Command Prompt” window and the “start” command does not work in this case, so you will need to open a second “Command Prompt” window in order to issue the shutdown command.

Some info taken from:

8 Comments

« Previous Entries