Archive for the ‘Windows 7’ Category

Robocopy can silently fail to copy directories with invalid UTF-16 names (or, why I always compare after copying)

December 11, 2014

Over the last few days I’ve been copying the bulk of the home directories on my primary file server over to a new volume (don’t ask) and, of course, I did a comparison afterwards to make sure the copy was successful [1].  I’m talking about 3070 home directories, comprising over seven million files structured in any number of strange and wonderful ways.

I wasn’t at all surprised to find that 3069 of those directories had copied perfectly; robocopy is pretty reliable.

I was a little surprised to find that one directory had an anomaly, but still, glitches happen.  I became puzzled, though, when I realized what the problem was: an entire subdirectory was missing.  Robocopy hadn’t reported any errors.  What’s more, when I ran robocopy over that home directory again, it reported that there was nothing to do: as far as it was concerned, source and destination were a perfect match.

Explorer didn’t show me much.  The name of the two directories in the source looked the same; the first character was shown as a box.  Another little tool of mine, though, could see the difference:


The tool escapes non-ASCII characters with a percent sign followed by a hexadecimal representation, so the first wide character is 0xD898 in the first directory and 0xDADB in the second.  Otherwise the names are the same.  Only the first one was present in the destination.

The next step, obviously, was to look up the Unicode code points 0xD898 and 0xDADB.  As it turns out, they are “high-surrogate code points”, used in UTF-16 to encode Unicode code points larger than 16 bits.  The key here is that surrogate code points are only valid in pairs: an individual surrogate code point is meaningless.

Of course, NTFS doesn’t care.  It doesn’t really understand Unicode, so one 16-bit character is much like another.  As far as NTFS is concerned, those are perfectly good (and distinct) names.  Robocopy, however, must for some reason be converting or normalizing the UTF-16 strings, and as a result it sees those two names as identical.  (It appears to be ignoring the second occurrence of the “same” name in a single directory; it doesn’t attempt to copy the second subdirectory at all.)

So, if you’re in the habit of creating files with invalid UTF-16 names, be warned. 🙂

[1] Using some code I wrote myself.  Microsoft don’t seem to have provided a reliable directory-level comparison tool, and I’m not aware of any existing third-party solutions.  I should open-source that tool one day.

Preventing executables from requiring UAC elevation

September 23, 2011

I’ve just found this great tip over at Stack Overflow.

In Windows Vista and later, an application can be coded to require UAC elevation.  If you try to run it as a non-administrator, you get asked for an administrator username and password, and if you don’t provide them the application doesn’t start.  That’s all very well, but some developers set this flag when it isn’t really needed (I’m looking at you, beepa) which locks out all non-administrators.

This isn’t usually a big problem on a home machine, because you probably have an administrator account even if you don’t use it for everyday activities.  In a teaching lab, however, as in many other contexts, it’s fatal; the students don’t know the administrator password (or at least I devoutly hope they don’t!) and obviously we’re not going to tell them what it is.

It turns out that this is as simple as setting an environment variable.  Set __compat_layer to RunAsInvoker, and Windows will ignore the application manifest.  You could set this globally via group policy, or write a simple wrapper program around specific applications that need it.  (Of course, if an application really does require administrator privilege it may fail in strange and unexpected ways, so take care.)

Hope this helps – and Norbert, if you’re reading this, thank you.  Knowing about this is going to come in very handy.

When Guest is the administrator

May 5, 2011

Nommo was kind enough to point me to his latest blog entry which discusses a troubleshooting case where the Guest account had somehow wound up being the only active administrator account on a Windows Vista computer.  This was reasonably easy to reproduce (although I used Windows 7 instead) and, indeed, user management tools don’t work as might be expected.

This is interesting.  In most other respects the Guest account still functions as an administrative account.  At first I thought I understood exactly why this happened (to do with the way LAN Manager handled security way back in the days of DOS) but a bit of experimentation showed I was wrong.  It now looks as though a Guest logon is tagged in some way and prohibited from doing any user management – the Guest account can’t even look up its own details – unless you elevate to it from another account, in which case it has user-level access to account management but not administrator-level.  Weird, huh?

Only the actual Guest account is affected; other accounts that are in both Administrators and Guests are not.

Because only account management is blocked, you can get around this in a few ways.  Probably the simplest in most cases is to download the psexec tool from Microsoft.  Start an elevated command-line window by typing “cmd” into the “Search Programs and Files” box in the Start Menu and pressing Control-Shift-Enter.  Then type:

cd /d c:\directory\where\psexec\was\downloaded\to
psexec -s \\ net localgroup Administrators /add myusername

pressing ENTER after each line and changing “myusername” to the username of the other (currently non-administrative) account.

Alternately, you could edit the registry as described in my earlier post but you don’t need to boot from external media:

  1. Go to the Start Menu and type “regedit” and press ENTER.
  2. Open HKEY_LOCAL_MACHINE, then SYSTEM, then Setup.
  3. Double-click on SetupType in the right-hand pane.  Enter 2 and press OK.
  4. Double-click on CmdLine.  Enter cmd.exe and press OK.
  5. Reboot the machine.  A command window should appear.
  6. Type “net localgroup Administrators /add myusername” and press ENTER.
  7. Type: “exit” and press ENTER.

Again, “myusername” should be replaced with the username of an existing, non-administrative account.  After this procedure, the account is administrative.  You could also use “net user myusername newpassword” to change the password if necessary.  (The same caveat applies as in my previous post: doing this permanently locks you out of any encrypted files in the account.)

Now, obviously Guest shouldn’t be an administrator.  The fact that things behave oddly in this situation is not a bug.  However, if Guest is an administrator the normal recovery options don’t work properly.  In particular you are supposed to be able to log in as Administrator if no usable administrative accounts exist, and in this situation you can’t, and this is a bug.

Hope this helps.

Installing 32-bit software as SYSTEM in Windows 7 x64

February 20, 2011


I recently ran into an issue updating the Sun Java runtime on our x64 machines.  We don’t have the budget for fancy deployment solutions, so we just use a startup script (actually an executable, but that’s just fine-tuning) that checks the version number and runs the installer(s) as necessary.

Installing the 32-bit JRE results in error code 1619, which NET HELPMSG translates as “This installation package could not be opened. Verify that the package exists and that you can access it, or contact the application vendor to verify that this is a valid Windows Installer package.”  Running the installer in interactive mode produces the same message.  The installer works normally when run from the context of a logged-in user.

Several hours of troubleshooting later, I identified the source of the problem.  Startup scripts run as local system.  In Windows 7, processes that run as local system have a special profile found in c:\windows\system32\config\systemprofile.  Unfortunately, on 64-bit systems, there are two system32 folders; one for 64-bit processes,and another (whose real name is syswow64) for 32-bit processes.  As a result, there are two separate system profiles; one for 32-bit, one for 64-bit.

So what?  Well, the Sun Java installer unpacks into a subfolder of the LocalLow application data directory.  In this case, the folder in question is c:\windows\system32\config\systemprofile\AppData\LocalLow\Sun\Java\jre1.6.0_24.  Because this is a 32-bit process, though, it is really writing to syswow64 instead of system32.

The Windows Installer, however, is a 64-bit process.  So when it is asked to open the MSI file, it’s looking in the wrong place; hence error code 1619.  The file can’t be opened because it can’t be found.

The same underlying problem (duplication of the system profile) seems to be the cause of this problem with Known Folders warning 1002 appearing repeatedly in the event log.  Some 32-bit system process is registering folder paths inside the (32-bit) system profile and of course these folders can’t be found by 64-bit processes.

For the problem with installing 32-bit software, there are a number of possible workarounds.  You could manually extract the installer files, copy them to a suitable path on the local system, and run them directly.  Most installers won’t mind this, although some will balk or fail to function properly.  Alternatively (and this is the solution I chose) you could create the necessary directory ahead of time and add a junction point (mklink /J) from the 64-bit profile to the 32-bit profile (note that this command line assumes you are in a 32-bit context, and has been split for readability):

mklink /J c:\windows\sysnative\config\systemprofile\AppData\LocalLow\Sun\Java\jre1.6.0_24

This is the equivalent command if you are in a 64-bit context:

mklink /J c:\windows\system32\config\systemprofile\AppData\LocalLow\Sun\Java\jre1.6.0_24

Another possible approach would be to merge the two system profiles together and create a junction point from one to the other.  That would solve this issue for all installers, as well as the Known Folders issue and any other variants.  However, I can’t recommend doing this; it’s too broad a change, and there’s no way to predict what it might break.  If you’re very brave, go ahead, but test thoroughly – and don’t blame me!

Hope this helps.


Incorrect Username or Password – or perhaps the clock is wrong

July 2, 2010


OK, this one had me puzzled for an hour or two.  My main test box having failed, I had taken one of the older machines hanging around my office and given it a once-over; some extra RAM, a bigger hard disk drive, that sort of thing.  I noticed the motherboard battery had died, but figured I could live without it.  What the hey; the machine stays connected to the mains most of the time anyway.

So I go to install Windows 7 on the box, using our automated install system on my USB stick.  This is a Windows PE image which runs Windows Setup in unattended mode over the network.  It also includes a bit of in-house code so that you can provide an administrator password to join the machine to the domain in advance; once this password has been accepted, everything else happens automatically.  A few hours later the machine will be ready to use, software and all.  This sort of streamlining may seem unnecessary, but it does make life easier.

Except that this time the domain controller wouldn’t accept my password.  Typed it in again; no luck.  After trying once more, well, okay, several times more, and logging in on another machine to make sure I hadn’t gotten confused about which password went with which account, I realised there really was something wrong.

So I try it on another machine, and it works perfectly.  I try booting the first machine from a copy on CD instead of the one on my USB stick; no difference.  Bringing up a command line window allows me to explicitly attempt to connect to the domain controllers as well as other network servers, with both my own account and a test account with a simpler password – just in case!

The results were puzzling to say the least.  Error code 86 – the network password is not correct – for any user account but only from that one machine and only when connecting to a domain controller.  The file server, by comparison, was perfectly happy to accept the supposedly “incorrect” passwords.  In case there was something wrong with the Windows PE image, I repeated the experiment using the standard Windows 7 install DVD, but this exhibited the same problem.  Swapping the keyboards and network cables between the machines was a long shot at best, this didn’t pan out either.  The event log on the domain controller showed the logon failures but provided no additional information.  For some reason, the problem didn’t occur when booted to a copy of Windows PE 2 (which is based on Vista rather than Windows 7) but this didn’t help me much.

Eventually it dawned on me that the flat motherboard battery meant that the on-board clock would have been reset; the computer thought the year was 2004.  Mismatched clocks have been known to cause authentication problems, and sure enough, once I had corrected the date and time via the BIOS the domain controller accepted my password and the automated installer set to work.

I’m not sure what the moral of this story is exactly.  I suppose I should have remembered the failed battery as soon as the machine started behaving oddly, and stopped to think about whether there could be a connection.  Anyway, I’ve tagged this as a bug, not because the authentication should necessarily have worked, but because the error code (or at least the event log!) should have indicated the actual problem instead of insisting that the password was wrong.


Resetting a password in Windows 7 or Windows Vista

March 9, 2010

One problem that comes up now and again for any OS (well, any modern OS) is how to recover administrative access to the system when the password has been forgotten.

There’s a reasonably straightforward solution [1] for Windows 7, which I haven’t seen on the web so far, so I thought I should publish it.  Be aware that I can’t offer or provide any warranty, support, or assistance with this procedure, apart perhaps from clarifying any part of the instructions that aren’t clear.  It’s always worked for me, but that’s all I can promise.

Additional note 13 June 2012: see also this question on, which provides a number of alternatives.

This procedure also works on Windows Vista; the underlying technique works on Windows XP as well but is not usually feasible because the Windows XP install CD does not contain a command-line recovery option.

One important caveat: since this resets the password rather than letting you find out what it is, any encrypted files belonging to the user will be permanently lost.

This is the short version, for advanced users and sysadmins:

  1. Boot to Windows 7 from the installation or repair DVD, or from Windows PE 3 boot media, or from a Windows 7 installation on another HDD.  If the target OS is Vista, use the Vista installation DVD, or Windows PE 2, or another Vista installation.  (Booting to a mismatched version of Windows might work, but I’ve never tried it; if the registry file formats aren’t exactly the same between versions, this could result in a corrupted registry and an unbootable system.)
  2. Load the SYSTEM registry hive from the target OS.  Back it up first.
  3. In the Setup key, change SetupType to 2 and CmdLine to cmd.exe.
  4. Boot the target OS.  You’ll get a command-line window in system context.

The long version, for everyone else:

  1. Boot to your Windows 7 or Windows Vista installation DVD, whichever matches the installed OS.  If you purchased your computer from a responsible vendor, they’ll have provided you with one, although unfortunately many vendors don’t.
    Additional note 8 September 2011: In Windows 7, there is an option in the Start Menu (under Maintenance) to Create a System Repair Disc.  The CD or DVD this option creates is perfect for the job.  However, you have to be an administrator to use it, so unless you’ve done it ahead of time or can use a friend’s Windows 7 machine you’re out of luck.
    Additional note 1 September 2011: If your computer is 64-bit capable (you don’t need to actually be running a 64-bit OS) then you can use the install disk for Microsoft’s free server product, Hyper-V.  You can find it here.  Note, however, that it is a fairly big download, a little more than a gigabyte.
    Additional note 5 May 2011: Nommo was kind enough to point me to this post on Microsoft Answers which provides a link to downloadable repair disks for Vista and Windows 7.  I can’t from my own knowledge confirm that these disks are legitimate, and Microsoft aren’t telling, so use only at your own risk.  Indications are that they are probably OK.  (Personally, I wouldn’t use the charged-download option until I’d checked how much my OEM was going to charge to provide an installation disk.  Make sure the OEM knows you need a Windows installation disk, not a system recovery disk.)
    A vendor system recovery disk might offer the same functionality, and in some cases you can order an installation DVD from your vendor (or from Microsoft?).
  2. Select your language options on the first screen and press Next to continue.
  3. Choose “Repair Your Computer”.
  4. Choose “Use recovery tools…” and select your OS.  Make a note of which drive letter it is on, e.g., C: or D:.  This might not be the same drive letter you see when booted normally.
    Additional note 1 September 2011: if you get an error message when you press Next, this might be because the install disk you are using is not compatible with the version of Windows you have installed.  This will happen, for example, if you are using the Hyper-V install disk.  Don’t panic.  Just press SHIFT-F10 to open a command prompt and skip ahead to step 6.
  5. Select Command Prompt.
  6. In the command prompt window that appears, type “regedit” and press ENTER.
  7. Select HKEY_LOCAL_MACHINE and then choose Load Hive from the File menu.
  8. Find and open the file named SYSTEM on the drive you noted in step 4.  If Windows is in the default configuration, this will be in windows\system32\config.
  9. Enter a key name, e.g., “xxx”.
  10. Click the plus icon to the left of HKEY_LOCAL_MACHINE to open this key.  Select the xxx key.
  11. Select Export from the File Menu.  Change the Save as type to Registry Hive Files.  Type a name for the backup, for example, systembackup, and press Save.  (This step creates a backup of the unmodified SYSTEM registry hive as a precaution.)
  12. Open the xxx key, and select Setup.
  13. Double-click on SetupType in the right-hand pane.  Enter 2 and press OK.
  14. Double-click on CmdLine.  Enter cmd.exe and press OK.
  15. Close Registry Editor.  Type “regedit” and press ENTER to open it again.  (This step does not appear to be necessary in Windows 7, but in Windows Vista if you do not do this the next step might fail with an Access Denied error.)
  16. Open HKEY_LOCAL_MACHINE, select xxx, and choose Unload Hive from the File Menu.  Push Yes.
  17. Close the command window and the Registry Editor.  Remove the installation DVD and select Restart.
  18. When your computer boots up, another command window should appear.
  19. Type “net user foo bar”, replacing foo with the username of the account whose password you want to reset, and bar with the new password.  For example, you might type “net user Administrator letmein”.  Press ENTER.
  20. If you want to use the built-in Administrator account, you will probably need to enable it: type “net user Administrator /active:yes” and press ENTER.
  21. If you don’t know what the administrative username(s) are, type “net localgroup administrators” and press ENTER to find out.
  22. Type “exit” and press ENTER.
  23. When the logon screen appears, use the username and the new password to log in.

Note that if the entire disk is encrypted, this procedure will not work at all.  System administrators who want to prevent users from using techniques like this one to reset passwords should consider disk encryption. [2]  Another option is to configure the system BIOS to disallow booting from removable media, although if the user can open the case of the machine this can usually be reset.

Hope this helps.


[1] Well, for some definitions of straightforward, anyway.

[2] I’ve heard tell of some administrators whose “solution” to this issue is to use the network firewall to block access to any web sites with instructions on resetting passwords!  Whether they also inspect all printed material entering the building, and ban anybody they think might be smart enough to just remember how to do it, I don’t know.

DISM corrupts images mounted to paths with short filenames

February 22, 2010

If you’ve been having trouble using DISM to update off-line Windows 7 images, make sure you aren’t mounting the image to a path which contains a short filename component.  In particular, note that the TEMP environment variable will contain a short filename component in Windows 2003 or earlier, and may contain a short filename component in later versions, depending on your username.

For example, if I were to create a folder on C drive named harryjohnston, this command would corrupt the mounted image:

dism /mount-wim /wimfile:c:\win7\install.wim /index:1 /mountdir:c:\harryj~1\mount

This wouldn’t:

dism /mount-wim /wimfile:c:\win7\install.wim /index:1 /mountdir:c:\harryjohnston\mount

If an image has been corrupted in this way, and is then used to install Windows 7 on a machine, the junction points on the target machine will retain the path of the mountpoint.  For example, the “Documents and Settings” junction point would point to c:\harryj~1\mount\Users instead of to c:\Users as it should.

Thanks to Joseph Conway from Microsoft, this issue has been filed as a bug and will hopefully be fixed in a future release of the WAIK (Windows Automated Installation Kit).  In the meantime, the workaround is obvious: use a different mountpoint. 🙂

Hope this helps.

Delays when connecting to Windows 7 clients for remote administration

December 18, 2009

OK, this is my first ever blog post.  Bear with me.

If you are remotely administering a Windows 7 client, for example, listing the services on the remote machine using the Computer Management tool or the sc.exe command line, there may be an unexpected delay when connecting.  If you use netstat -a -n during this delay you will see a TCP connection from your machine to the target machine sitting in the SYN_SENT state.  After a little while this connection attempt times out and the operation succeeds anyway.

Another example of a remote administration tool that suffers from this problem is psexec.exe.

This will happen if you are connecting from another Windows 7 machine  (or, presumably, Windows 2008 R2) and the firewall on the target machine is configured by group policy with the “Allow inbound remote administration exception” setting enabled.

The cause: the group policy setting configures one of the relevant firewall rules incorrectly.  The “Remote Administration (RPC)” rule is set to apply to svchost.exe instead of services.exe.  My best guess is that this is a bug in the Windows 7 group policy client.

The problem can be worked around by turning on an appropriate rule locally on the affected clients.  If you are using the GUI, turning on the “Remote Service Management” exception will solve the problem.  From the command line:

netsh advfirewall firewall set rule name="Remote Service Management (RPC)" 
profile=domain new enable=yes

Note this is all a single line, but I have split it for readability.  You could use group policy to include this command in a startup script, or run it remotely for each machine using psexec.  It only needs to be run once on each machine.  Note that the command-line version only enables one of the rules associated with the “Remote Service Management” exception, but if you have the above-mentioned group policy exception defined the other necessary rules are already present.

Hope this helps.