Wednesday, May 23, 2007

How I Managed To Successfully Update My Zune to 1.3

Until today, I haven't been able to update my Zune from firmware version 1.2 to the 1.3 release, which has been out for about 1.5 months now.  I would get the screen inside of the Zune software (running on my computer) stating that there was an update, but when I tried to actually install it, the software would simply stare back at me with a messagebox stating "It is not possible to change sync settings at this time. Try again later."

Searching the Internet proved that others were having the same issue, but there wasn't a clear explanation about what the problem was, or more importantly, how to resolve it.  It also seems that this error message is generic in nature: if any problems at all are encountered while trying to upgrade the firmware, you get this message box.  (That's my disclaimer to state that what worked for me may not work for you)

In typical fashion, there were a lot of posts saying "ZOMFG, Microsoft is teh sux0rz...  Screw dis, I'm buying an iPod".  But, I personally love my Zune, so I made it my morning mission to figure out what was wrong, and then fix it if it turned out to be something that was under my control.

My particular environment (details that matter, in this case):

  • Windows XP SP2
  • A corporate network using ISA Server (that requires authentication against Active Directory) for web proxy.  Note that I never attempted the upgrade from home or any other network - I just never thought about the firmware except while at work.
  • I log into my machine as a local user and have password rules established for network resources (Control Panel -> User Accounts | Manage My Network Passwords).

The first thing that I did was try to find where the new update might have been saved to my disk (to see if it actually existed, and then to see if there was a way that I could manually install the update).  I found the following (hidden) directory:

C:\Documents and Settings\yourUserNameHere\Local Settings\Application Data\Microsoft\Zune\Firmware Updates

Inside was another directory with a GUID name:

{EF56D6F0-08D7-4866-AB2F-208E48FF9796}

And inside of that, a lot of zero-byte files like: BIT6A4.tmp, BIT4C2.tmp, etc.

Hmmm... That looks an awful lot like temporary files for the Background Intelligent Transfer Service (BITS).  Since there were scores of these files (roughly coinciding with the number of times that I've attempted to install the update), I think that I found an important clue.

I fired up Fiddler, the HTTP Debugging Proxy, and started logging HTTP traffic that took place on my machine.  I then selected "Check for Zune Device Updates" within the Zune software (context menu for the device itself), and observed the resulting HTTP traffic. 

One request retrieved an XML document from a Zune server that contained information about the latest firmware version, including a link to the updated file.  There were actually 3 HTTP requests that occurred before the XML itself was returned: the first tried without proxy authentication, and failed with a 407 status code.  The second was to get the NTLM challenge from the proxy server, and also resulted in a 407.  The third was the same request again with an appropriate NTLM response in the Proxy-Authorization header, and this time, the resulting status code was 200 and the payload from the remote web server was successfully returned.  The user agent for all three of these requests was "Zune/1.3.5728.0", so it was the software itself trying to make the connection.

Note: This might seem like a noisy exchange, and it kind of is, but this is what happens behind the scenes when you web browser makes HTTP requests using a proxy server and NTLM authentication.

Another request, which tried to get the updated firmware CAB file itself, would fail with a status code of 407 in the same fashion as the very first web request made by the Zune software.  Interestingly enough, the user agent for this request was "Microsoft BITS/6.6", confirming my suspicion that the Zune software uses BITS to download the firmware images.  Furthermore, as soon as the 407 came through, the error messagebox displayed in the Zune software, and no further attempts were made to negotiate authentication with the proxy server by the BITS service.

At this point, I concluded that the Zune software was able to connect to the Internet, but the BITS client was not. 

Since I couldn't change the Zune software, I looked around to see if I could force the BITS client to authenticate against the proxy server, just like Internet Explorer (and even the Zune software itself) does automatically.  One option sounded promising: I could set up a small proxy server on my computer that would accept an unauthenticated request, but would then authenticate with the real proxy server when it forwarded that request to the Internet.

Luckily, though, I stumbled upon the following KB article:

http://support.microsoft.com/kb/842309

Interesting snippet from the article:

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

BITS 2.0 clients may not successfully complete the transfer of a file

HTTP 407-Proxy Authentication.

Note When the BITS 2.0 client receives this error message, it may also receive error code 0x80190197.

This problem occurs when all the following conditions are true:

  • The client program that is using BITS 2.0 does not specify that the credentials of the client can be used by calling the IBackgroundCopyJob2::SetCredentials method.
     
  • The Microsoft LAN Manager compatibility level (LmCompatibilityLevel) on the BITS 2.0 client contains a value that is set to 1 or to 0. You can find the LAN Manager compatibility level entry n the following registry subkey:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\LmCompatibilityLevel
     
  • The file transfer is performed through a Windows-based server or a Windows-based Internet proxy server that requires Integrated Windows authentication.

Authentication behavior in BITS 2.0

For security reasons, to avoid passing credentials to any proxy or server that requests logon credentials, BITS 2.0 allows logon credentials to be used only if one of the following conditions is true:
  • A program prompts for credentials to be applied by calling the following function and similar parameters:
     
    IBackgroundCopyJob2::SetCredentials. (target=ProxyorServer, Negotiate/NTLM, username=NULL,password=NULL)
     
  • If the LmCompatibilityLevel value on the client is set to a value that is greater than or equal to 2.

Important By default, Windows XP sets the LmCompatibilityLevel to 0. With BITS 2.0, logon credentials will not be used if the LmCompatibilityLevel is set to 0.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Now I was getting somewhere!  I checked the registry, and LmCompatibilityLevel was indeed set to 0, which exactly explains why BITS was not automatically authenticating with the proxy server.

Before changing values, I used the BITSADMIN program (you can find it on your XP CD-ROM in Support Tools) to see how many BITS jobs existed (bitsadmin /list) and then cleared out the list (bitsadmin /reset).  This eliminated the scores of failed (inactive) jobs that were still hanging around.

I then changed LmCompatibilityLevel to a value of 2, and then restarted the BITS service, reset the Zune (hold the Back Arrow and Up at the same time until it resets), and restarted the Zune software.  This time, the firmware CAB file was successfully downloaded by the BITS client, and the moment that this finished, the Zune client proceeded with installing the firmware onto the device.

Sweet!

Note: The reason for this BITS behavior is based on security: you really don't want background processes to be automatically authenticating with servers on your behalf if you're not aware of what they might be doing.  The nature of BITS is that you can disconnect from one network, and later connect to a different network (maybe at a coffee shop), and your download will resume automatically without requiring any action on your part.  So, if you do happen to change your LmCompatibilityLevel in order to upgrade the Zune, then be sure to change it back when you are finished in order to preserve your system's integrity.