This story began about 15 months ago, in November 2006. That was the time when Microsoft was getting very close to releasing Windows Vista, and it was the time for me to start getting serious about making sure my applications were compatible with it.
At that time I was using two computers for the development and testing, one with two single-core Intel processors, and another one with one single-core AMD x64 processor. Both were set up for development and testing of my programs: I was using the first one to test the 32-bit versions, and the second one to test the 64-bit editions of my programs. Since many people reported that Vista was more hardware hungry than XP, I thought it was a good occasion for me to also get a more powerful computer that would run Vista reasonably well. So I bought a new Core 2 Duo (dual core) processor, a motherboard to support it (P5L-MX from ASUS), a new video card to support the Aero user interface of Vista, put them together in a spare computer case I had, loaded up Vista Release Candidate on it, and started working on porting my applications to Vista.
It all went well for a while, except every couple of days or so my new powerful computer would once of a sudden “blue screen” and reboot.
After it happened a few times, I fired up WinDbg and loaded a few latest minidumps into it. They indicated that the crashes were happening in the FASTFAT.SYS driver, and the common reason for the errors was IRQL_NOT_LESS_OR_EQUAL, a common reason for a crash caused by a sloppily written device driver. It seemed like a bug in the FASTFAT.SYS driver shipped with the pre-release version of Vista. I decided there was not much I could do but hope that the bug would be fixed in the final (RTM) Vista release.
A couple of months later the RTM release of Vista became available, so I’ve reformatted the hard drive to get rid of the release candidate of Vista, installed a fresh copy of Vista RTM on it, and started using it.
In a day or two the same crashes started to happen again.
Figuring Microsoft would not release a new version of Vista with a buggy version of such an important driver as FASTFAT.SYS, I started looking for another reason. What made it difficult was that the blue screens appeared not very often, sometimes a week would go by and I started to hope I finally found out the reason, but inevitably, it would crash again no matter what I tried. And I tried a plenty:
I vacuumed the inside of the case and reseated the processor and the RAM modules.
The blue screens kept happening.
I ran the memtest program to check the RAM for errors for a few hours, it did not find any problems with the RAM.
The blue screens kept happening.
I installed the SpeedFan program to monitor the temperature of the hardware components. Although it did not show an overheating, I added another fan to the case.
The blue screens kept happening.
I’ve replaced the video card with another one.
The blue screens kept happening.
I’ve bought a new SATA hard drive (previously I was using an old IDE drive), and moved the Vista installation to it.
The blue screens kept happening.
I thought that maybe I got a faulty motherboard, so I bought a new one, this time P5LD2, again from ASUS. I also picked up another Core 2 Duo processor and a new set of RAM modules to go with it. I reinstalled Vista RTM from scratch, and set up my development environment, and started working as usual.
The blue screens kept happening.
As you can see, at that point I already had two computers which were giving me the blue screens every couple of days or so. I ran out of the new theories about the reason for the crashes, and I returned to the one I started with: the bug was probably in the FASTFAT driver of Vista after all, maybe I should have waited till Vista SP1 was out before switching to Vista as my main development platform. I started thinking about switching back to Windows XP. It so happened that at that time (in September 2007) I was locked out of both of my Vista computers by the buggy Genuine Advantage code of Windows Vista (I plan to share that experience of mine in a separate post, later on, stay tuned). That made the decision to switch back to XP real easy.
I was using Windows XP for several years, and never had a problem like that before, so imagine my surprise that after I’ve reinstalled Windows XP on each of my new computers, the blue screens started to happen almost from the day one. As before, they were occurring in the FASTFAT.SYS driver. It made it clear for me that I was blaming Vista in vain, it did not introduce a new bug, or, at least, if the bug was there, it was not Vista-specific.
I started analyzing the similarities between the two new computers, hoping that would give me a clue. They had different motherboards (although from the same manufacturer), slightly different processors, different RAM modules, different video cards, different hard drives (one was using a WD SATA drive, another one a Maxtor IDE drive). I came up with the idea that maybe I got very unlucky and I got two faulty motherboards. Luckily, at that time the built-in network adapter on one of the motherboards died, and I took this opportunity to RMA the motherboard back to ASUS. I got the replacement back in a few days, and installed it.
The blue screens kept happening.
Thinking that getting three faulty motherboards in a row was very unlikely, I started to try other things. Even though my two old computers were plugged in the same UPS device as the new ones and were working just fine, I thought maybe the new computers were more sensitive to the quality of the power they were getting.
I replaced a cheapo generic power supply in one of the new computers with a considerably more expensive and supposedly better one from Antec.
The blue screens kept happening.
I bought a new, more powerful UPS, specifically for use by the new computers.
The blue screens kept happening.
Out of desperation, I started all over and repeated every troubleshooting step I did before, with each of the crashing systems: reseated the modules, replaced the cables, ran the memtest.
The blue screens kept happening.
At that point, about a month ago, I ran out of ideas. I was ready to surrender and just live with it. Or maybe throw out both of the new computers I’ve built and buy a completely new one, and I was seriously contemplating that, when on January 15 it hit me: what that FASTFAT.SYS driver was doing there anyway? All of my hard drives have been formatted with the NTFS file system, I didn’t remember formatting a drive with the FAT or FAT32 system recently. Why would Windows load the FASTFAT driver?
I reviewed the properties of the drives listed in My Computer, and sure enough, there was one of them formatted with the FAT32 system. It was a virtual encrypted drive I created a while back with the TrueCrypt software. I used the drive as a backup place for sensitive files of mine. Periodically, I would burn the image to a DVD-R disc, to make a backup of it. And yes, there was a copy of this image on each of the new computers experiencing the crashes.
I reformatted the encrypted volumes with the NTFS file system.
The blue screens stopped.
It’s been almost a month since I’ve made the last change, and I have not had a single blue screen. Previously, they were happening every couple of days. I’m very confident now that I’ve found the culprit that caused me so much grief. I believe the following list describes the common conditions for the blue screens to occur:
- The computer should have a multi-core processor, such as Intel Core 2 Duo.
- The computer should have TrueCrypt 4.3a installed, and there should be an encrypted FAT32 volume mounted.
Why do I think the first condition is important? Because previously I was using TrueCrypt with FAT32 virtual drives for several years on the computers that had single-core processors, and never experienced such crashes with them. Only when I switched to the Core 2 Duo processors the crashes started to occur.
I’ve looked through the source code of TrueCrypt 4.3a and noticed that its driver was compiled with the NT_UP switch in its Makefile. This is definitely wrong. It means that the driver was targeted at the uni-processor systems. Since the multi-core processors are essentially multi-processors, defining NT_UP means asking for trouble.
Why did the crashes stopped after I’ve reformatted the encrypted drives with the NTFS file system? I don’t know. Apparently the NTFS file system driver is more robust and can tolerate the imperfect drivers such as the ones compiled with the NT_UP switch. Why didn’t I get crashes with my old two-processor computer? Again, I don’t know. Maybe the old computer was not fast enough for the error conditions to occur so frequently, and when it did crash once in a blue moon, I just dismissed that as something insignificant and did not pay attention to it.
Now, I noticed that a few days ago a new version of TrueCrypt 5.0 had been released. It uses a new driver build procedure, that does not seem to have the NT_UP flag anymore. This is good. However, looking through their support forums it seems like the new version introduced quite a few new bugs. I guess I’ll postpone upgrading to it until version 5.1 comes out. I want to get some rest from the blue screens for awhile :-)
Update: April 15, 2008
A few days ago I decided to try the latest release of TrueCrypt, 5.1a. I reformatted the NTFS encrypted volume back to FAT32 and the blue screens started to occur almost immediately. After two days of bluescreening, I reformatted the volume back to NTFS, and the blue screens stopped. It looks like TrueCrypt 5.1a still causes this problem. HTH.
Update: May 13, 2008
A week ago I started another experiment : connected a spare hard drive about the same size as the TrueCrypt volume I use, formatted the hard drive with the FAT32 system (just like the TrueCrypt volume that was giving me the blue screens), and copied everything from the encrypted volume to that hard drive. Then I dismounted the TrueCrypt volume, and assigned its drive letter to the hard drive I’ve just attached. Restarted the computer and kept using it as before, the only difference was that instead of a FAT32-formatted encrypted volume I was now using a regular FAT32-formatted unencrypted hard drive. A week passed by, no blue screens. Today I copied everything back from the hard drive to the FAT32-formatted TrueCrypt volume, and disconnected the extra hard drive. About an hour later a blue screen occurred. I think that proves conclusively that TrueCrypt is the real culprit behind these blue screens. HTH.
Update: July 16, 2008.
A week ago I’ve installed a new version 6.0a of TrueCrypt. One of the new things in it was an updated device driver with the improved support for the multi-core processors. That gave me the hope that this version might have finally fixed this bug. For a week it was running smooth, no BSoDs, even though I’ve switched to using a FAT-formatted encrypted volume. I was thinking about reporting success here, but today – boom, blue screen with IRQL_NOT_LESS_OR_EQUAL status in fastfat.sys. So I’m switching back to the NTFS volume and reporting for now that version 6.0a of TrueCrypt still has not fixed this problem. HTH.
Did you know? Our USB Encryption Software can protect any USB flash drive or any other external drive with a password and strong encryption. Try it free for 30 days! Read more…
February 17th, 2007 at 10:10 pm
I believe that this is because there is an NTFS fork on the directory that says that anything in that directory shouldn’t be trusted. This is similar to XP in how it knows that a file was downloaded recently by IE.
February 19th, 2007 at 6:06 pm
Hi Andrei,
Have a quick question about TweakUAC. Can I suppress UAC messages only for a single application using TWeak? Or does it suppress all UAC messages, system wide?
Thanks.
Regards,
Soumitra
Hi Soumitra,
> Can I suppress UAC messages only for a single application using TWeak?
No, it’s impossible.
> Or does it suppress all UAC messages, system wide?
Yes, that’s how it works.
Andrei.
February 20th, 2007 at 11:22 am
Hi Myria,
> I believe that this is because there is an NTFS fork on the directory that says that anything in that directory shouldn’t be trusted. This is similar to XP in how it knows that a file was downloaded recently by IE.
It may very well be so, but it does not make it any less of a bug. If a file contains a valid digital signature, Windows should not misrepresent it as coming from an unidentified publisher.
Andrei.
February 26th, 2007 at 1:09 pm
How did you take a screenshot of the UAC? I can’t get Print Screen to copy it to the clipboard, and the snipping tool isn’t working either.
February 26th, 2007 at 9:55 pm
Hi Chris,
> How did you take a screenshot of the UAC? I can’t get Print Screen to copy it to the clipboard, and the snipping tool isn’t working either.
Those tools don’t work because UAC displays its messages on the secure desktop, to which the “normal” user tools have no access. To solve this problem, I’ve changed the local security policy to make the UAC prompts to appear on the user’s desktop. After that, I used the regular Print Screen key to capture the screenshots.
Hope this helps,
Andrei.
March 4th, 2007 at 2:58 am
Hi Andrei,
I sell software to a *very* non-technical customer base. My setup procedure includes installation of an .ocx file into the \windows\system32 folder and registration of it using regsvr32. In order to copy anything into the \windows\system32 folder under Vista I have to turn off UAC. I would like to be able to do this automatically, programmatically, so I don’t have to make my users mess with UAC. I’d like to be able to turn off UAC for a second or two programmatically, then turn it back on. Will your software enable me to do that?
Thanks
March 4th, 2007 at 11:42 am
Hi Matthew,
> I’d like to be able to turn off UAC for a second or two programmatically, then turn it back on.
Unfortunately it’s impossible: if you enable or disable the UAC, Windows must be restarted before the change would have take effect.
To solve your problem:
> In order to copy anything into the \windows\system32 folder under Vista I have to turn off UAC.
It looks like your setup process is executing non-elevated, that’s why it cannot do that. You may want to try to start it elevated and see if it would have solved the problem without turning off the UAC.
HTH
Andrei.
March 25th, 2007 at 10:24 am
don’t use TweakUAC because this program makes your Vista unsafe!
March 29th, 2007 at 10:30 pm
> How are you supposed to make the decision whether to trust a certain program or not if UAC does not provide you with the correct information? (Nevermind, it’s a rhetorical question).
The answer is you are not. A guest should not be allowed to make any decision about installing software. If you log on as a valid user, the prompt works just fine. If you log on as a guest, you shouldn’t be installing software, so any dire warning is fine.
Yes, this might be unintended behavior (or perhaps it is not), but its impact is null.
March 31st, 2007 at 4:47 pm
Hi Herbys, you wrote:
> The answer is you are not. A guest should not be allowed to make any decision about installing software.
Sorry, but you are missing the point: the UAC displays this information for the _administrator_ to use and to make a decision, not for the guest user. The administrator is supposed to review the information and enter his or her password to approve the action. Take a look at the screenshot and see for yourself.
> If you log on as a guest, you shouldn’t be installing software
Why shouldn’t I? What if I want to install a program for use by the guests only? For example, I use only one web browser (IE), but I never know what browser a guest may want to use. So, being a good host I want to install also Firefox and Opera, but I don’t want them to clutter my desktop, etc., I want them to be used by the guests only. To achieve that, I would log in to the guest account and install the additional browsers from there.
> so any dire warning is fine.
Wrong.
> Yes, this might be unintended behavior (or perhaps it is not), but its impact is null.
May be, may be not. In any case, it does not make it any less of a bug!
April 10th, 2007 at 9:28 pm
Is there any plan to adapt your program into a Control Panel Applet? I think that would be very clever.
April 11th, 2007 at 9:07 am
Hi Timothy, you wrote:
> Is there any plan to adapt your program into a Control Panel Applet?
No, we don’t have such plans at this time, sorry.
June 18th, 2007 at 9:19 pm
Anyone know why Vista won’t let me rename any new folder?
The permissions are all checked for me as administrator, still I get an error message, “folder does not exist”. I can put things in folder and move it, but can’t rename it?
January 29th, 2008 at 9:46 am
To be honest, I have always thought that digitally signing was merely a way of generating more revenue. It doesn’t offer you any more security and windows will always moan at you regardless of an application having a signature or not.
Even if your application has the “all powerful” and completely unnecessary Windows Logo certification, it still offers nothing to you as a user other than the reassurance that the person/s developing the software has allot of spare cash.
February 24th, 2008 at 10:34 am
Bob Said:
> To be honest, I have always thought that digitally signing was merely a way of generating more revenue.
I have to agree. I’ve heard the argument of how it’s all designed to protect users from malicious software, and that’s all well and good as far as that goes — but since Vista, and most mobile OSes, don’t offer a way for users to say “okay, I understand the risk, I accept full responsibility, please go ahead and run this unsigned application without restrictions, and never bother me again when I try to run this application”… That makes it pretty clear it’s just a racket initiated by VeriSign and the like, and happily endorsed by Microsoft.
May 17th, 2008 at 9:03 am
This is not a bug.
The first screen shot shows that Windows doesn’t trust the identity contained in the certificate. In other words, “I can read this, but I don’t know if I should trust the person who wrote it.”
The second screen shot just shows that the certificate is well-formed, that Windows can understand the information contained within it. It says nothing about what Windows will do with that information.
Who did you did you pay to sign the certificate for you? If they’re not someone with a well-established reputation, then I don’t WANT my computer to automatically trust them.
It’s just like how web browsers automatically trust SSL certificates signed by Thawte or Verisign, but will ask you before accepting a certificate from Andy’s Shady Overnight Certificate Company. As always, it’s a balancing act between usability and security.
May 17th, 2008 at 10:04 am
Andrew, you wrote:
> This is not a bug.
OK, there is a fine line between a bug and a feature, let’s assume for a moment that it’s a feature rather than a bug. If so, what benefit is this feature supposed to provide? As the second screen shows, the file is digitally signed, and Vista can detect that. Yet, it shows the publisher as “unidentified” on the first screen. Note also (as I mentioned in the post), that if you move the file to one a few specific folders (such as C:/Program files), Vista would magically begin to recognize the publisher. Move the file to some other folder, and it’s unidentified again.
If you can explain why they designed it that way, I would agree with you. Until then, it’s a bug. Guilty until proven innocent!
> Who did you pay to sign the certificate for you?
That particular file was signed with a Verisign certificate, but the same problem occurs with _any_ file, signed with _any_ certificate. Try it yourself and you will see.
June 1st, 2008 at 12:53 pm
I think that it occurs because of IE7 protected mode – see http://victor-youngun.blogspot.com/2008/03/internet-explorer-7-protected-mode-vs.html, it’s a guide to run firefox in protected mode, and this explains very good how the protected mode works… the prompt is because the “Download” Folder is a protected folder (level “low”) and I think it only displays at the guest, because Windows forces UAC to display certificate in normal user mode in “low” level folders, but NOT for the MUCH MORE RESTRICTED “guest” account. This would be my explanaition.
It doesn’t mean that I like it how Microsoft handels this but this would eventually explain WHY the warning appears sometimes and sometimes not.
June 1st, 2008 at 12:55 pm
sorry, in my last post there is a comma in the link, the correct link is:
http://victor-youngun.blogspot.com/2008/03/internet-explorer-7-protected-mode-vs.html
June 3rd, 2008 at 7:16 pm
Farthen: thank you for the information and explanation!