noexecute (NXE), DEP and PAE in XP with SP2[]

Note: CoLinux version 0.7.1 and newer runs without changes in XP and Vista now. Problem with DEP/NX is solved.

On Windows XP with SP2, coLinux versions before 0.7.1 crash your system. This happens on new CPU's with hardware support for noexecute.

To solve this problem, you should modify your file C:\BOOT.INI, the line after operating systems and change options to /noexecute=AlwaysOff /NOPAE. This has no side effect for your running system (except breaking important protection against security exploits). All your other programs will continue to work. Your PC runs as if the processor does not have this feature.

Modifired BOOT.INI:

[boot loader]
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="MS Windows XP" /noexecute=AlwaysOff /NOPAE /fastdetect

Windows Vista does not use BOOT.INI. Instead, the bcdedit command line tool must be used:

 bcdedit /set nx AlwaysOff
 bcdedit /set pae ForceDisable

Here is an overview where you will need to change boot.ini.

Vendor Type 1,5 Speed 1 RAM 1 OS 1 PAE 2 DEP 3 BOOT.INI changed
AMD AthlonXP+2600 1.9GHz 512MB XP-Prof, SP2 no no /noexecute=OptIn no
AMD Athlon 64 3000+ 2.0GHz 1GB XP-Pro, SP2 yes yes /noexecute=OptOut yes 4
Intel Centrino (M758) 1.5GHz 1.5GB XP, SP2 yes yes /noexecute=OptIn no
Intel Centrino (M778) 1.6GHz 1.5GB XP, SP2 yes yes /noexecute=OptIn no
Intel Pentium 4M 1.8GHz 480MB XP-Home, SP1 no such option in SP1
Intel Xeon (HT) 3.2GHz 2GB XP, SP2 yes yes /noexecute=AlwaysOff /PAE yes
Intel Pentium 4 CPU 630 3.0 Ghz 1GB XP-Corporate, SP2 yes yes /noexecute=AlwaysOff /PAE yes
  1. You can get this information from your running Windows with Windows keys Menu + Pause, this opens the system overview.
  2. Current state of PAE (Physical Address Extension) will show on general overview. {de} Picture
  3. Hardware support for DEP you find in Advanced tab, Performance Settings, last tab Data Execution Prevention. Picture
  4. Worked with default (/noexecute=OptIn), too.
  5. Tool from intel to gets model number
  • This describes, why DEP is good or not
  • I'm not sure, but feel it is typical only on Intel CPU (please correct me, if not). On this side you will find the support "Execute Disable Bit" in the last right tab: HenryNe
  • Both current AMD and Intel processors support this; all 64-bit capable processors (from AMD and Intel) support it, as do select 32-bit processors (e.g. Intel's Core Solo, Core Duo processors). Disabling DEP is bad, as it's an extremely useful security measure. -- DrPizza
  • A per program selection (OptOut) for only the colinux-daemon.exe doesn't help under colinux. The bug is inside the hardware driver linux.sys, and this you can not add to the list of exclusion for DEP. We need a good assembler guru with background about differences between AMD and Intel NX handling. The question is: What is different in Intel 'Xeon' for handling NX and PAE ? What is not exactly compatible to the AMD processor? The source is in file src/colinux/arch/i386/antinx.c HenryNe
  • Problem solved as "flush tlb" in Snapshots (0.8.0-20070302, 0.7.1-20070303 and newer). The difference on such CPU was the cache for page tables. The pte was not updated from memory to CPU after changing the NX bit. We force it with a reload of all page tables. Read more in devel mailing list or SF. This changes goes into the coLinux mainline now. -- HenryNe