This article will discuss a very small utility to check if an EXE file is a console or GUI application. There are no direct API calls, which gives us this information about an application. Whenever an application is created, the EXE file stores a whole lot of information in certain format. Therefore to dig out the information it is important to understand the format of executable files. All the data structure types used for various file headers are defined in WINNT.H file supplied with Win NT OS.
Starting from Win NT 3.1, a new file format was introduced by OS called Portable Executable (PE). The PE file format draws primarily from the COFF (Common Object File Format) specification that is common to UNIX® operating systems. The entire format consists of an MS-DOS MZ header, followed by a Real-Mode Stub Program, the PE file signature, the PE file header, the PE optional header, all of the section headers, and finally, all of the section bodies.
The MS-DOS header occupies the first 64 bytes of the PE file. The contents of this header are represented by IMAGE_DOS_HEADER data structure. The e_magic field of this structure is used to identify an MS-DOS compatible file type. All MS-DOS compatible executable files set this value to 0x54AD, which represents the ASCII characters MZ. MS-DOS headers are sometimes referred to as MZ headers for this reason. The final field, e_lfanew, is a 4-byte offset into the file where the PE file header is located.
The Real-Mode Stub Program is an actual program run by MS-DOS when the executable is loaded. For an actual MS-DOS executable image file, the application begins executing here.
Starting with Windows and OS/2 executables, .EXE files were given file signatures to specify the intended target operating system. For the PE file format in Windows NT, this signature occurs immediately before the PE file header structure.
Following are the signatures defined in WINNT.H file.
#define IMAGE_DOS_SIGNATURE 0x4D5A // MZ
#define IMAGE_OS2_SIGNATURE 0x4E45 // NE
#define IMAGE_OS2_SIGNATURE_LE 0x4C45 // LE
#define IMAGE_NT_SIGNATURE 0x50450000 // PE00
Once we have the location of the file signature, the PE file follows four bytes later. The next 224 bytes in the executable file make up the PE optional header. The information corresponding to optional header is defined by IMAGE_OPTIONAL_HEADER data structure. The optional header contains most of the meaningful information about the executable image, such as initial stack size, program entry point location, preferred base address, operating system version, section alignment information, and so forth. And the information we are looking for i.e. application type. "Subsystem" field identifies the target subsystem for this executable. Each of the possible subsystem values are listed in the WINNT.H file
#define IMAGE_SUBSYSTEM_UNKNOWN 0 // Unknown subsystem.
#define IMAGE_SUBSYSTEM_NATIVE 1 // Image doesn't require a subsystem.
#define IMAGE_SUBSYSTEM_WINDOWS_GUI 2 // Image runs in the Windows GUI subsystem.
#define IMAGE_SUBSYSTEM_WINDOWS_CUI 3 // Image runs in the Windows character subsystem.
#define IMAGE_SUBSYSTEM_OS2_CUI 5 // image runs in the OS/2 character subsystem.
#define IMAGE_SUBSYSTEM_POSIX_CUI 7 // image runs in the Posix character subsystem.
#define IMAGE_SUBSYSTEM_NATIVE_WINDOWS 8 // image is a native Win9x driver.
#define IMAGE_SUBSYSTEM_WINDOWS_CE_GUI 9 // Image runs in the Windows CE subsystem.
The strategy to get to IMAGE_OPTIONAL_HEADER data structure has been implemented in OnOK function of the attached dialog based application. To test enter complete path to the EXE file.
For more information on the PE file format, you can read the SDK documentation.