Windows NT (and successive operating system versions) expose a set of API functions and structures for debugging a running process. This article shows how these can be accessed from Visual Basic (version 5 or 6). It is recomended that the article Inside the executable: an introduction to the Portable Executable format for VB programmers be read in conjunction with this article.
Getting a Process to Debug
There are two ways to get a process to debug. Either (a) attach a debugger to a process that is already running or (b) start a new process with a debugger attached to it.
Starting a new process with a debugger attached
To start a process, you can use the CreateProcess API call:
Private Declare Function CreateProcess Lib _ "kernel32" Alias "CreateProcessA" (ByVal lpApplicationName As String, _ ByVal lpCommandLine As String, _ = ByVal lpProcessAttributes As Long,_ ByVal lpThreadAttributes As Long,_ ByVal bInheritHandles As Long, _ = ByVal dwCreationFlags As ProcessCreationFlags, _ ByVal lpEnvironment As Long, _ ByVal lpCurrentDirectory As String, _ lpStartupInfo As STARTUPINFO,_ lpProcessInformation As PROCESS_INFORMATION) _ As Long
In addition to the functionality of the Shell command, this allows you to specify additional flags that affect how the process is created:
Public Enum ProcessCreationFlags DEBUG_PROCESS = &H1 DEBUG_ONLY_THIS_PROCESS = &H2 CREATE_SUSPENDED = &H4 DETACHED_PROCESS = &H8 CREATE_NEW_CONSOLE = &H10 NORMAL_PRIORITY_CLASS = &H20 IDLE_PRIORITY_CLASS = &H40 HIGH_PRIORITY_CLASS = &H80 REALTIME_PRIORITY_CLASS = &H100 CREATE_NEW_PROCESS_GROUP = &H200 CREATE_UNICODE_ENVIRONMENT = &H400 CREATE_SEPARATE_WOW_VDM = &H800 CREATE_SHARED_WOW_VDM = &H1000 CREATE_FORCEDOS = &H2000 CREATE_DEFAULT_ERROR_MODE = &H4000000 CREATE_NO_WINDOW = &H8000000 End Enum
To start the process with an attached debugger, you specify the flags DEBUG_PROCESS + DEBUG_ONLY_THIS_PROCESS.
Attaching a debugger to an existing process
To attach a debugger to a process that is already running, you need to obtain a handle to it and then attach a debugger using the DebugActiveProcess API call.
Private Declare Function DebugActiveProcess Lib "kernel32" _ (ByVal dwProcessId As Long) As Long
The Debug Loop
Once you have attached your debugger to the process, you need to go into a debug loop. This consists of waiting for a debug event, processing the event when it comes in, and then allowing the debugee to continue.
Waiting for a debug event to occur
To wait for a debug event, you call the WaitForDebugEvent API call:
Private Declare Function WaitForDebugEvent Lib "kernel32" _ (lpDebugEvent As DEBUG_EVENT_BUFFER, _ ByVal dwMilliseconds As Long) As Long
This will return TRUE when a debug event has occurred and filled out the DEBUG_EVENT_... structure, which depends on what event occurred but always starts with a DEBUG_EVENT_HEADER:
Private Type DEBUG_EVENT_HEADER dwDebugEventCode As DebugEventTypes dwProcessId As Long dwThreadId As Long End Type
Processing the debug event
How you deal with a debug event depends, naturally enough, on what event occurred. The event types are:
Public Enum DebugEventTypes EXCEPTION_DEBUG_EVENT = 1& CREATE_THREAD_DEBUG_EVENT = 2& CREATE_PROCESS_DEBUG_EVENT = 3& EXIT_THREAD_DEBUG_EVENT = 4& EXIT_PROCESS_DEBUG_EVENT = 5& LOAD_DLL_DEBUG_EVENT = 6& UNLOAD_DLL_DEBUG_EVENT = 7& OUTPUT_DEBUG_STRING_EVENT = 8& RIP_EVENT = 9& End Enum
This debug event is thrown whenever an exception occurs in the application being debugged. For example, if there was code in the application that was attempting to divide by zero, you would get an EXCEPTION_DEBUG_EVENT. The buffer that is passed back for this event is:
Public Enum ExceptionCodes EXCEPTION_GUARD_PAGE_VIOLATION = &H80000001 EXCEPTION_DATATYPE_MISALIGNMENT = &H80000002 EXCEPTION_BREAKPOINT = &H80000003 EXCEPTION_SINGLE_STEP = &H80000004 EXCEPTION_ACCESS_VIOLATION = &HC0000005 EXCEPTION_IN_PAGE_ERROR = &HC0000006 EXCEPTION_INVALID_HANDLE = &HC0000008 EXCEPTION_NO_MEMORY = &HC0000017 EXCEPTION_ILLEGAL_INSTRUCTION = &HC000001D EXCEPTION_NONCONTINUABLE_EXCEPTION = &HC0000025 EXCEPTION_INVALID_DISPOSITION = &HC0000026 EXCEPTION_ARRAY_BOUNDS_EXCEEDED = &HC000008C EXCEPTION_FLOAT_DENORMAL_OPERAND = &HC000008D EXCEPTION_FLOAT_DIVIDE_BY_ZERO = &HC000008E EXCEPTION_FLOAT_INEXACT_RESULT = &HC000008F EXCEPTION_FLOAT_INVALID_OPERATION = &HC0000090 EXCEPTION_FLOAT_OVERFLOW = &HC0000091 EXCEPTION_FLOAT_STACK_CHECK = &HC0000092 EXCEPTION_FLOAT_UNDERFLOW = &HC0000093 EXCEPTION_INTEGER_DIVIDE_BY_ZERO = &HC0000094 EXCEPTION_INTEGER_OVERFLOW = &HC0000095 EXCEPTION_PRIVILEGED_INSTRUCTION = &HC0000096 EXCEPTION_STACK_OVERFLOW = &HC00000FD EXCEPTION_CONTROL_C_EXIT = &HC000013A End Enum Public Enum ExceptionFlags EXCEPTION_CONTINUABLE = 0 EXCEPTION_NONCONTINUABLE = 1 '\\ Noncontinuable exception End Enum Private Type DEBUG_EXCEPTION_DEBUG_INFO Header As DEBUG_EVENT_HEADER ExceptionCode As ExceptionCodes ExceptionFlags As ExceptionFlags pExceptionRecord As Long ExceptionAddress As Long NumberParameters As Long ExceptionInformation (EXCEPTION_MAXIMUM_PARAMETERS) As Long dwFirstChance As Long End Type
The exception flags tell you whether it is possible to resume from the exception.