Detection
The fundamental problem with rootkit detection is that if the operating system has been subverted, particularly by a kernel-level rootkit, it cannot be trusted to find unauthorized modifications to itself or its components. Actions such as requesting a list of running processes, or a list of files in a directory, cannot be trusted to behave as expected. In other words, rootkit detectors that work while running on infected systems are only effective against rootkits that have some defect in their camouflage, or that run with lower user-mode privileges than the detection software in the kernel. As with computer viruses, the detection and elimination of rootkits is an ongoing struggle between both sides of this conflict.
Detection can take a number of different approaches, including signatures (e.g. antivirus software), integrity checking (e.g. digital signatures), difference-based detection (comparison of expected vs. actual results), and behavioral detection (e.g. monitoring CPU usage or network traffic). For kernel-mode rootkits, detection is considerably more complex, requiring careful scrutiny of the System Call Table to look for hooked functions where the malware may be subverting system behavior, as well as forensic scanning of memory for patterns that indicate hidden processes.
Unix rootkit detection offerings include Zeppoo, chkrootkit, rkhunter and OSSEC. For Windows, detection tools include Microsoft Sysinternals RootkitRevealer, Avast! Antivirus, Sophos Anti-Rootkit, F-Secure, Radix, GMER, and WindowsSCOPE. Any rootkit detectors that prove effective ultimately contribute to their own ineffectiveness, as malware authors adapt and test their code to escape detection by well-used tools.
Read more about this topic: Rootkit