Countermeasures
The geographic dispersal of botnets means that each recruit must be individually identified/corralled/repaired and limits the benefits of filtering. Passive OS fingerprinting can identify attacks: network administrators can configure newer firewall equipment to take action on a botnet attack by using information obtained from passive OS fingerprinting. The most serious preventive measures use rate-based intrusion prevention systems using specialized hardware. A network based intrusion detection system (NIDS) can be effective. NIDS monitors a network: it sees a protected host in terms of external interfaces to the rest of the network, rather than as a single system, and gets results by network packet analysis.
Some botnets use free DNS hosting services such as DynDns.org, No-IP.com, and Afraid.org to point a subdomain towards an IRC server that harbors the bots. While these free DNS services do not themselves host attacks, they provide reference points (often hard-coded into the botnet executable). Removing such services can cripple an entire botnet. The botnet community refers to such efforts as "nullrouting", because the DNS hosting services usually re-direct the offending subdomains to an inaccessible IP address. Some botnets implement custom versions of well-known protocols. The implementation differences can be used for detection of botnets. For example, Mega-D features a slightly modified SMTP protocol implementation for testing spam capability. Bringing down the Mega-D's SMTP server disables the entire pool of bots that rely upon the same SMTP server.
The botnet server structure mentioned above has inherent vulnerabilities and problems. For example, finding one server with one botnet channel can often reveal the other servers, as well as their bots. A botnet server structure that lacks redundancy is vulnerable to at least the temporary disconnection of that server. However, recent IRC server software includes features to mask other connected servers and bots, eliminating that approach.
Security companies such as Afferent Security Labs, Symantec, Trend Micro, FireEye, Umbra Data and Damballa have announced offerings to counter botnets. Norton AntiBot was aimed at consumers, but most target enterprises and/or ISPs. Host-based techniques use heuristics to identify bot behavior that has bypassed conventional anti-virus software. Network-based approaches tend to use the techniques described above; shutting down C&C servers, nullrouting DNS entries, or completely shutting down IRC servers.
Some newer botnets are almost entirely P2P, with command-and-control embedded into the botnet rather than relying on C&C servers, avoiding any single point of failure. Commanders can be identified just through secure keys, and all data except the binary itself can be encrypted. For example, a spyware program may encrypt all suspected passwords with a public key hard coded or distributed with the bot software. Only with the private key, which only the commander has, can the data captured by the bot be read.
Some botnets are capable of detecting and reacting to attempts to figure out how they work. A large botnet that learns it is being studied can even attack those studying it.
Researchers at Sandia National Laboratories are analyzing botnets behvior by simultaneously running one million Linux kernels as virtual machines on a 4,480-node high-performance computer cluster.
Read more about this topic: Botnet