Online crime is big business. Billions of dollars are at stake and earning those dollars depends on not being caught. Obviously, criminals don’t want to be caught after the crime, but in this case I mean that the most remunerative attacks depend on secrecy. If attackers are discovered by system administrators, the game is up and a potentially lucrative source of revenue is closed.
Sometimes criminals just want a server to spray spam for as long as they can get away with, but the attacks that pose the most risk for businesses are advanced persistent threats. They’re executed over weeks and months and incentivize criminals to be as sneaky as possible.
Attackers have developed tools and techniques to infiltrate and infect servers without system administrators being aware. Sysadmins can’t take the apparent smooth functioning of their servers and the software running on them as evidence that all is well. They must implement processes to constantly monitor the condition of the code on the machines under their care.
To understand whether a server has been changed without authorization, it’s necessary to know its expected state — specific versions of the software it runs, hashes of the binary data, knowledge of its processes. The expected state must then be compared to the current state, any deviances are considered an indication of a security breach.
It’s worth mentioning that state should be monitored for deviations throughout the life of a server. There’s very little point installing software from secure sources, only to neglect security monitoring of the sort we’ll discuss for six months. To have a high confidence of a server’s trustworthiness, system administrators need to know it’s state from first install to decommissioning.
All of which begs the question, which tools can we use to understand the state of our servers?
Root kits are malware-infected versions of the binaries that a server would ordinarily be running. They’re almost impossible to spot: firstly, because they don’t betray their presence in the usual ways, and secondly, because the tools you’d use to monitor the server will also have been replaced by compromised versions.
Intrusion Detection Systems
As I said earlier, it’s important to understand the state of the server and its software stack and to be able to compare that state to the expected state. That’s exactly how intrusion detection systems like Tripwire work. In a nutshell, they take hashes of binaries in a known good state and periodically check for changes. On a stable server — ignoring legitimate updates — core tools should not change. If hashes of those binaries change, it’s a clear indication that something untoward is happening.
Malware has to do something. Otherwise it isn’t especially useful. Those actions leave traces in the server’s logs, although they’re frequently difficult to find. Log analysis is a complex subject. I suggest that novice sysadmins take a look at Log Analysis for Web Attacks: A Beginner’s Guide as a starting point.
Given the incentives that drive online criminals, it’s not enough to deploy a server and leave it alone so long as all appears well. System administrators must make active efforts to determine if their machine remains uncompromised.