My system was clearly under attack. But by whom? Then I remembered: I had asked SPI Dynamics to unleash its website auditing tool, WebInspect, against my home server. Not just any auditing tool, WebInspect is specifically designed for penetration testing Web-based applications. The program uses a Web spider to map out every page on the server, examines each page for Web errors that an outsider could exploit, and then tries to exploit them.

"Go ahead and whack my system," I had told the company two days before the incident. And so it did.

Now if this had been a normal attack, I would have responded by setting up a rule blocking my server from the attacker's IP address. But not this time, because I wanted SPI Dynamics to use its tool against my website—I wanted to know if I had any vulnerabilities. What I hadn't expected was that the tool would find the one script on my Web server that required 10 CPU seconds to run, and then repeatedly run that script 30 times a minute, firing off each new request long before the previous one had a chance to finish. That's why the load on my server had spiked.

Accidents like that have given penetration testing a bad name in the past. The goal of penetration testing is to find vulnerabilities in production systems so that those vulnerabilities can be patched. But if the person conducting the test doesn't apply extreme care, the test itself can become destructive. Such situations quickly escalate from being mere embarrassments to becoming full-fledged money-losing events. Oops.

Penetration testing goes back decades. In the 1970s, members of the U.S. military set up "tiger teams" or "red teams" with hotel rooms filled with communications equipment. Their goal was to see if they could break into sensitive computer systems or communications links run by other groups inside the military. A few security-conscious companies outside the defense establishment started pen-testing in the 1980s. Sometimes the attacks were physical, sometimes they relied on social engineering, and sometimes they were purely electronic. Alas, they were almost always effective.

One company that was hired to penetrate the network of another accidentally broke into a third company. Hilarity—and a lawsuit—ensued. You have to trust your penetration testers. But they have to trust you too.

I've never been a particularly big fan of pen-testing for the simple reason that a negative report from the red team doesn't always convey useful information. Certainly, if the testers find a way to break into your system and they tell you, then you can fix the hole. But what if they don't find a way to break in? It could mean there is no hole to be found—or it could mean the testers just didn't find it. In the worst case, the pen-testers do find a way in, but they don't tell you about it. Instead, they sell the information to your competitors, or they leak it to the press, or they just use it themselves for their own personal enrichment.

How often do such nightmare problems happen? Nobody is sure. In one case I heard, a company that was hired to penetrate the network belonging to company A accidentally broke into the network of company B, which had set up a leased line with company A for sharing special information. Hilarity—and a lawsuit—ensued. In another case, the penetration testing company outsourced to one of its friends. Unfortunately for everybody, those friends turned out to be real, live criminals.

Clearly, you have to trust your penetration testers. But they have to trust you too, which is another delicate aspect of this all-but-unsavory auditing practice. Before SPI Dynamics would whack my server, it had me sign and fax back a piece of paper authorizing the company to test my system. In the trade, such a paper is called a "get out of jail free card"—after all, breaking into a computer without permission is in many jurisdictions a prosecutable offense.

So what's the problem? SPI Dynamics assumed that the permission was mine to give! It's easier to imagine that a person interested in the secrets of a company would pose as an employee of that company than hire an outside team to conduct a penetration test. (In fact, that's loosely the plot of the movie Hackers.)

Most legitimate penetration testing today is done by allegedly white hat or gray hat computer hackers who closely monitor computers underground, download copies of all the latest tools and attacks, and use them like individual irons in a golf bag for hacking into target systems. Is the target an older Linux system? If so, the testers might try their UW IMAP buffer overflow attack. Does the network have a router? Then they might see if the machine's default password has been changed.

WebInspect and a product from Core Security Technologies called Core Impact are part of a new generation of penetration testing tools—tools that can be thought of as "attack caddies." Like traditional vulnerability scanners, they have a database of operating systems and known vulnerabilities for which to check. But when they find a vulnerability, they then figure out how to exploit it and test to see if that vulnerability can be used to leverage additional authority (a technique called privilege escalation).

But back to my system. Apparently, the large number of log entries and corresponding error reports from the WebInspect session had caused my mail and log partition to overflow (it had previously been at 95 percent capacity). Fixing that was easy: I moved my Web logs to a different disk partition. Mail started flowing again, but slowly.

The high load was caused by a different problem: WebInspect had discovered that the script I had written for displaying a photo album allowed any directory on my server to be indexed, and whenever a new directory was indexed, the photo album program tried to create tiny little thumbnails in the directory. I fixed that problem by modifying the script so that requests outside the photo album directory would be summarily rejected. I should have done that when I first wrote the program, of course.

With the second fix, the load on my system promptly dropped back down to less than 1.0. I then called the folks at SPI Dynamics to get their full report. As it turned out, the company had discovered yet another vulnerability: A script I had written was vulnerable to so-called parameter-based cross-site scripting. In other words, I had a script on my site that would allow a user to type in HTML and send that same HTML right back to the Web browser. This can be used to hijack an authenticated Web session from the browser on the same site. In my case, it would have allowed a sufficiently skilled and motivated attacker to possibly hijack my Web browser and muck with my MailMan mailing list configurations.

Overall, I didn't think that this HTML vulnerability was a particularly big deal. In fact, there was a much bigger vulnerability, but the automated testing tool didn't find it. Probably just as well!

When most people think about penetration testing, they think about top-rank "ethical hackers," perhaps straight out of the Air Force Information Warfare Center, who might have once been on the dark side but now apply their skills for good.

But what do you do if you are just a regular guy who wants to do some basic penetration testing? In that case, you might need some help.

Core Impact is a full-blown penetration workbench that lets you size up a remote system, analyze the system with a variety of data reconnaissance tools, and then penetrate the system using a variety of exploits that come bundled with the program. (Additional exploits are available to customers who purchase support.)

Oftentimes in the world of penetration testing, the penetration tester will break into one machine, only to discover that a second machine can be reached from the first—and perhaps a third from the second and so on. Some attackers call this "network weaving" or "leapfrogging." It's a powerful technique that can be used, for example, to penetrate behind a company's firewall by successfully breaking into a "trusted" Web server located on the company's DMZ.

Core Impact has been specially designed to make leapfrogging child's play. Once a system is penetrated, Core Impact downloads a small agent into the memory of the compromised process. That agent then allows the penetrated system to be used as an attack point against other machines on the target network, allowing the operator to leapfrog his way in. Since the agent resides solely in the computer's memory, there's no lasting damage to the penetrated machine—and usually no evidence of penetration either. So while Core Impact is a good tool for penetration testing, it's a great tool for espionage as well.

No matter whether you have your penetration testing done with automatic tools, with an outsourcing company or with your own insiders, it's best to be ever vigilant. The results described in a penetrating testing report are a step-by-step checklist of how to break into your system. As such, you're best off making sure it doesn't fall into the wrong hands.


Simson Garfinkel, CISSP, is a technology writer based in the Boston area. He is also CTO of Sandstorm Enterprises, an information warfare software company. He can be reached at machineshop@cxo.com.

> Storing the Mind