Lessons from a pentest run

Detection and Prevention

Over on Twitter, @TinkerSec live tweeted a pentest and created a moment thread of it.

It’s fascinating reading, and well worth reading. Even non-technical people should be able to get something out of this. I like that it’s a form of insider attack (industrial espionage by a newly hired employee? disgruntled employee? vendor allowed unaccompanied access?) rather than an external attack.

One of the things that typically comes out of an event like this is a series of action items. I could imagine the following bullet points on some powerpoint somewhere

  • User education; remind them to never give out passwords or 2FA numbers to anyone. Possible training course?
  • USB lockdown; disable storage devices totally except for people on an exception list
  • Time based One Time Passwords for IT infrastructure accounts
  • Review all devices checking for disk encryption
  • Lock support areas

You get the idea. A nice solid list of action items that all sound reasonable.

The problem with such lists is that they end up focusing on one attack path and closing that down. We can see how Tinker changed tactics during his test; one path is blocked, so try another. If we shut down the path he eventually used, would another one be found (block USB storage; use a rubber ducky instead?).

Aside: many Virtual Desktop (VDI) solutions provide some level of control here; e.g. if you log out of your VDI session then the VM is destroyed and next login is on a newly created VM, which makes unquoted path attacks harder (your code doesn’t persist between logins). Similarly thin desktops typically don’t allow random USB devices access (which I find annoying; it means I can’t use Ubikey U2F devices for 2FA).

This is not to say that prevention is a bad thing; it can raise the bar and require smarter attackers. It can prevent accidental data breaches (e.g. someone copies important data to a USB stick, loses it on the train) but you can always go too far down the rabbit hole and end up doing the equivalent of destroying a $200 drone with a $3mm missile. Your time, effort and money may be better spent elsewhere.

One such area is in detection. And, indeed, this is ultimately what caught this attack; anomalous behaviour on a desktop (base64 encoded powershell, of all things) led to the compromised machine being isolated and security called.

Intrusion Detection Systems (IDS) are the partners of IPS (Prevention). They can highlight where IPS has been circumvented or avoided. It can take place in many places (network demarcation points such as firewalls, routers; endpoints such as desktops and servers; central infrastructure tools such as scanners and integrity monitors). It should even exist at the application layer (anomalous activity).

IDS work better the more data they have so it is common to start feeding logs into a central Security Operations Center (SOC). A common form is just to log everything to Splunk, and then let the SOC deal with it.

But an IDS is pointless if there’s no knowledge behind the data. A tonne of data streams results in just noise. We need to know what is normal behaviour; what is expected; what thresholds are normal (we expect 50 failed logins a minute from a user population of 10,000; if we see 500 then there’s odd stuff happening). This requires data modelling. Tools exist to try and “self-learn” baseline activity, but these may sometimes get confused and learn the wrong thing. So teams onboarding to the SOC must work to ensure their data is understood as information. The SOC needs to be aware of all their datastreams and potentially how the interact in order to gain knowledge.

And then, finally, once suspicious activity has occured we need the ability to take action. Isolating a rogue device is good, but may not be viable everywhere (“our IDS detected suspicious activity from server X, so we shut down the network port… oh, did we just break our primary trading platform for 2 hours at the cost of $$$$? Let me pack up my desk…“) so the response needs to be tuned to the type of activity and risk to the company.

As we saw with IPS, there will always be gaps in coverage. A common refrain is “red teams have it easy; we have to close every security gap but they just need to find one.”

Tinker turns this on its head; “Blue Team has it easy. All they need to do is find ONE IoC. Red Team? We have to be perfect in every attack to not get caught.”

As with IPS, IDS coverage won’t perfect. Things will slip through.

Summary

Using tools and processes are an essential part of preventing an attacker from succeeding in the first place. However many responses to a break are focused on the immediate issue. To use an ITIL term; they are incident response focused, rather then problem management focused. Of course no one likes having a known vulnerability, so we will try and close that gap, but sometimes we need to look at the bigger picture.

First rule of security: your system will be broken into. So let’s also focus on setting up tripwires and alarm bells. A bank vault is hard to break into (IPS), but we also have alarms that ring (IDS). Detecting a breach is just as important as preventing them in the first place.