It’s well known in the security industry that attackers “live off the land” – they make use of whatever tools and infrastructure they find on the network to avoid detection. It’s an effective technique for both malicious actors as well as penetration testers.
How can we leverage the concept of ‘living off the land’ as incident responders?
It can be easy to fall into the trap of working cases the same way we’ve worked similar cases in the past – by relying on the same familiar artifacts, without giving much thought to what else may be available. It’s only human nature to stick with what we know, and what has worked previously.
But what happens when that doesn’t work? A case in point:
- A client had an exposed Elasticsearch database – tens of TBs across multiple Linux servers, publicly exposed for nearly a year
- Anybody who found the port could query the data, which contained PII
- The port was exposed following a config change
- There were no firewalls, load balancers, or other infrastructure in place to show access to the database
- The only Elasticsearch logging configured was for errors – not successful queries
Many investigators would stop here; the evidence we usually rely on simply isn’t available. But what about tools on the system that may be useful?
Forensic Investigators Use “Living Off the Land” Methodology
During the digital investigation process, Crypsis identified a third-party utility used to track server metrics: DataDog. DataDog tracks things like processor usage, memory usage, disk errors, and daily network bandwidth (bytes transmitted and received). Logs are retained, at least for this client, for 15 months. Not only did that cover the window of exposure, it gave us a baseline of what “normal” looks like for six months prior to the exposure.
Using this baseline, we were able to show that the network usage remained consistent during the window of exposure. Additionally, there was a large “spike” at the end, representing Crypsis’ queries into the Elasticsearch database to identify what data was contained in it. Presumably, an attacker would have had to do the same thing, and there were no other spikes during the period covered by DataDog.
This is circumstantial but compelling. To add further context, Crypsis was able to utilize the atop utility included in many Linux distros. Atop is a utility that monitors processes from the time they are started and tracks metrics such as disk I/O and network utilization. As Elasticsearch runs under the Java process (and was started at the same time the server rebooted, causing the exposure), the statistics tracked covered the entire window of exposure. We were able to see the following:
- Atop showed high amounts of disk write activity for the Java process – consistent with data being written into the database
- But it showed low amounts of disk read activity, which would indicate data being accessed or queried
- Similarly, the network utilization was consistent with what was seen in DataDog – raising the confidence level in those circumstantial findings
Together, these findings helped the client and their legal counsel draw conclusions regarding their data breach notification obligations.
Using utilities not normally used in forensic investigations, Crypsis was able to not only provide insight into the likelihood of a breach, but also save a company from significant disclosure and scrutiny by regulators.
Living off the land isn’t just for hackers and penetration testers; it is a useful methodology for incident responders, and in this case, helped a client have much-needed peace of mind.
At Crypsis, our daily mission is to fight cybercrime. With our team of cybersecurity experts, we help and protect our clients by defending against and responding to cyber threats. Staying ahead of the rapidly evolving threat landscape is what we do best.
Looking for more information? Contact us and we’d be happy to help you.