During the first quarter of 2020, Crypsis has seen a unique spike of over a dozen Citrix Application Delivery Controller (“ADC”), formerly known as NetScaler ADC, investigations. As most are aware by now, unauthenticated attackers were able to perform arbitrary code execution on these Citrix ADC systems due to a directory traversal vulnerability (CVE-2019-19781). These cases have presented many unique challenges, as few examiners have had experience with FreeBSD-based Linux investigations. The appliances themselves tend to have a very limited amount of logging due to default log retention policies, and relatively obscure file systems to boot. This is especially relevant, as these investigations continued far beyond the timeframe of the initial exploits that began in early January. Regardless, we’ve been able to successfully answer our clients’ questions and remediate these intrusions. Much has been blogged about regarding these exploits technically, and we don’t want to duplicate those efforts. For more technical information CVE-2019-19781, we recommend you read the Citrix Security Bulletin. Instead, we thought we’d share issues from our client investigations that have caused headaches and cost them many sleepless nights, as well as lessons learned that may help others resolve these attacks.
Before diving into the lessons learned, we want to share the kinds of malicious activity we’ve seen from our forensic investigations. If a company used one of these ADC appliances, chances are it has been scanned or exploited. We’ve helped clients from the healthcare industry, local governments, aviation companies, managed security service providers (MSSPs), and more. From a forensic investigations standpoint, Crypsis has seen cryptominers, system backdoors, web shells, ransomware (including the previously unseen “Ragnarok” ransomware), and payloads executed to steal credentials, private certificates, and TLS/SSL keys – all from exploiting this one vulnerability! We’ve observed some attackers even remove the vulnerable scripts themselves so others couldn’t exploit the same system.
Lesson 1: Official Citrix Mitigations Didn’t Work
Prior to releasing a patch, Citrix released official mitigation steps to prevent CVE-2019-19781 from being exploited. However, they also acknowledged that there were NetScaler software versions that wouldn’t be helped by the mitigation steps, namely, 220.127.116.11. In practice, Crypsis saw customers on other versions deploy the mitigations unsuccessfully. To their surprise days later, these same customers were notified by the FBI that they were still infected. Furthermore, there was a false sense of security even with the mitigation steps and the official patch. These mitigations and eventual patches only secured the vulnerability itself; many exploits took things a step further by installing backdoors and cryptominers as well as stealing credentials that could be used to access other systems. The damage was already done. Clients mitigated and patched, without realizing threat actors had already gained persistent access to their environment.
Lesson 2: The Feds Are Paying Attention
Several of our clients were notified of the infection on their systems by the FBI, and in two cases, this was the first the client had heard about the vulnerability. Needless to say, an unexpected call from the FBI was fairly alarming! In high-profile instances such as these where a vulnerability is widespread and being publicly exploited, the FBI (and presumably other agencies) can identify and monitor known servers and network traffic to known-bad IP addresses. In our cases, the FBI notifications served only to let the client know they had a problem. They did not detail specific Indicators of Compromise (IOCs) or other technical information.
Lesson 3: Attackers Can Move from Windows to Linux
It’s relatively rare in incident response investigations to find attackers moving laterally across platforms, but Crypsis saw this in a few instances. In one case, the attackers entered the network through CVE-2019-19781, then moved laterally via a Server Messaging Block (SMB) vulnerability (exploited by EternalBlue) to deploy a Windows ransomware variant. Well, there’s another lesson learned from this – patch deployment is important on not just these Citrix ADC appliances, but also on all Windows platforms, especially older operating systems. Keep up with security and patch management for all systems in your environment.
In other instances, we’ve found attackers used credentials stolen from the Citrix appliance and then accessed other Windows systems remotely. Many initial payloads from these exploits captured the contents of the “ns.conf” file; while only hashed passwords and keys were stored in there, open source tools were able to easily crack them.
Lesson 4: Monitoring NetScalers (and Other Infrastructure) Is Hard
Many of our clients had Managed Security Service Providers (MSSPs) monitoring their endpoints and servers and were disappointed to find out that this monitoring did not identify NetScaler exploitation. Likely, this is due to the fact that most Endpoint Detection and Response (EDR) tools don’t support Linux (or FreeBSD specifically). If they do, the support they offer is often limited. Fortunately, from an investigative perspective, most clients had firewalls with good logging in front of the NetScalers to help piece together what happened. Unfortunately, many of them were unaware that their infrastructure had been exploited for several weeks. None of our clients had a Web Application Firewall (WAF) or other heuristic-based detection or alerting in front of their NetScalers.
For those facing the risk of CVE-2019-19781 now or in the future, we hope that our shared experience and knowledge about critical lessons learned from our investigations can be used to further harden networks and make future zero-day exploits less likely to succeed.
Additional Resource: Mitre