What a relay web shell does inside a Linux process

A relay-style web shell doesn’t just execute commands — it forwards attacker instructions through your server to reach other targets. On Linux, that means a single web server worker can simultaneously hold inbound request buffers, parsed header tables, outbound client buffers, and downstream response data in memory.

If disk logs are missing or intentionally removed, the process heap and request buffers are often the only remaining record of the attacker’s pivot path.

Evidence timing

Volatile artifacts are a race against allocator reuse. Acquire memory as soon as compromise is suspected; high-traffic servers can overwrite request remnants quickly.

X-Forwarded-For chain semantics for pivots

X-Forwarded-For (XFF) is typically a comma-separated list where the left-most value is the claimed origin and the right-most value is the immediate sender. In pivot relays, attackers may preserve, append, or completely forge this chain.

example-http.txt 
X-Forwarded-For: 185.220.101.47, 10.10.5.14
Trust boundary

Never treat XFF alone as authoritative. Confirm the immediate TCP peer and correlate it with the right-most XFF value before using any part of the chain for attribution or scoping.

Where headers live in Linux web servers

Apache httpd (APR pools)

Apache parses headers into APR structures allocated from request pools (apr_pool_t). Header key/value pairs are stored in an apr_table_t backed by an array of apr_table_entry_t pointers to C strings.

When the request ends, the pool is destroyed and memory is returned to the process heap allocator (commonly glibc malloc; sometimes jemalloc). Freed heap memory is generally not zeroed, so header strings can persist until overwritten.

apr-layout.c 
// Simplified APR layout (conceptual)
typedef struct { apr_array_header_t a; } apr_table_t;

typedef struct {
  char *key;   // "X-Forwarded-For"
  char *val;   // "185.220.101.47, 10.10.5.14"
} apr_table_entry_t;

Nginx (ngx_str_t + request pools)

Nginx stores request headers as ngx_table_elt_t entries where key/value are ngx_str_t (length + pointer). Strings are length-prefixed and not guaranteed to be null-terminated or adjacent in memory, so naive string greps may miss some cases.

PHP-FPM (Zend heap artifacts)

For PHP web shells, headers are exposed via $_SERVER and commonly appear as HTTP_X_FORWARDED_FOR. Zend strings are length-prefixed and stored inline after zend_string metadata; request-scoped allocations may remain readable until overwritten.

Linux memory acquisition

Acquire memory quickly and record acquisition metadata (time, operator, tool, hashes). Prefer stable process dumps (gcore) or full RAM capture (LiME).

identify-process.sh 
ps aux | grep -E 'httpd|apache2|nginx|php-fpm'
gcore.sh 
gcore -o web_dump <pid>
lime.sh 
insmod lime.ko "path=/evidence/memory.lime format=lime"
/proc caveats

Direct reads of /proc/<pid>/mem may be blocked by YAMA ptrace_scope, SELinux, or hardening. Use gcore when possible; otherwise document the control changes you make.

Extracting XFF from Linux process dumps

Start with literal header searches, then pivot to pattern-based extraction for multi-hop chains. Permissive patterns are acceptable for hunting (validate later).

strings-grep.sh 
strings -n 8 web_dump.<pid> | grep -i "X-Forwarded-For"
chain-regex.sh 
grep -aP "(\d{1,3}\.){3}\d{1,3}(,\s*(\d{1,3}\.){3}\d{1,3})+" web_dump.<pid>

For full-memory images, process-aware scanning reduces noise and helps correlate findings to specific processes.

volatility-linux.sh 
# Identify candidate processes
python3 vol.py -f memory.lime linux.pslist | grep -E 'apache|httpd|nginx|php'

# Dump process mappings (version-dependent plugin naming)
python3 vol.py -f memory.lime linux.proc.Maps --pid <pid> --dump

# Scan within a process for XFF
python3 vol.py -f memory.lime linux.yarascan.YaraScan --pid <pid> \
  --yara-rules 'rule xff { strings: $a = "X-Forwarded-For" nocase condition: $a }'

Validating the chain against TCP ground truth

XFF reconstruction is only defensible if the immediate hop is validated at the TCP layer. Correlate the right-most XFF IP with the actual socket peer recorded in kernel/connection artifacts.

live-ss.sh 
ss -tnp | grep <pid>
vol-netstat.sh 
python3 vol.py -f memory.lime linux.netstat
  1. Extract the candidate chain from heap/request buffers.
  2. Identify the actual TCP peer IP for the inbound connection.
  3. Confirm it matches the right-most XFF element (or explain why it cannot).
  4. Treat the confirmed peer as the previous pivot hop and repeat on that node.

Allocator reuse and overwrite timing on Linux

Persistence windows vary with traffic and allocator behavior. glibc malloc uses arenas and bins that frequently reuse recently freed chunks; jemalloc may keep freed chunks in per-thread caches. Neither zeroes memory on free by default.

Operationally: low-traffic internal servers can preserve request remnants much longer than busy edge servers, but no persistence is guaranteed — assume overwrite is imminent.

Defensive implications

Logging hygiene

If you log XFF, only trust values appended by proxies you control. Treat attacker-supplied XFF as untrusted input to avoid access-control bypass and rate-limit evasion.

Conclusion

On Linux, a relay-style web shell pivot often leaves the attacker’s traversal chain in process memory long after disk artifacts are gone. Recovering XFF chains from Apache/Nginx/PHP-FPM memory can rapidly expose intermediate hops — but only if validated against TCP-layer evidence.

In short: capture fast, extract broadly, validate rigorously, and iterate hop-by-hop.

$ man memory-forensics
The practice of acquiring and analyzing volatile memory (RAM) from a running or recently running system to recover evidence such as process artifacts, network connections, cryptographic keys, and in-memory strings that would not appear on disk.
$ man incident-response
The structured process of identifying, containing, eradicating, and recovering from a security event. IR procedures typically include evidence preservation, triage, root-cause analysis, and post-incident review.
$ man webshell
A script (commonly PHP, JSP, or ASP) uploaded to a compromised web server that provides an attacker with remote code execution through HTTP requests, often used as a persistent foothold or pivot point.
$ man nginx
A high-performance, event-driven web server and reverse proxy. Nginx handles incoming HTTP requests using a non-blocking architecture and stores request headers in per-request memory pools (ngx_pool_t) allocated from the heap.
$ man apache
Apache HTTP Server (httpd) is a widely deployed open-source web server. It uses APR memory pools (apr_pool_t) to manage per-request allocations, which means request headers including X-Forwarded-For can persist in process heap after request completion.
$ man php-fpm
PHP FastCGI Process Manager -- a process management interface for PHP that runs worker processes separate from the web server. Request data including headers is handled by the Zend engine and allocated in Zend memory arenas on the heap.
$ man volatility
An open-source memory forensics framework written in Python. Volatility provides plugins for analyzing memory images across Windows, Linux, and macOS -- including process listing, network connections, heap scanning, and string extraction from volatile memory.
^ back to top