Earlier this afternoon, my server was upset. At 15:57
, a duo of IP addresses begun making rapid and repeated POST requests to an auxiliary component of WordPress, forcing apache to begin consuming significant amounts of system memory. Disappointingly this went undetected, and less than half an hour later, at 16:24
, the system ran out of memory, invoked the OOM killer and terminated mysqld
. Thus at 16:24
, denial of service to all applications requiring access to a database was successful.
Although the server dutifully restarted mysqld
less than a minute later, the attack continued. Access to apache
was denied intermittently (by virtue of the number of requests) and the OOM killer terminated mysqld
again at 16:35
. The database server daemon was respawned once more, only to be killed just short of half an hour later at 17:03
.
It wasn’t until 17:13
that I was notified of an issue, by means of a Linode anomaly notification, disk I/O had been unusually high for a two hour period. I was away from my terminal but used my phone to check my netdata
instance. Indeed I could confirm a spike in disk activity but it appeared to have subsided. I had run some scripts and updates (which can occasionally trigger these notifications) in the previous two hours so assumed causation and dismissed the notification. Retrospectively, it would be a good idea to have some sort of check list to run through upon receipt of such a message, even if the cause seems obvious.
The attack continued for the next hour and a half, maintaining denial of the mysqld
service (despite the respawner’s best effort), at 18:35
(two and a half hours after the attack began) I returned from the field to my terminal and decided to double check the origin of the high disk I/O. I loaded the netdata
visualiser (apache
seemed to be responsive) and load seemed a little higher than usual. Disk I/O was actually higher than usual, too. It would seem that I had become a victim of y-axis scaling; the spike I had dismissed as a one-off burst in activity earlier had masked the increase in average disk I/O. Something was happening.
I checked system memory, we were bursting at the seams. The apache
process was battling to consume as much memory on the system as possible. mysqld
appeared to be in a state of flux, so I tried to reach database backed applications; Phabricator, and my blog – both returned some form of upset “where is my database” response. I opened the syslog
and searched for evidence that the out of memory killer had been swinging its hammer. At this point I realised this was a denial of service.
I located the source of the high disk I/O when I opened the apache
access log. My terminal spewed information on POST
requests to xmlrpc.php
aimed at two WordPress sites hosted on my server. I immediately added iptables
rules for both IP addresses, and two different IPs from the same block took over the attack. I checked the whois
and discovered all the origin IPs were in the same assigned /24
block, so I updated iptables
with a rule to drop traffic from the whole block. The requests stopped and I restarted the seemingly mangled mysqld
process.
I suspect the attack was not aimed at us particularly, but rather the result of a scan for WordPress sites (I am leaning towards for the purpose of spamming). However I was disappointed in my opsec-fu, not only did I prevent this from happening, but I failed to stop it happening for over two hours. I was running OSSEC
, but any useful notifications failed to arrive in time as I had configured messages to be sent to a non-primary address that GMail must poll from intermittently. A level 12 notification was sent 28 minutes after the attack started as soon as the OOM was invoked for the first time, but the message was not pulled to my inbox until after the attack had been stopped.
The level of traffic was certainly abnormal and I was also frustrated that I had not considered configuring fail2ban
or iptables
to try and catch these sort of extreme cases. Admittedly, I had dabbled in this previously, but struggled to strike a balance with iptables
that did not accidentally ban false positives attempting to use a client’s web application. Wanting to combat this happening in future, I set about to implement some mitigations:
Mitigation Implementation
Configure a crude fail2ban jail for apache DOS defence
My first instinct was to prevent ridiculous numbers of requests to apache
from the same IP being permitted in future. Naturally I wanted to tie this into fail2ban
, the daemon I use to block access to ssh
, the mail servers, WordPress administration, and such. I found a widely distributed jail configuration for this purpose online but it did not work; it didn’t find any hosts to block. The hint is in the following error from fail2ban.log
when reloading the service:
1 2 3 |
fail2ban.jail : INFO Creating new jail 'http-get-dos' ... fail2ban.filter : ERROR No 'host' group in '^ -.*GET' |
The regular expression provided by the filter (failregex
) didn’t have a ‘host’ group to collect the source IP with, so although fail2ban
was capable of processing the apache
access.log
for lines containing GET
requests, all the events were discarded. This is somewhat unfortunate considering the prevalence of the script (perhaps it was not intended for the combined_vhost
formatted log, I don’t know). I cheated and added a CustomLog
to my apache
configuration to make parsing simple whilst also avoiding interference with the LogFormat
of the prime access.log
(whose format is probably expected to be the default by other tooling):
1 2 |
LogFormat "%t [%v:%p] [client %h] \"%r\" %>s %b \"%{User-Agent}i\"" custom_vhost CustomLog ${APACHE_LOG_DIR}/custom_access.log custom_vhost |
The LogFormat
for the CustomLog
above encapsulates the source IP in the same manner as the default apache
error.log
, with square brackets and the word “client”. I updated my http-get-dos.conf
file to provide a host group to capture IPs as below (I’ve provided the relevant lines from jail.local
for completeness):
I tested the configuration with fail2ban-regex
to confirm that IP addresses were now successfully captured:
1 2 3 4 5 6 7 8 9 |
$ fail2ban-regex /var/log/apache2/custom_access.log /etc/fail2ban/filter.d/http-get-dos.conf [...] Failregex |- Regular expressions: | [1] \[[^]]+\] \[.*\] \[client <HOST>\] "GET .* | `- Number of matches: [1] 231 match(es) [...] |
It works! However when I restarted fail2ban
, I encountered an issue whereby clients were almost instantly banned when making only a handful of requests, which leads me to…
How to badly configure fail2ban
This took some time to track down, but I had the feeling that for some reason my jail.conf
was not correctly overriding maxretry
– the number of times an event can occur before the jail action is applied, which by default is 3
. I confirmed this by checking the fail2ban.log
when restarting the service:
1 2 3 |
fail2ban.jail : INFO Creating new jail 'http-get-dos' ... fail2ban.filter : INFO Set maxRetry = 3 |
Turns out, the version of the http-get-conf
jail I had copied from the internet into my jail.conf
was an invalid configuration. fail2ban
relies on the Python ConfigParser
which does not support use of the #
character for an in-line comment. Thus lines such as the following are ignored (and the default is applied instead):
1 2 3 |
maxretry = 600 # 600 attempts in findtime = 30 # 30 seconds (or less) |
Removing the offending comments (or switching them to correctly-styled inline comments with ‘;’) fixed the situation immediately. I must admit this had me stumped and seems pretty counter-intuitive especially as fail2ban
doesn’t offer a warning or such on startup either. But indeed, it appears in the documentation, so RTFM, kids.
Note that my jail.local
above has a jail for http-post-dos
, too. The http-post-dos.conf
is exactly the same as the GET counterpart, just the word GET
is replaced with POST
(who’d’ve thought). I’ve kept them separate as it means I can apply different rules (maxretry
and findtime
) to GET
and POST
requests. Note too, that even if I had been using http-get-dos
today, this wouldn’t have saved me from denial of service, as the requests were POST
s!
Relay access denied when sending OSSEC notifications
As mentioned, OSSEC
was capable of sending notifications but they were not delivered until it was far too late. I altered the global ossec.conf
to set the email_to
field to something more suitable, but when I tested a notification, it was not received. When I checked the ossec.log
, I found the following error:
1 2 |
ossec-maild(1223): ERROR: Error Sending email to xxx.xxx.xxx.xxx (smtp server) |
I fiddled some more and in my confounding, located some Relay access denied
errors from postfix
in the mail.log
. Various searches told me to update my postfix
main.cf
with a key that is not used for my version of postfix
. This was not particularly helpful advice, but I figured from the ossec-maild
error above that OSSEC
must be going out to the internet and back to reach my SMTP server and external entities must be authorised correctly to send mail in this way. To fix this, I just updated the smtp_server
value in the global
OSSEC
configuration to localhost
:
1 2 3 4 5 6 7 8 |
<ossec_config> <global> <email_notification>yes</email_notification> <email_to>someone@example.org</email_to> <smtp_server>localhost</smtp_server> <email_from>ossecm@example.org</email_from> </global> ... |
Deny traffic to xmlrpc.php entirely
WordPress provides an auxiliary script, xmlrpc.php
which allows external entities to contact your WordPress instance over the XML-RPC
protocol. This is typically used for processing pingbacks (a feature of WordPress where one blog can notify another that one of its posts has been mentioned), via the XML-RPC pingback API, but the script also supports a WordPress API that can be used to create new posts and the like. For me, I don’t particularly care about pingback notifications and so can mitigate this attack in future entirely by denying access to the file in question in the apache
VirtualHost
in question:
1 2 3 4 5 6 7 |
<VirtualHost> ... <files xmlrpc.php> order allow,deny deny from all </files> </VirtualHost> |
tl;dr
Timeline
1557 (+0'00")
: POSTs aimed atxmlrpc.php
for two WordPressVirtualHost
begin1624 (+0'27")
:mysqld
terminated by OOM killer1625 (+0'28")
:OSSEC
Level 12 Notification sent1625 (+0'28")
:mysqld
respawns but attack persists1635 (+0'38")
:mysqld
terminated by OOM killer1636 (+0'39")
:mysqld
respawns1700 (+1'03")
:OSSEC
Level 12 Notification sent1703 (+1'06")
:mysqld
terminated by OOM killer1713 (+1'16")
: Disk IO 2-Hour anomaly notification sent from Linode1713 (+1'16")
: Linode notificationX-Received
and acknowledged by out of office sysop1835 (+2'38")
: Sysop login,netdata
accessed1837 (+2'40")
:mysqld
terminated by OOM killer, error during respawn1839 (+2'42")
:iptables
updated to drop traffic from IPs, attack is halted briefly1840 (+2'43")
: Attack continues from new IP,iptables
updated to drop traffic from block1841 (+2'44")
: Attack halted, load returns to normal,mysqld
service restarted1842 (+2'45")
: AllOSSEC
notificationsX-Received
after poll from server
Attacker
POST
requests originate from IPs in an assigned/24
blockwhois
record served by LACNIC (Latin America and Caribbean NIC)- Owner company appears to be an “Offshore VPS Provider”
- Owner address and phone number based in Seychelles
- Owner website served via CloudFlare
- GeoIP database places attacker addresses in Chile (or Moscow)
traceroute
shows the connection is located in Amsterdam (10ms away fromvlan3557.bb1.ams2.nl.m24
) – this is particularly amusing considering thewhois
owner is an “offshore VPS provider”, though it could easily be tunneled via Amsterdam
Suspected Purpose
- Spam: Attacker potentially attempting to create false pingbacks (to link to their websites) or forge posts on the WordPress blogs in question
- Scan hit-and-run: Scan yielded two
xmlrpc.php
endpoints that could be abused for automatic DOS
Impact
- Intermittent
apache
stability for ~3 hours - Full service denial of
mysql
for ~2.25 hours - Intermittent disruption to email for ~2 hours
Failures
- No monitoring or responsive control configured for high levels of requests to
apache
OSSEC
configured to deliver notifications to non-primary address causing messages that would have prompted action much sooner to not arrive within actionable timeframe- Failed to recognise (or consider) disk I/O anomaly message as a red herring for something more serious
- Forgetting that the attack surface for WordPress is always bigger than you think
Positives
- Recently installed
netdata
instance immediately helped narrow the cause down toapache
based activity - Attack mitigated in less than five minutes once I actually got to my desk
Mitigations
OSSEC
reconfigured to send notifications to an account that does not need to poll from POP3 intermittently- Added simple
GET
andPOST
jails tofail2ban
configuration to try and mitigate such attacks automatically in future - Drop traffic to offending WordPress script to reduce attack surface
- Develop a check list to be followed after receipt of an anomaly notification
- Develop a healthy paranoia against people who are out to get you and be inside your computer (or make it fall over)
- Moan about WordPress
Mitigation Tips and Gotcha’s
- Set your
OSSEC
notificationsmtp_server
tolocalhost
to bypassrelay access denied
errors - Make use of
fail2ban-regex <log> <filter>
to test your jails - NEVER use
#
for inline comments infail2ban
configurations, the entire line is ignored - If you are protecting yourself from
GET
attacks, have you forgottenPOST
?