Steve Gibson (Packet Attack)

Since I know that you guys have been troubled with some similar problems –
it might actually be the same style of attack — I wanted to share this
report that I wrote to notify my Washington D.C. contacts. Feel free to
share it with your Lockergnome networking guys.

As many of you know, we were the target of a new sort of attack very early
this morning. We were restored to operation after four hours of complete
outage when I characterized the attack and was able to filter a large part
of the nonsense at Verio's upstream routers. However, after 13 hours, well
into the day, the attack continues unabated, though blocked.

We were initially driven off the Internet by a flood of SYN/ACK and RST/ACK
packets – all having a source port of 179 and apparently random
destination ports. These flooding packets were apparently (and confirmed)
coming from the Internet's core routers! Our own ISP's (Verio) routers
were 'attacking' us, as were the routers of other large ISPs, a few of the
main DNS root servers, and many of the web servers belonging to YAHOO.COM.

I placed 'attacking' in quotes above, since, as we'll see below, all of
these machines were innocent bystanders in the attack. However, they were
utilized to swamp our bandwidth and force us off the Net.

Since we do not have any need for port 179 traffic – which is the Border
Gateway Protocol (BGP) used for inter-router communications – we were able
to readily block that traffic at Verio's upstream “aggregation” router.

But… the second we did that, another whole class of additional flooding
attack traffic (which had been unable to compete with the core routers
overwhelming BGP attack) showed itself for the first time. It had been
buried “beneath” the main thrust of the attack.

With the unneeded BGP protocol now blocked, we were getting SYN/ACK and
RST/ACK packet floods coming from ports 22 (SSH), 23 (TELNET), 53 (DNS) and
80 (HTTP). These packets were flooding in from hundreds of actual machines
(including many at YAHOO.COM and even one at NSA.GOV) and widely spread
across the Internet.

This second-tier of traffic was being sourced by well-connected but
similarly innocent servers.

Someone, or some group of people, have accumulated a large list of
well-connected server IPs and corresponding TCP service ports out on the
Internet. Among these are several hundred well-connected routers serving
the BGP (port 179) protocol. This list also includes many other servers
listening to other common service ports including SSH, TELNET, DNS, and HTTP.

Using full raw sockets on an unknown platform (presumably UNIX) in order to
spoof the source IP and generate isolated SYN packets, the SYN packets are
“sprayed” at each machine in their list, at that machine's corresponding
listening TCP service port. (BGP for the routers, HTTP for YAHOO. etc.)

In each case, the source IP being spoofed is the attack's target. For this
attack it was the IP of the grc.com server. The high bandwidth of the
attack, crossing our Internet connection, saturated our available bandwidth
many MANY times over.

The bandwidth of the “spray” is such that no one of these “intermediary”
target servers receives a “flooding-level” of SYN packets. In other words,
the SYN packet rate to each server is modest so that the server is never
placed into DoS by the packet originator, and no alarms are raised on that
server locally.

However, for every new SYN packet arriving at one of these servers, that
server, believing that a TCP connection is being requested and initiated,
replies to the apparent sender with a SYN/ACK packet. That SYN/ACK packet
is, of course, aimed at the APPARENT source of the SYN … in this case
grc.com.

So… grc.com is consequently subjected to a substantial Distributed
Denial of Service (DDoS) attack from many hundreds of well-meaning and
innocent Internet servers. And, significantly, none of those servers is
being inconvenienced by its role in this attack.

Unlike traditional DDoS attacks, where a high-bandwidth spoofed-SYN flood
can be traced back up the router chain towards its actual bandwidth source
(because all packets have a common destination address although a spoofed
source IP), the use of a large number of intermediate TCP servers prevents
any single “stream” from having enough bandwidth to sound any alarms. And
all of these low-bandwidth streams are coming from very different
physically isolated servers.

Conceivably, only a very few malicious and well-connected SYN-generating
machines would be required to “spray” the SYN packets out across a large
population of intermediate servers. Each of these malicious well-connected
machines may be generating packets at its maximum speed, but since those
packets are so widely and immediately distributed – rather than all being
aimed at a single traditional DoS target – none of the intermediate
recipient machines is unduly inconvenienced. Furthermore, since the
intermediate machines are widely spread across the Internet, the
SYN-source's local flood thins and spreads out quickly, even a few hops
away from its source machine. This further impedes successful SYN-source
backtracking.

This also creates a potentially more manageable architecture than the
traditional comparatively difficult to establish many-machine multi-tier
DDoS system.

And finally, there is a very good chance that the intermediate machine will
generate MULTIPLE SYN/ACK packets for every SYN packet received. Since
traditional TCP stacks will typically generate four SYN/ACK packets in
response to the receipt of a single unanswered SYN packet (believing that
their responding SYN/ACK might have been lost in transit) this system has
the potential for creating a FACTOR FOUR attack bandwidth multiplication.
(If the receiving target system can get a “RST” packet back to the
originating server, then further SYN/ACKs will be averted. However, since
many commercial enterprise-level firewalls simply drop unsolicited SYN/ACK
packets, the bandwidth multiplication effect tends to rein.)

Therefore, the use of intermediate TCP servers has the effect of
quadrupling the original input bandwidth … which they are innocently
aiming at the attack target.

(Note that the fact that we were also seeing some RST/ACK packets amid the
SYN/ACK flood tells us that the malicious hacker's IP and Server Port list
is not perfect. Those RST/ASK packets were generated in response to a
spoofed source IP SYN packet hitting a non-open service port.)

However, the list of IPs and Ports was certainly accurate enough, and it is
trivial and maintain such a list. Every “TraceRoute” across the Internet
yields a list of intermediate, probably very-well-connected routers, and
any large well-known web server farm – such as YAHOO.COM which was used
here – provides plenty of web servers for the “packet bounce”.

Any site lacking extremely comprehensive and unusually tight upstream
filtering will probably be placed into a sustained Denial of Service by the
flood of SYN/ACK and RST/ACK traffic pouring into it from many hundreds of
well-connected public servers.

This attack will NOT appear on the “radar screens” of the organizations
monitoring traditional spoofed source IP SYN floods and other similar
attacks. There will be no monitorable “backscatter” from this attack since
every IP used is for that of a valid machine.

For all of these reasons, this new style of attack creates a very potent,
highly effective, difficult to filter distributed denial of service.

The trick to defending against this attack is recognizing the very
different roles of an Internet server and client. Since TCP packets are
bouncing off an Internet server, the originator and final destination of
the traffic are both presumed to be Internet clients.

In other words, the packets flooding our network necessarily had a SOURCE
port in the “service port” range from 1-1023, since they were “bouncing”
off servers handling and answering well known service ports (22, 23, 53, 80
and 179). But our server, when it is acting as a server, NEVER NEEDS to
receive packets with a source port less than 1024. Internet clients which
access our web, news, pop, and other servers always generate packets with
source ports in the client range, typically 1024 to 5000 or more.

Only when our server is acting as a client – when sending eMail via SMTP
to a remote port 25, or perhaps when querying external DNS servers, is
there any need to receive traffic with source ports in the “service port”
range. Specific rules can usually be created to safely allow traffic in
those special cases from specific authorized IPs, while denying ALL other
traffic with source ports less than 1024.

After adding these rules to our upstream router all, malicious “Packet
Bounce” attack traffic was filtered and we returned to the Internet with a
completely clean connection.

What's your #1 source for Internet needs? GoDaddy has new domain names, transfers and renewals as low as $1.99. Plus, check out their hosting plans, Web site builders, secure certificates and much more. Plus, as a listener of The Chris Pirillo Show, enter code CHRIS2 when you check out, and save an additional $5 off any order of $30 or more. Get your piece of the internet at GoDaddy!