50638 0,nop,wscale 0> (DF) (ttl 52, id 797)
Again, systematically:
- IPs: cable.modem.net is concentrating on two hosts we monitor: dns.one.org and dns.two.org. Although not shown, each host is hit with the second set of SYN packets three total times.
- Ports: The first half of the activity targets tcp ports 23 (telnet) and 143 (imap). The second half involves those ports plus 111 (SUN Remote Procedure Call, or portmapper), 79 (finger), 53 (dns), 31337 (Back Orifice tcp port), and 21 (ftp). All are of use to a potential intruder. Of more interest, perhaps, are the source ports involved. Note the stealthy nature of the first stage, where source port is set to destination port, in an attempt to confuse packet-filtering devices. The second stage is less cunning, but more analyst-friendly. Observe the orderly incrementation of ports used to contact dns.two.org, starting with 1146, then 1147, then 1149. Where is 1148? Most likely this packet was destined for a port not monitored by our NIDS. It was probably not lost, as the traffic to dns.two.org shows. Here, we see source port 1162, then 1163, then 1165 (another port missing!) Using this "gap-counting" technique, we can assume packets were sent to at least four ports not watched by our NIDS. This does not count the four "missing" ports between port 781 and 786, where attention shifts from dns.two.org to dns.one.org!
- Flags: The first half of the event involves no flags set, with RST ACK packets sent back from the targets. These initial packets do not occur naturally unless a preceded by the SYN / SYN ACK exchange of the three way handshake. The RST ACK packets are assumed to be returned from closed ports, as an open port would usually remain silent. (This is the default for the Linux TCP/IP stack, as documented by Vicki Irwin and Hal Pomeranz. Your mileage may vary.) Interestingly, the second half of the event shows only SYN packets sent, with zero replies. This may indicate the cablem.modem.net's initial packets, with the ACK bit set, successfully evaded a packet filtering device. This device, however, probably intercepted the later packets with the SYN bit set.
- Traffic direction/activity: All traffic seems to involve a prompt by cable.modem.net, followed by an indication that the target ports are closed.
- Time: The entire event elapses in six seconds, with an apparent five second delay between the ACK and SYN stages.
- Window size, TTL, and other features: We see a wide variance between the TTL 30 of stage one and TTL 52 of stage two. As these packets presumably come from the same host, we assume the tool generate the packets sets
initially TTLs differently for each technique. Stage one shows IP id values each forged to be 39426. This may provide a signature clue for future encounters with this tool. The IP id values increment nicely in stage two, matching the TCP port technique mentioned earlier. Window sizes for stage one (1028) contrast strongly with stage two (16324).
- Bottom line: We appear to have an ACK scan combined with some sort of SYN scan. The packet filtering device which allows ACK packets but prevents answers to the SYN packets keeps us from knowing more about stage two. This case emphasizes the need to understand the operation of your IDS, as it helped us to recognize the port "gaps" and their possible relevance to our investigation.
To Flood or Not to Flood
Now we turn to a core issue of this paper -- the SYN flood. Anyone unfamiliar with SYN floods would greatly benefit by reading Route's definitive work on the subject in Phrack 48. Essentially, a SYN flood is a denial of service attempt, where an attacker attempts to fill the backlog queue of a victim machine's TCP server. To prevent the victim from tearing down these memory-consuming connections, the attacker spoofs one or more source IPs, choosing IPs which presumably do not exist. The victim of a properly executed SYN flood cannot reply to the spoofed source(s), as the source(s) will not exist and therefore cannot clear the victim's potential connections. An
attacker might take these actions to attempt a TCP hijack, as Kevin Mitnick did against the rlogin port of a machine owned by Tsutomu Shimomura. By shutting down the TCP service of a host trusted by Shimomura, Mitnick was
able to impersonate that host without it interfering in his communications with Shimomura's box.
A SYN flood consists of dozens of SYN packets sent from a spoofed source IP, or multiple spoofed source IPs, to a victim. Note the high frequency of packets sent:
11:46:14.212003 spoofed.ip.one.1053 > flood.victim.com.23:
S 322286:322286(0) win 8192 (DF)
11:46:14.598008 spoofed.ip.one.1054 > flood.victim.com.23:
S 322286:322286(0) win 8192 (DF)
11:46:14.975522 spoofed.ip.one.1055 > flood.victim.com.23:
S 322286:322286(0) win 8192 (DF)
etc...
The desperate victim tries to reply to the spoofed source IP. If the spoofed host truly does not exist, the victim is out of luck. But what if the spoofed source does exist? Below is what an intrusion detection analyst at a
site owning the spoofed IP might see, if the target port is open and behaves as traditionally expected:
11:46:14.765043 flood.victim.com.23 > spoofed.ip.one.1053:
S 4137483508:4137483508(0) ack 322287 win 8192
11:46:14.891108 flood.victim.com.23 > spoofed.ip.one.1054:
S 4164828806:4164828806(0) ack 322287 win 8192
11:46:15.019029 flood.victim.com.23 > spoofed.ip.one.1055:
S 4192020032:4192020032(0) ack 322287 win 8192
etc...
Why would an you, an innocent bystander, witness such an act? The answer is: you own spoofed.ip.one, which may or may not exist! Why would anyone spoof your IPs? (Hint: do your routers or firewalls block ICMP?) In some cases, SYN flood utilities allow the attacker to select a range of IPs for the spoofed source, or it will generate its own list. The utility may ping that range, trying to determine if any hosts exist. If no ICMP echo replies are heard, the SYN flood program assumes the IPs do not exist and are ideal spoofed sources. However, if those IPs belong to hosts behind a router or firewall denying ICMP echo requests, they will not respond with ICMP echo replies. This "flaw" in choosing good spoofable IPs causes much of the "third party" traffic in this paper. Essentially, the site you monitor may become a third party to a SYN flood, by virtue of having blocked ICMP echo requests. A second possibility, which is demonstrated in actual code below, involves SYN flooding tools which randomly choose source IPs to spoof. In such cases, your network addresses could be selected without your involvement. This method of source IP selection appears to be the reason why I detected activity from two commonly seen ACK numbers, described later.
The preceding example appears straightforward. A single IP is spoofed, and the sender increments his source ports in an orderly manner (1053, 1054, 1055). The trace as seen by the innocent bystander shows the flood victim's open port 23 replying with SYN ACK packets, in an attempt to establish a TCP three-way handshake. What happens if the target port of a SYN flood is closed? The following was confirmed as a SYN flood by the author, who observed the traffic, contacted victim.isp.net, and learned the ISP was indeed SYN flooded on the date and time in question.
20:31:15.794717 victim.isp.net.68 > spoofed.ip.one.29470:
R 0:0(0) ack 723645348 win 0 (ttl 242, id 40923)
20:31:20.190800 victim.isp.net.68 > spoofed.ip.one.48926:
R 0:0(0) ack 960212644 win 0 (ttl 242, id 56829)
20:31:24.622496 victim.isp.net.68 > spoofed.ip.one.2846:
R 0:0(0) ack 1196779940 win 0 (ttl 242, id 7276)
20:31:37.634797 victim.isp.net.68 > spoofed.ip.one.61214:
R 0:0(0) ack 1906481828 win 0 (ttl 242, id 54177)
20:31:42.021607 victim.isp.net.68 > spoofed.ip.one.15134:
R 0:0(0) ack 2143049124 win 0 (ttl 242, id 4485)
20:31:17.754903 victim.isp.net.77 > spoofed.ip.two.44376:
R 0:0(0) ack 1861342051 win 0 (ttl 242, id 25377)
20:31:22.054453 victim.isp.net.77 > spoofed.ip.two.13400:
R 0:0(0) ack 454770019 win 0 (ttl 242, id 40905)
20:31:26.394198 victim.isp.net.77 > spoofed.ip.two.47960:
R 0:0(0) ack 1195681635 win 0 (ttl 242, id 56409)
20:31:39.370211 victim.isp.net.77 > spoofed.ip.two.20568:
R 0:0(0) ack 1270932835 win 0 (ttl 242, id 38330)
20:31:43.735814 victim.isp.net.77 > spoofed.ip.two.55128:
R 0:0(0) ack 2011844451 win 0 (ttl 242, id 54069)
Here we see a SYN flood directed against two closed ports, 68 (boot-p) and 77 (RJE private service). (Although many other ports were targeted, these two are shown as examples. Since spoofed.ip.one and spoofed.ip.two occupied separate class C networks, I chose to separate the two traces.) Observe the two closed ports return RST ACK packets to the spoofed source IPs. The ACK numbers appear random, as do the ports of the two spoofed IPs. Furthermore, a fairly high packet rate is seen. This may not always be true from the vantage point of the intrusion detection analyst. If the SYN flood tool does not spoof IPs which are all monitored by your IDS, you may not get a complete picture of the event. (And, from the perspective of the victim, more than one organization appears to be the originator of the attack!) For example:
05:23:07.968535 victim.isp.net.22 > spoofed.ip.one.10180:
R 0:0(0) ack 1 win 0 (ttl 53, id 57295)
05:23:55.594577 victim.isp.net.23 > spoofed.ip.two.64876:
R 0:0(0) ack 1 win 0 (ttl 53, id 62163)
05:27:36.116580 victim.isp.net.23 > spoofed.ip.three.48279:
R 0:0(0) ack 1 win 0 ttl 53, id 18796)
05:32:34.963053 victim.isp.net.23 > spoofed.ip.four.55483:
R 0:0(0) ack 1 win 0 (ttl 53, id 48851)
05:33:01.308930 victim.isp.net.23 > spoofed.ip.five.48412:
R 0:0(0) ack 1 win 0 (ttl 53, id 51512)
05:35:12.400935 victim.isp.net.22 > spoofed.ip.six.57306:
R 0:0(0) ack 1 win 0 (ttl 53, id 64704)
05:36:40.823582 victim.isp.net.22 > spoofed.ip.seven.46819:
R 0:0(0) ack 1 win 0 (ttl 53, id 8090)
05:38:50.740540 victim.isp.net.23 > spoofed.ip.eight.29217:
R 0:0(0) ack 1 win 0 (ttl 53, id 21089)
This trace shows several seconds elapsing between each packet as a malicious Internet user spoofs our IPs, SYN flooding ports 22 (ssh) and 23 (telnet) at victim.isp.net. Given the time delay, it is reasonable to assume the attacker is also spoofing IPs not monitored by our IDS, and could be truly pounding the victim.
ACK 674711610 and 674719802: The Mystery Tool?
The following cases involve specific signatures which many of you may recognize. Steven Northcutt notes two acknowledgement numbers which he believes characterize a tool which conducts "reset scans." Here I outline two confirmed cases showing the 674711610 and 674719802 acknowledgement numbers as third party effects of SYN floods.
Observe the following trace:
06:20:51.570058 firstclass.server.edu.510 > spoofed.ip.one.7002:
R 0:0(0) ack 674711610 win 0 (ttl 116, id 48680)
23:30:53.567215 firstclass.server.edu.510 > spoofed.ip.two.32771:
R 0:0(0) ack 674711610 win 0 (ttl 117, id 25440)
13:55:27.737433 firstclass.server.edu.510 > spoofed.ip.three.6666:
R 0:0(0) ack 674711610 win 0 (ttl 118, id 54468)
This trace seemed to conform to the model of a third party effect of a SYN flood. However, there is an extreme delay in the time between packets. This could be the result of a wide variety of spoofed sources, and I saw only a few. I guessed firstclass.server.edu to be a target host. These packets looked like responses, where port 510 was closed, or at least some mechanism was in place to resist a SYN flood. These three packets are a sample of the total traffic collected.
Researching port 510, I found it is the "firstclass" service, registered by SoftArc. SoftArc sells a product called the FirstClass Intranet Server, which can provide email, collaboration, and other services. The source IP belonged to a university, and the hostname resolution included the word "firstclass." It seemed that if a malicious Internet user wanted to perform a denial of service against this university, it might make sense to target port 510 (firstclass) on the school's FirstClass server. Given the presence of RST ACK packets from port 510 to multiple IPs, it seemed the host's buffer for port 510 was flooded and the port was now closed.
I contacted the school and confirmed their FirstClass server had been under a denial of service attack at the time and date noted in the packets sent to my hosts. The attacker was SYN flooding ports 68 (boot-p) and 510 (firstclass). The firstclass.server.edu system was not compromised and it was not originating the packets sent to my hosts. It was an innocent victim. The ACK 674711610 was generated by the tool used to SYN flood the hapless host. (To be precise, the packets sent by the tool used initial sequence numbers of 674711609, to which firstclass.server.edu replied with RST ACK 674711610.) "shaft" is one such tool; it chooses source IPs randonly. An analysis can be found here: http://packetstorm.securify.com/distributed/shaft_analysis.txt
While I specifically confirmed this case as being the third party effect of a SYN flood against an innocent victim, I have found dozens of similar traffic involving ACK 674711610. Here are two cases: the first with the SYN flooded ports open (6666 and 6667), replying SYN ACK; the second with the SYN flooded ports closed (23), replying RST ACK.
SYN flooded port open:
05:41:36.772836 major.irc.host.6666 > spoofed.ip.one.1578:
S 1822395560:1822395560(0) ack 674711610 win 4096 (DF)
05:41:53.834459 major.irc.host.6666 > spoofed.ip.two.1578:
S 311457256:311457256(0) ack 674711610 win 4096 (DF)
05:42:00.765914 major.irc.host.6667 > spoofed.ip.three.1433:
S 1074583123:1074583123(0) ack 674711610 win 61440 (DF)
05:42:08.955560 major.irc.host.6666 > spoofed.ip.four.1433:
S 2056091293:2056091293(0) ack 674711610 win 4096 (DF)
05:43:08.425388 major.irc.host.6666 > spoofed.ip.two.1578:
S 311457256:311457256(0) ack 674711610 win 4096 (DF)
05:43:09.175868 major.irc.host.6666 > spoofed.ip.one.1578:
S 1822395560:1822395560(0) ack 674711610 win 4096 (DF)
05:43:09.816458 major.irc.host.6666 > spoofed.ip.four.1433:
S 2056091293:2056091293(0) ack 674711610 win 4096 (DF)
05:43:10.558625 major.irc.host.6667 > spoofed.ip.three.1433:
S 1074583123:1074583123(0) ack 674711610 win 61440 (DF)
SYN flooded port closed:
12:52:10.879563 auction.this.com.23 > spoofed.ip.one.1985:
R 0:0(0) ack 674711610 win 0
12:54:37.882708 auction.this.com.23 > spoofed.ip.one.1554:
R 0:0(0) ack 674711610 win 0
12:55:38.961649 auction.this.com.23 > spoofed.ip.one.1409:
R 0:0(0) ack 674711610 win 0
Again, note the time delay between packets. This indicates not all the IPs spoofed by the attacker belonged to our monitored network. The next trace prompted a similar investigation:
10:20:52.097570 commercial.web.server.21 > spoofed.ip.one.1485:
R 0:0(0) ack 674719802 win 0 (ttl 50, id 1034)
10:22:28.994103 commercial.web.server.23 > spoofed.ip.one.1485:
R 0:0(0) ack 674719802 win 0 (ttl 50, id 38438)
10:25:43.004888 commercial.web.server.53 > spoofed.ip.one.1485:
R 0:0(0) ack 674719802 win (ttl 50, id 43626)
10:20:40.594667 commercial.web.server.21 > spoofed.ip.two.2104:
R 0:0(0) ack 674719802 win 0 (ttl 45, id 14598)
10:22:17.576229 commercial.web.server.23 > spoofed.ip.two.2104:
R 0:0(0) ack 674719802 win 0 (ttl 45, id 11298)
10:25:31.402693 commercial.web.server.53 > spoofed.ip.two.2104:
R 0:0(0) ack 674719802 0 (ttl 45, id 33894)
10:20:31.126616 commercial.web.server.21 > spoofed.ip.three.1667:
R 0:0(0) ack 674719802 win 0 (ttl 44, id 35589)
10:22:08.074117 commercial.web.server.23 > spoofed.ip.three.1667:
R 0:0(0) ack 674719802 win 0 (ttl 44, id 20256)
10:25:22.038942 commercial.web.server.53 > spoofed.ip.three.1667:
R 0:0(0) ack 674719802 win 0 (ttl 44, id 14437)
This source IP belonged to a commercial web site. While the three "source" ports, 21 (ftp), 23 (telnet), and 53 (dns) made little sense as true source ports, they might be good candidates as targets of a SYN flood. Sure enough, after contacting the web site, the system administrator told me a hired security consultant had tested the web server with a denial of service attack at the exact date and time indicated by my logs. Therefore, it is likely similar traces with ACK 674719802 are also the result of an attacker spoofing our IPs to SYN flood a separate victim. I do not believe these packets are generated to scan the destination IPs (here listed as spoofed.ip.xxxx). A tool called "synk4.c" creates SYN 674719801 packets, according to DDoS guru David Dittrich, who made this discovery by analyzing the following section of code:
#define SEQ 0x28376839 (0x28376839 is decimal 674719801)
According to Dave, "synk4 takes a source address on the command line for outgoing packets, and if zero, it generates them randomly using this code":
. . .
for (i=1;i>0;i++)
{
srandom((time(0)+i));
srcport = getrandom(1, max)+1000;
for (x=lowport;x<=highport;x++)
{
if ( urip == 1 )
{
a = getrandom(0, 255);
b = getrandom(0, 255);
c = getrandom(0, 255);
d = getrandom(0, 255);
sprintf(junk, "%i.%i.%i.%i", a, b, c, d);
me_fake = getaddr(junk);
}
. . .
Source code is here:
http://packetstorm.securify.com/spoof/unix-spoof-code/synk4.zip
As with ACK 674711610, I have found many examples of third party effects of SYN floods, where innocent victims are sending response packets to spoofed source IPs.
SYN flooded port open:
22:23:08.281683 biology.web.com.23 > spoofed.ip.one.1502:
S 2894258800:2894258800(0) ack 674719802 win 8192
22:25:46.030135 biology.web.com.23 > spoofed.ip.one.2154:
S 4154715243:4154715243(0) ack 674719802 win 8192
22:26:24.456103 biology.web.com.23 > spoofed.ip.one.2026:
S 159261598:159261598(0) ack 674719802 win 8192
22:29:38.265734 biology.web.com.23 > spoofed.ip.one.1838:
S 1866996756:1866996756(0) ack 674719802 win 8192
SYN flooded port closed:
22:34:47.629194 van.smack.net.21 > spoofed.ip.two.2031:
R 0:0(0) ack 674719802 win 0
22:36:01.282720 van.smack.net.21 > spoofed.ip.two.1071:
R 0:0(0) ack 674719802 win 0
22:36:11.483963 van.smack.net.21 > spoofed.ip.two.2143:
R 0:0(0) ack 674719802 win 0
At this time I am convinced that packets bearing ACK 674711610 and 674719802 are most likely the third party effects of a SYN flood against an innocent victim. This has been shown in my experience by contacting the sites which are the "sources" of these packets, and investigating their associated network histories.
What about reset scans? Do they exist? Presumably, the purpose of a reset scan is to determine the presence of live hosts on a network. A technique known as inverse mapping can be used to find live hosts on a network which allows its border routers or firewall to transmit ICMP error messages. If an attacker sends a RST ACK packet to a host which does not exist, the destination network's last router or firewall should send an ICMP host unreachable message. If the router/firewall is silent, we assume the target host MIGHT exist. Again, this technique relies returning ICMP error messages to source hosts. A reset scan of a network preventing outbound ICMP error messages would not yield nothing but false positive results to a reconnaissance gatherer. Reset scans can not be used to determine if ports are open on target machines. Why? Both open and closed ports should remain silent if a RST ACK packet is received. While not all vendors may implement this aspect of the RFC appropriately, most attempts to exploit these differences would be swamped by the false positive rate. Given these limiting factors, I tend not to invoke "reset scan" as an explanation for these sorts of packets.
A Final Case
I will conclude with a set of interesting traces which initially stumped me. With the help of my colleagues, and especially Mark Shaw, I pieced together the following case. Assume all the activity was registered by a single NIDS monitoring name.server.net.
09:22:56.960442 tester.newjersey.net.2100 > name.server.net.53:
S 2070441966:2070442030(64) win 2048 (ttl 246, id 34960)
09:22:56.960555 tester.newjersey.net.2101 > name.server.net.53:
S 1884680148:1884680212(64) win 2048 (ttl 246, id 8490)
09:22:56.960669 tester.newjersey.net.2102 > name.server.net.53:
S 938156752:938156816(64) win 2048 (ttl 246, id 17966)
09:26:30.485472 tester.newjersey.net.2100 > name.server.net.53:
S 593222604:593222668(64) win 2048 (ttl 246, id 10971)
09:26:30.485586 tester.newjersey.net.2101 > name.server.net.53:
S 171736880:171736944(64) win 2048 (ttl 246, id 6989)
09:26:30.486219 tester.newjersey.net.2102 > name.server.net.53:
S 1199445751:1199445815(64) win 2048 (ttl 246, id 47166)
09:24:13.867591 tester.brazil.net.2100 > name.server.net.53:
S 795939539:795939603(64) win 2048 (ttl 241, id 53652)
09:24:13.868783 tester.brazil.net.2101 > name.server.net.53:
S 2049322111:2049322175(64) win 2048 (ttl 241, id 13883)
09:24:13.873062 tester.brazil.net.2102 > name.server.net.53:
S 1779866028:1779866092(64) win 2048 (ttl 241, id 14298)
09:28:04.337763 tester.brazil.net.2600 > name.server.net.53:
S 535782194:535782258(64) win 2048 (ttl 241, id 7673)
09:28:04.339246 tester.brazil.net.2601 > name.server.net.53:
S 1049573717:1049573781(64) win 2048 (ttl 241, id 37399)
09:28:04.339383 tester.brazil.net.2602 > name.server.net.53:
S 148280449:148280513(64) win 2048 (ttl 241, id 25525)
09:23:26.765186 tester.argentina.net.2100 > name.server.net.53:
S 1616673589:1616673653(64) win 2048 (ttl 241, id 21017)
09:23:26.765744 tester.argentina.net.2101 > name.server.net.53:
S 1351385345:1351385409(64) win 2048 (ttl 241, id 9204)
09:23:26.766781 tester.argentina.net.2102 > name.server.net.53:
S 184647009:184647073(64) win 2048 (ttl 241, id 8397)
09:24:26.275614 tester.argentina.net.2100 > name.server.net.53:
S 1577583159:1577583223(64) win 2048 (ttl 241, id 10735)
09:24:26.276245 tester.argentina.net.2101 > name.server.net.53:
S 1874158503:1874158567(64) win 2048 (ttl 241, id 44674)
09:24:26.276922 tester.argentina.net.2102 > name.server.net.53:
S 1571547407:1571547471(64) win 2048 (ttl 241, id 20440)
09:25:42.915131 tester.argentina.net.2100 > name.server.net.53:
S 988147012:988147076(64) win 2048 (ttl 241, id 41923)
09:25:42.915743 tester.argentina.net.2101 > name.server.net.53:
S 819957179:819957243(64) win 2048 (ttl 241, id 40998)
09:25:42.916419 tester.argentina.net.2102 > name.server.net.53:
S 1343568781:1343568845(64) win 2048 (ttl 241, id 22882)
Let us apply structured analysis to classify this activity.
- IPs: We see three separate machines -- tester.newjersey.net, tester.brazil.net, and tester.argentina.net -- attempting to connect to a single machine, name.server.net. You cannot determine anything more about the three initiating IPs, but name.server.net (you guessed it) is your name server.
- Ports: On the initiating side, we see a possible pattern. From each source IP, ports 2100, 2101, and 2102 are used. The tester.brazil.net box also employs 2600 (greets), 2601, and 2602. All destination ports are 53 (domain name service). Normal DNS traffic typically employs UDP, while zone transfers are done via TCP. Note BIND versions 8.2 and higher offer name queries via TCP. This process complicates our analysis and must be saved for a future paper.
- Flags: Every connection is a single SYN. This would indicate an attempt to begin the three-way handshake to exchange data, or perhaps start a scan.
- Traffic direction/activity: All traffic is sent from one of the three hosts to name.server.net. No replies are seen. Each source packet seems to contain 64 bytes of data. This differs from the very first trace we presented, showing an exchange between ftp.client.org and ftp.server.org. In the SYN packet which started that transfer, no data was passed. We can only guess at the data contained, as it was not saved with the rest of the TCP packet. For comparison's sake, observe the difference in the second line of each trace:
Case 1: No data in SYN packet:
14:05:27.083238 ftp.client.org.1057 > ftp.server.edu.21:
S 1484414:1484414(0)
Case 2: 64 bytes in SYN packet:
09:22:56.960442 tester.newjersey.net.2100 > name.server.net.53:
S 2070441966:2070442030(64)
- Time: All of the packets are sent between 09:22 and 09:28 on the same day. This indicates some level of coordination.
- Window size, TTL, and other features: Window size for each packet is 2048 bytes. TTLs for the two South American hosts are smaller than the New Jersey host, indicating they may have hopped through more routers on their way to your American-based name.server.net. This is to be expected if each host sets its initial TTL to the same value, such as 255.
- Bottom line: Why would three hosts all try to connect to one of our name servers, nearly simultaneously? Could they be responding to an action by one of our hosts? Is this activity malicious?
After discussing the situation with my colleagues, I formed a theory and sent emails to the points of contact listed in ARIN information for the three hosts. One of the three responded and explained the situation. The three IPs are part of a system which performs "load balancing" and dynamic redirection to a commercial web site. The process occurs as follows, using a fictitious example:
1. A web-browsing client in Chile wants to visit the web site of a major e-commerce site. She enters the URL in her browser. Her host contacts her local DNS to find the IP address associated with that hostname.
2. The local DNS server does not have the IP address in its cache, so it begins querying DNS servers until it reaches the authoritative name server of the domain owning the IP in question. This system, a "load balancing manager" (LBM), is either tied to, or serves as, the DNS for the domain.
3. The LBM checks its cache for any traffic management rules which declare how to handle requests from the client's IP address. At this stage the LBM may immediately return an IP address to the client's local DNS, or it may
proceed to step four.
4. Not finding any cached values, and choosing not to deliver a less-than-optimal IP choice to the client, the LBM queries its load balancing systems (LBS) at its three web sites, in New Jersey, Brazil, and Argentina.
5. The LBS' at the three sites conduct latency testing against the client's local DNS. These may include ICMP or TCP packets for which round trip time (RTT) is calculated, based upon responses from the client's DNS. The site whose tests result in lowest RTT is deemed "closest" (in Internet space) to the client. The IP of the "closest" site is returned to the LBM. Remember the "closest" IP could belong to a host with a very fast pipe, but very far away.
6. The LBM provides the client's local DNS with the IP of the Argentina web site.
7. The client's local DNS provides the IP of the Argentina web site to her host.
8. Her host makes contact with the web site in Argentina, displaying content.
Once the client has visited a web enterprise employing load balancing, her local DNS server may be subject to repeated and seemingly aggressive latency testing for extended periods of time. These are not malicious probes, however.
The goal of the system is to provide the quickest response time to the client while efficiently managing activity on the web server. While some in the security community view this activity as a malicious attempt to map the customer's network, I see it as a realistic attempt to serve the hundreds of thousands to millions of customers who visit the more popular web sites each day.
I found this particular load balancing system begins its tests by sending ICMP packets. If ICMP is denied by the client's routers or firewalls, the load balancer then attempts to connect to TCP port 53 on the client's name server. This explains the packets we are investigating. Since the name server in our example did not appear to respond, we can assume the load balancing program did not work out as planned, unfortunately.
What might be the next step? The network engineer responsible for these load balancers told me a final, more aggressive latency test can be made. Here the system would essentially scan the client's name server for an open port, then use the replying SYN ACK packet to test response time. Yes, this would look exactly like a multiple service port scan! For this reason, the network engineer said he has disabled this feature. Have you seen activity fitting this description against your name server?
The final trace is from another load balancing system. It uses a different packet type to do the job. Rather than SYN packets with 64 bytes of data, it sends SYN ACKs with no data. This activity was recorded after a visit to a site which employs the load balancing products. Neither the client (X) nor the web server (Y) are shown below, but four hosts involved with load balancing are included. They are:
name1.server.net: DNS for web browsing client X
name2.server.net: DNS for web browsing client X
mayfield.ohio.net: Load balancer 1 for web server Y
greenbelt.maryland.net: Load balancer 2 for web server Y
Here is the first load balancing server in action:
06:01:15.001304 mayfield.ohio.net.44132 > name1.server.net.53:
S 10399587:10399587(0) ack 10399586 win 4128 (ttl 241, id 0)
06:01:16.999359 mayfield.ohio.net.44132 > name1.server.net.53:
S 10399587:10399587(0) ack 10399586 win 4128 (ttl 241, id 0)
06:01:17.498365 mayfield.ohio.net.44133 > name2.server.net.53:
S 10399588:10399588(0) ack 10399587 win 4128 (ttl 241, id 0)
06:01:18.528689 mayfield.ohio.net.44135 > name1.server.net.53:
S 10399590:10399590(0) ack 10399589 win 4128 (ttl 241, id 0)
06:01:20.524742 mayfield.ohio.net.44135 > name1.server.net.53:
S 10399590:10399590(0) ack 10399589 win 4128 (ttl 241, id 0)
... (thirteen similar packets deleted for clarity)
06:01:58.754918 mayfield.ohio.net.44172 > name2.server.net.53:
S 10399627:10399627(0) ack 10399626 win 4128 (ttl 241, id 0)
Here is the second load balancing server, simultaneously testing the same two name servers:
06:01:14.967214 greenbelt.maryland.net.63604 > name1.server.net.53:
S 34541003:34541003(0) ack 34541002 win 4128 (ttl 249, id 0)
06:01:17.461642 greenbelt.maryland.net.63607 > name2.server.net.53:
S 34541006:34541006(0) ack 34541005 win 4128 (ttl 249, id 0)
06:01:18.503320 greenbelt.maryland.net.63609 > name1.server.net.53:
S 34541008:34541008(0) ack 34541007 win 4128 (ttl 249, id 0)
06:01:19.464217 greenbelt.maryland.net.63607 > name2.server.net.53:
S 34541006:34541006(0) ack 34541005 win 4128 (ttl 249, id 0)
06:01:20.682888 greenbelt.maryland.net.63615 > name2.server.net.53:
S 34541014:34541014(0) ack 34541013 win 4128 (ttl 249, id 0)
... (seven similar packets deleted for clarity)
06:01:56.995151 greenbelt.maryland.net.63764 > name2.server.net.53:
S 34541163:34541163(0) ack 34541162 win 4128 (ttl 249, id 0)
I reconstructed the load balancing process based upon my contacts with vendors and my understanding of load balancing operation. It is my best interpretation of the network traces, and shows how one can try to rebuild a puzzle given one or two crucial pieces.
Conclusion
In this paper, we began with a warning to know and potentially mistrust your NIDS. We introduced TCPDump, used it to look at a simple exchange of data via ftp, and discussed SYN floods. Multiple variations of SYN flood traffic was shown, and third party traffic was shown to not be "reset scans." We finished with two examples of load balancing software signatures. I hope this paper has encouraged you to take a closer look at your NIDS data, and share what you find. I look forward to hearing from you.
Appendix: Trace Excerpt with Absolute Sequence Numbers Printed
Relative sequence numbers are usually used, since we are typically interested in the amount of data passed once the initial sequence numbers are established. Plus, listing every full sequence number involves showing many distracting digits! Nevertheless, I found the following trace useful to understand whom is ACKing whom.
11:42:18.407029 dialup.modem.net.1052 > web.server.org.80:
S 382137:382137(0) win 8192 (DF)
11:42:18.582348 web.server.org.80 > dialup.modem.net.1052:
S 1616321351:1616321351(0) ack 382138 win 9112 (DF)
11:42:18.593124 dialup.modem.net.1052 > web.server.org.80:
. ack 1616321352 win 8576 (DF)
11:42:18.659933 dialup.modem.net.1052 > web.server.org.80:
. 382138:382674(536) ack 1616321352 win 8576 (DF)
11:42:18.664698 dialup.modem.net.1052 > web.server.org.80:
P 382674:382684(10) ack 1616321352 win 8576 (DF)
11:42:18.884944 web.server.org.80 > dialup.modem.net.1052:
. ack 382674 win 9112 (DF)
11:42:18.949336 web.server.org.80 > dialup.modem.net.1052:
. ack 382684 win 9112 (DF)
11:42:19.106286 web.server.org.80 > dialup.modem.net.1052:
P 1616321352:1616321766(414) ack 382684 win 9112 (DF)
11:42:19.232579 dialup.modem.net.1052 > web.server.org.80:
. ack 1616321766 win 8162 (DF)
11:42:19.320803 web.server.org.80 > dialup.modem.net.1052:
P 1616321766:1616321774(8) ack 382684 win 9112 (DF)
11:42:19.359277 web.server.org.80 > dialup.modem.net.1052:
P 1616321774:1616321854(80) ack 382684 win 9112 (DF)
11:42:19.366198 dialup.modem.net.1052 > web.server.org.80:
. ack 1616321854 win 8074 (DF)
Notice one sequence number is used by each side before any data is passed. web.server.org ACKs 382138 (showing 382137 was "used"), and dialup.modem.net ACKs 1616321352 (showing 1616321351 was "used"). Knowing these ACK numbers, we know the first byte of data passed from dialup.modem.net will be 382138, and the first byte passed by web.server.net will be 1616321352. Sure enough, the fourth packet,
11:42:18.659933 dialup.modem.net.1052 > web.server.org.80:
. 382138:382674(536) ack 1616321352 win 8576 (DF)
and the eighth packet,
11:42:19.106286 web.server.org.80 > dialup.modem.net.1052:
P 1616321352:1616321766(414) ack 382684 win 9112 (DF)
confirm this understanding of sequence numbers. Check the format again to be sure:
sequence number of first byte in packet:sequence number of first byte in NEXT packet (data)
Armed with this knowledge, the relative sequence numbers should make sense as well.
References
Daemon9, a.k.a. Route. "Project Neptune." (Phrack 48, Article 13, 1996)
Dietrich, Sven, Long, Neil, and Dittrich, David. "An Analysis of the ‘Shaft’ Distributed Denial of Service Tool."
http://packetstorm.securify.com/distributed/shaft_analysis.txt
Irwin, Vicki and Pomeranz, Hal. "Advanced Intrusion Detection and Packet Filtering." (SANS Network Security
99, 1999)
Newsham, Tim, and Ptacek, Tom. "Insertion, Evasion, and Denial of Service: Eluding Network Intrusion
Detection." (Secure Networks, Inc., 1998)
Northcutt, Stephen. Network Intrusion Detection: An Analyst's Handbook. (Indianapolis, Indiana: New Riders,
1999)
Postel, Jon (ed.). "RFC 793: Transmission Control Protocol." (Defense Advanced Research Projects Agency,
1981)
Stevens, W. Richard. TCP/IP Illustrated, Volume 1: The Protocols. (Reading, Massachusetts: Addison-Wesley,
1994)
Acknowledgements
I would like to thank the following people for reading and commenting upon this paper, or giving me guidance prior to writing: Dave Dittrich, Chad Renfro, Bamm Visscher, Mark Shaw, Chuck Port, Cheryl Knecht, Sam Adams, John Green, Dustin Childs, Judy Novak, and all members of the Intrusion Detection Flight!
Revision History
v1.0 Draft only, unpublished
v2.0 Corrected errors, added confirmed case studies
v2.1 Corrected minor error regarding DF flag (thanks Judy Novak!)
v2.5 Reformatted and reorganized for 12th FIRST Conference; improved explanation of MSS; deleted discussion of using Snoop formatted data
v2.6 Corrected interpretation of FIN scan, thanks to presentation by John Green at SANS
v2.7 Changed personal email and home page URL
v2.8 Incorporated information on "shaft" and "synk4.c" from Dave Dittrich.