NSA.gov Hosed?

By: Danny McPherson -

Someone nudged me a little while ago and asked if I was aware of any issues with nsa.gov. I plugged http://www.nsa.gov into my browser to no avail. I then tried to resolve www.nsa.gov’s DNS name to an IP address, to no avail. So, I had a bit of a deeper look..

danny@pork% host www.nsa.gov <- seeing if name will resolve
;; connection timed out; no servers could be reached
danny@pork% host -Cv nsa.gov
<- checking for SOAs on authoritative servers
Trying “nsa.gov”
;; connection timed out; no servers could be reached

Hrmm… So, what are those authoritative servers? I ask a.gtld-servers.net and get pointed to [a-m].root-servers.net, which makes sense because they serve the .gov domain. I ask a.root-servers.net, and get pointed to [a-g].GOV.ZONEEDIT.COM, which makes sense. So, I ask a.gov.zoneedit.com, and it returns:

danny@pork% dig @a.gov.zoneedit.com nsa.gov ns

; <<>> DiG 9.4.1-P1 <<>> @a.gov.zoneedit.com nsa.gov ns
; (1 server found)
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30111
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;nsa.gov. IN NS

;; AUTHORITY SECTION:
nsa.gov. 86400 IN NS ROMULUS.NCSC.MIL.
nsa.gov. 86400 IN NS TOPSCALE.nsa.gov.

;; ADDITIONAL SECTION:
TOPSCALE.nsa.gov. 86400 IN A 144.51.68.4

;; Query time: 161 msec
;; SERVER: 216.55.155.29#53(216.55.155.29)
;; WHEN: Thu May 15 09:54:25 2008
;; MSG SIZE rcvd: 9

So, I get two name servers as authoritative for nsa.gov, romulus.ncsc.mil, and topscale.nsa.gov. A an record (144.51.68.4) was included for topscale.nsa.gov in the NS query response, so I query for the A record of romulus.ncsc.mil:

danny@pork% host romulus.ncsc.mil
romulus.ncsc.mil has address 144.51.5.2

And now know that the two authoritative servers and their corresponding IP addresses for nsa.gov are:

romulus.nscs.mil (144.51.5.2)
topscale.nsa.gov (144.51.68.4)

You might notice something peculiar with the above two IP addresses. They both reside within the same /16 prefix (144.51.0.0/16), registered to the National Computer Security Center (NCSC):

danny@pork% whois -h whois.arin.net 144.51.0.0

OrgName: National Computer Security Center
OrgID: NCSC-3
Address: 9800 Savage Road
City: Fort George G. Meade
StateProv: MD
PostalCode:
Country: US

NetRange: 144.51.0.0 – 144.51.255.255
CIDR: 144.51.0.0/16
NetName: NCSC
NetHandle: NET-144-51-0-0-1
Parent: NET-144-0-0-0-0
NetType: Direct Assignment
NameServer: ROMULUS.NCSC.MIL
NameServer: ZOMBIE.NCSC.MIL
NameServer: BARRIER.NCSC.MIL
NameServer: GRIZZLY.NRL.NAVY.MIL
Comment:
RegDate:
Updated: 1997-11-17

RTechHandle: AMM32-ARIN
RTechName: McCool, Anna M.
RTechPhone: +1-301-688-5267
RTechEmail: amm@romulus.ncsc.mil

# ARIN WHOIS database, last updated 2008-05-14 19:10
# Enter ? for additional hints on searching ARIN’s WHOIS database.

Internet routing reachability for that /16 is announced by Qwest (AS 209):

route-views.oregon-ix.net>sh ip bgp 144.51.0.0
BGP routing table entry for 144.51.0.0/16, version 109664
Paths: (37 available, best #30, table Default-IP-Routing-Table)
..
3356 209
4.68.1.166 from 4.68.1.166 (4.68.1.166)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: 3356:3 3356:22 3356:86 3356:575 3356:666 3356:2008
..

And both these IPs (intuitively) are downstream from the same Qwest edge access router in DC. When trying to query these servers directly, as with the host -C (-C == compare SOAs) method used above, both timed out. Just to be sure, let’s try dig and see if there are still problems:

danny@pork% dig @144.51.5.2 nsa.gov ns

; <<>> DiG 9.4.1-P1 <<>> @144.51.5.2 nsa.gov ns
; (1 server found)
;; global options: printcmd
;; connection timed out; no servers could be reached

danny@pork% dig @144.51.68.4 nsa.gov ns

; <<>> DiG 9.4.1-P1 <<>> @144.51.68.4 nsa.gov ns
; (1 server found)
;; global options: printcmd
;; connection timed out; no servers could be reached

And this means mail servers that don’t have a locally cached copy of the record (that’s working towards expiry) won’t be able to resolve the IP address or identify the MX record for the nsa.gov domain, so no email either, even if the mail server is reachable and hosted elsewhere.

danny@pork% dig nsa.gov mx

; <<>> DiG 9.4.1-P1 <<>> nsa.gov mx
;; global options: printcmd
;; connection timed out; no servers could be reached

Also interesting is that www.ncsc.mil is actually an alias for romulus.ncsc.mil, so they’re presumably the same machine. This would mean that presumably, an authoritative name server for nsa.gov, is also running a web server process, one that’s also not reachable at the moment:

danny@pork% host www.ncsc.mil
www.ncsc.mil is an alias for romulus.ncsc.mil.
romulus.ncsc.mil has address 144.51.5.2


danny@pork% telnet !$ 80
telnet www.ncsc.mil 80
Trying 144.51.5.2…
telnet: connect to address 144.51.5.2: Operation timed out
telnet: Unable to connect to remote host

Ahh, but alas, if you really need to reach nsa.gov you could try to google for the A record associated with nsa.gov (AND cross your fingers you get a legit one), see if an IP (e.g., 12.110.110.204) is in the same address block as the servers above, which it’s not, and try to connect, which indeed you can, and it appears to be serving an nsa.gov page (though I’m not sure it’s a current or legit one):

danny@pork% telnet 12.110.110.204 80
Trying 12.110.110.204…
Connected to 12.110.110.204.
Escape character is ‘^]’.
^C^]
telnet> q
Connection closed.

It’s interesting that both the authoritative servers are off a Qwest DC router, yet the web server itself is connected to AT&T and addressed our of 12.0.0.0/8 space. With this, one of a few things have occurred:

  • NCSC has a security policy or router/network misconfiguration or network outage that’s causing this problem
  • They’ve got problems on the servers that result in neither authoritative server responding to DNS queries
  • They’ve got much bigger problems and raised the drawbridge (we’ve seen nothing in our attack databases targeting NSA or NCSC names or IP addresses)

So, what are the take aways from this?

  • Running a web server on the same machine (or at least same IP address) as an authoritative name server for nsa.gov isn’t necessarily an ideal separation of tasks. If one is owned or broken, the other is.
  • Just as with YouTube a week or so ago, no apparent separation of primary and secondary authoritative name servers for a DNS zone is a BAD thing (note: one might argue it’s setup this way for centralized control; duly noted, but I’m not buying it).
  • A DNS outage has the same effect as a completely effective DoS attack – from an end-user experience perspective. This just further highlights how fragile the Internet’s control plane (e.g., DNS & routing infrastructure) is
  • READ RFC 2182

I am told that NSA is aware of and working to resolve the problem, let’s hope it’s simply a misconfiguration of some sort.

Comments

  1. Danny,

    Did you happen to do any traceroutes to the NS IP’s during the outage? If so were they reachable? So was it that the machines were unreachable or was the DNS zone broken?

    Paul.

  2. Danny McPherson 05/21/2008, 10:02 pm

    Most of the stuff (e.g., traceroute probes) to those IPs is filtered at the ingress of the NCSC network just after it leaves the ISP network. Routing was fine to there, but there was no visibility or response after those hops. Apparently it wasn’t just a DNS process crash issue or the like, as the http server running on one of the boxes was also unreachable. HTH, -danny