Network Drool: Unsecuring Your Network
Author: Jerry Benton

Prior to becoming a new father, my wife and I did our best to make our home “baby bullet proof” by installing every type of child gizmo sold in the store. We put plastic plugs in the electrical outlets. We put child-proof (adult-proof) locks on all the kitchen cabinets accessible by someone two feet tall. We put those corner rounding gizmos on all of our furniture. We bolted all the bookshelves to the walls. We removed just about everything from the coffee tables, end tables, kitchen table, and everything else within reach of our soon-to-be bundle of joy. Then my son, Riki, was born. If you are a parent, you know the drill. My life became full off sleepless nights, diaper changing, sharp teeth piercing my shoulder, strange smells, spit up, hair pulling, and more toys than you could shake a stick at. Then he learned how to crawl. This is where things got even more interesting. One day, I found Riki in the living room sitting in the middle of a mountain of VHS tapes performing a taste test on each tape. As he cackled with joy upon seeing me, the level of drool coating my VHS library was astounding. These tapes had been organized alphabetically and by genre. They had also been “secured” in a cabinet with those child-proof (adult-proof) locks. The only thing that saved him from the wrath of dad was the fact that he was cute while performing his cackle and drool routine with my prized library. The moral of the story is simple: No matter how hard you try to lock something down, someone will probably get into it and, possibly, change it.

I thought about this event just recently when I found some issues on the network I currently perform technical Information Assurance administration on. It started with a simple trouble ticket regarding web browsing by a user. The ticket was assigned to me because of the very suspicious nature a user was experiencing when trying to browse to a military web site. Apparently, the user was getting a hacker web site every time he entered the URL for the military site. My eyes popped at this one. I logged into my regular user account, increased the security level of my web browser to the maximum, and entering the military URL. The title of the page was “Y0ur Administration r00ted by Apocalypse” and the page contained droves of text that was apparently designed to get web crawlers to recognize it as a hacker site. My eyes got a bit wider. I then rolled over to a laptop that I keep in the DMZ for administration. The systems in the DMZ are hardened, so I was hoping the same problem was not happening there. When I entered the URL into the laptop’s browser, the correct page was displayed. I then checked what IP the web site resolved to on the laptop. I rolled back over to my network workstation and checked again. I blinked a few times when I saw that the IP returned was different.

When I first arrived on site about two years ago I sat down with the customer and drew out a new long term migration plan to improve their security. For example, I designed a DMZ, hardened external DNS servers, and hardened mail gateways. The plan was to move all connections inbound from the Internet to the internal network. Prior to the migration mail connections were allowed into the production network to the Microsoft Exchange servers. Also, the web site for the organization was in the internal network and the internal DNS servers were using a third party to do name resolution outside of our domain. Over about a three month period I moved all of this to a DMZ by building Linux-based mail gateways that handled inbound and outbound mail, building hardened external DNS servers using BIND to perform external name resolution, and moving the web server into the DMZ. The idea was that no connections would be allowed into the internal network and we would provide our own lookups for DNS. The organization had once before been burned by using external DNS. The server they were directing their requests to either went down or stopped accepting requests from outside of its network. Either way, the organization that owned the DNS server didn’t notify us of the change, and I would not have expected them to since our organization started using it without notifying them that we were. It shut the entire network down for a full day until we figured it out. No email. No web browsing. No ftp. No nothing. That really helped drive my point home to the management. After that, I grabbed the local Windows guru and we made sure that the internal DNS servers were pointed at the servers in the DMZ and no where else. I also modified the firewall rules to allow SMTP and DNS traffic from the internal servers only to the servers in the DMZ. For two years everything ran rock solid.

So, I am now sitting at the helm of two systems that are giving two different answers for the same question. The obvious answer was that the internal DNS servers or the proxy server handling web requests was getting the wrong IP address. All the servers are pointed at the servers in the DMZ for name resolution, just as my laptop, which was giving the correct answer. So, I rolled over to the proxy server. The number one issue I was concerned with was that the proxy server had just been rebuilt by someone other than me, so I was only able to assume it was setup correctly. (Of course it wasn't.) Well, it became apparent very quickly that I should have involved myself in the rebuild. The proxy server was setup to get name resolution from external DNS servers. Apparently, one of those servers had been compromised and since we were using it, it poisoned the proxy server’s cache. After correcting it, flushing the cache, and restarting some services, I rolled back over to my workstation. I flushed all of its cached DNS and did a new lookup. I was still getting the wrong IP address. I blinked again for a few minutes.

Next, I rolled over to the internal DNS servers. I checked the forwarders on it. They had been changed to get external resolution from the same external DNS servers. After a few minutes of throwing things in the data center, I changed the settings back to what they were supposed to be, cleared the cache, and restarted some services. I again checked from my internal workstation and was finally getting the correct answers. To make sure that lookups were being done as planned, I tested some domains that I had created black hole zones for on the DNS servers in the DMZ. The results had my blood boiling. The domains were no longer black holed. I checked the internal DMZ servers again and saw that recursion had been turned back on, which makes a Microsoft DNS server go to the Internet if it gets a null answer, such as what the DMZ servers were giving for certain domains. More items started spontaneously flying across the data center followed by lots of foul language.

So, I rolled back over to the proxy/firewall. I checked the rules. At this point, I was ready to just leave. Our network had been totally unsecured for nearly a month. Here is what I found:

            DNS Outbound

DNS is specifically not allowed outbound. The internal DNS servers are Microsoft products, which have a history of being susceptible to DNS poisoning. (SANS Institute, 2004, Part 2, p. 67) Therefore, they get their answers from trusted DMZ servers that are hardened against these types of vulnerabilities. They internal servers had been pointed to other servers and a hole was made in the firewall to support it.

Mail Inbound

SMTP connections were being allowed to the internal Microsoft Exchange servers. This bypasses the layered defense that the mail gateways provide. The gateways provide virus scanning, spam scanning, dangerous content scanning, and a variety of over thirty other tests. The current configuration would allow an attacker straight into the internal mail server that has nothing more than a virus scanner. (Julian Field, 2005)

ICMP

ICMP in all forms was allowed in all directions. This left us vulnerable to mapping of the internal network and could even lead to our network becoming a Smurf amplifier or possibly susceptible to attack such as the Ping of Death, or even Firewalking, which is a form of network mapping that attempts to defeat firewalls. (SANS Institute, Part 3, 2004, p. 150)

PPTP Outbound

PPTP, or VPN connections, were being allowed outbound to the world from our internal network. This vulnerability allows any user to create a bridge between our network and another. We also inherit the remote network’s security since we have no control over what they do. Anything they have compromised could be used as a straight shot into our network.

Holes

Holes were created for the s specific workstation. In short, this workstation was allowed to bypass all proxy and firewall rules and go straight to the Internet. Other systems that should not have that kind of access were added to do the same.  I will also point out that the user of this system had domain administrator privileges. Do the math on that one.

Do you have vulnerabilities in your network? Are you sure? In my example, it wasn ’t technical. It was a people problem. It may have even been a political problem. Not only did we know the technical solution and how to implement it, we knew how to monitor for holes. We had it secure, and someone heinously unsecured it. The moral of the story is simple: No matter how hard you try to lock something down, someone will probably get into it and, possibly, change it.

Back to Articles and Tutorials

References

Julian Field. (2005). About MailScanner [Brochure]. Author. Retrieved October 14, 2050, from MailScanner Web site: http:/​/​www.mailscanner.info

SANS Institute. (2004). Computer and Network Hacker Exploits, Part 2. Bethesda, MD: SANS Institute.

SANS Institute. (2004). Computer and Network Hacker Exploits, Part 3. Bethesda, MD: SANS Institute.