LAN Allowed, WAN Blocked

This is probably going to end up being some stupid mistake I made, but I'm really confused right now. Without intentionally accomplishing it, I've managed to block all access to my SL 6.1 server outside my home LAN. Inside the LAN, I've got full access to any port that I've opened in iptables even using the external DNS names. The router I have uses NAT loopback and its internal port forwards to know where to route the requests.

Outside the LAN, I can't get to anything. I've got a home router running the Tomato firmware (which, for those not familiar, is an embedded Linux firmware), and it has the proper ports forwarded to the proper IP. I can see in the logs that the packets are being forwarded to the correct internal IP, but I don't see a thing in any logs for the services running on the corresponding ports. This applies to SSH, LDAP, and HTTP. Obviously, there can be a ton of different things causing it, but does anybody have any suggestions for things to look at?

To narrow it down some, my hosts.allow and hosts.deny files are empty, SELinux is disabled, and I've even tried turning off iptables entirely. I also disabled ebtables, though honestly I'm not entirely sure how that works. I just saw that it's another firewall. Have I missed any other firewalls that are either installed by default or available in the default repos that I could have accidentally ended up with? Are there any network settings that could be tripping me up that anyone is familiar with? I know it's not a lot of information to go on, but I have no idea where to look or what you might want to see. I can post configs or logs as requested. Thanks a lot in advance for any help!

Install tcpdump (it's in the official SL6 repo) and see if you can see any incoming packets on your server when attempting to connect from the outside.

Thanks for the suggestion. I installed it and tried to load up a couple of websites while it was running. I see packets showing up with the correct source IP heading for the correct destination IP on the correct destination port. The pages never load in the browser though, and I see no hints of problems or access attempts at all in the Apache logs. I've copied a line of tcpdump below in case I'm not reading it correctly, but I think that indicates that I'm getting the data to the server properly.

10:17:18.247390 IP (tos 0x20, ttl 45, id 345, offset 0, flags [none], proto TCP (6), length 48)
199.xxx.xxx.xxx.44395 > 10.xxx.xxx.xxx.80: Flags [S], cksum 0x42a0 (correct), seq 811201926, win 65535, options [mss 1460,sackOK,eol], length 0

Hi,
from what you say, the problem is on your router. I dont know you home LAN topology (if you have ip forwarding up on server, properly setup gateway, routes, NICs, etc etc.) but from you home server, you have to be able reach out through the router.
Check you router settings again, proper forwarding, rules.

cheers

Yes if 10.xxx is your server (on which you are running tcpdump) and 199.xxx is the remote client then your server is receiving the packets from the client. Do you see any response packets leaving the server towards 199.xxx?
Have you tried disabling the firewall on your server?

You can check to see that the firewall is really disabled by running:

WIth a disabled firewall the output should look like this:

Correct, 10.xxx is my server and 199.xxx is the remote client. I do not see any packets returning. The firewall is definitely off (output matches what you posted exactly), so that doesn't seem to be it.

For the router, if I point the exact same port forward to the internal IP of a different Linux server running CentOS, it works perfectly. Also, I forgot to mention before that this machine has four physical NICs, one of which (eth3) is set up with a bridge for KVM virtual machines to use. The port is being forwarded to eth0, and I've never had a problem with this setup in the past. Granted, I did switch from CentOS 5.x to SL 6.1 recently, but there was a significant gap between the OS installation and this problem showing up. Outside of the bridge, I don't believe I have any unusual network configurations, but if there's anything specific in there I should be looking at I'm all ears.

What applications are you trying to reach on the server from the outside client (httpd or dns server or mail server or whatelse)?

Do the logs of these server apps show that they receive the requests from the outside client?

Can you post the output of netstat -rn on the server?
(posting your internal IPs here shouldn't be a problem)

I'm attempting to reach OpenSSH and Apache. Both are on default ports. OpenSSH has almost no default settings changed. I have no indication in any logs that the connection attempt made it to those apps. I don't see any 199.xxx references anywhere.

Here's the output of netstat:

Also, thanks so much for your patience and help!

Also, thanks so much for your patience and help! 

Your routing table looks wrong to me.

I thought you said eth0 was your local network interface, instead it looks like eth0, eth1, eth2, bridge3 are all configured to route packets to your local network (which won't work unless they are all physically connected to the local network too).
Also any packets to outside destinations (default gateway / 0.0.0.0 ) are being routed via eth1.

Can you also post the output of ifconfig ?

All four NICs are physically connected to the local network. At one point, my plan was to set up VLANs or subnets for segmenting different network tasks, but I haven't gotten around to actually setting anything up yet. Would you suggest changing the routing somehow? And, if so, how do I do that?

I also tried forwarding the ports to eth1 instead of eth0 with the same results.

Unfortunately you xxx'ed out the 10.xx IP address in the earlier post when you where monitoring incoming packets with tcpdump so I can't tell for sure, but I would guess that your server is receiving the initial handshake packets on eth0 and then sending responses on eth1 (since that's the default gw).
Your NAT-router will likely ignore these responses as they are arriving from the wrong address.

I think the best way to solve this is if you disconnect all but one eth port to the local lan (and unconfigure the respective interfaces), then check if it works correctly and only then think about how to make correct use of the additional interfaces again.

Also I don't have much experience with bridges, but AFAIK bridges are layer2 devices and therefore shouldn't have an IP address assigned? Why do you have a local lan IP address on the bridge3 device?

Ok,
resume:
(outside IP of router)
199.bla.bla.bla --| router | --10.179.1.1 (inside IP on router)

server with httpd and sshd ports open:
--eth0 --10.179.2.1
--eth1 --10.179.2.65
--eth2 --10.179.2.66
--eth3-- no IP since bridged --bridge3-- 10.179.2.67

1 default route set to inside IP defined on router -> correct
2. ?? what is default gateway setting on the interfaces?? - check in ifcfg-eth* and on bridge, you should have there the "GATEWAY=" stanza
3. ?? what is the card your packet arriving to? - you stated it with 'XXX' at your posted tcpdump - dont worry, this is inside IP and you can post it freely (dont post outside IP)
4. is you NetworkManager turned off?

Tux, the bridge setting is correct in this way, eg:
ethx:
DEVICE=eth0
TYPE=Ethernet
HWADDR=00:14:5E:C2:1E:40
ONBOOT=yes
NM_CONTROLLED=no
BRIDGE=br0

bridge:
DEVICE=br0
TYPE=Bridge
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=10.10.1.152
NETMASK=255.255.255.0
ONBOOT=yes

cheers,

This post has been edited by helikaon: Oct 19 2011, 05:27 AM

Sorry about the delayed response, was sick most of the week.

I tried unplugging all but eth0's cable and shutting off eth1-3. I also noticed that, despite not seeing NetworkManager installed, the configs were set to be NM_CONTROLLED = "yes". So I changed that to no. Still nothing. I'm completely stumped.

Me too, I don't know what else to suggest.

Hi,
you didnt answer my questions, so again:

1. is you ip forwardin on server on
cat /etc/sysctl.conf has net.ipv4.ip_forward = 1?
2. past output
netstat -r

3. i noticed now (i overlooked it) that your default route state 10.179.1.1 while your interfaces are 10.179.2.x is this on purpose? your default route points onto different network segment. ..
check your IP on router's 'inside' interface and check your default route, this seams to me weird, i'd expect your default route point to 10.179.2.1

4. output of
chkconfig --list | grep NetworkManager
chkconfig --list | grep network

5. what is default gateway setting on the interfaces
cat /etc/sysconfig/network-scripts/ifcfg-ethxx ??
gateway=??

cheers