Repository navigation
Using Source NAT option on Private Gateway does not work #2680
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
VPC (Private Gateway Source NAT)
CLOUDSTACK VERSION
At least: 4.9, 4.10, and 4.11
CONFIGURATION
When you use the Source NAT feature of Private Gateways on a VPC. This should Source NAT all traffic from CloudStack VMs going towards IPs reachable through Private Gateways
OS / ENVIRONMENT
I happen to be running KVM, but shouldn't be hypervisor specific. it is an issue in the SystemVM / VPC Router
SUMMARY
There is a bug in the Private Gateway functionality, when Source NAT is enabled for the Private Gateway. When the SNAT is added to iptables, it has the source CIDR of the private gateway subnet. Since no VMs live in that private gateway subnet, the SNAT doesn’t work.
STEPS TO REPRODUCE
Below is an example:
- VMs have IP addresses in the 10.0.0.0/24 subnet.
- The Private Gateway address is 10.101.141.2/30
In the outputs below, the SOURCE field for the new SNAT (eth3) only matches if the source is 10.101.141.0/30. Since the VM has an IP address in 10.0.0.0/24, the VMs don’t get SNAT’d as they should when talking across the private gateway. The SOURCE should be set to ANYWHERE.
BEFORE ADDING PRIVATE GATEWAY
Chain POSTROUTING (policy ACCEPT 1 packets, 52 bytes)
pkts bytes target prot opt in out source destination
2 736 SNAT all -- any eth2 10.0.0.0/24 anywhere to:10.0.0.1
16 1039 SNAT all -- any eth1 anywhere anywhere to:46.99.52.18
EXPECTED RESULTS
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 SNAT all -- any eth3 anywhere anywhere to:10.101.141.2
2 736 SNAT all -- any eth2 anywhere anywhere to:10.0.0.1
23 1515 SNAT all -- any eth1 anywhere anywhere to:46.99.52.18
ACTUAL RESULTS
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 SNAT all -- any eth3 10.101.141.0/30 anywhere to:10.101.141.2
2 736 SNAT all -- any eth2 10.0.0.0/24 anywhere to:10.0.0.1
23 1515 SNAT all -- any eth1 anywhere anywhere to:46.99.52.18
SUGGESTED FIX
It looks like CsAddress.py treats the creation of the Private Gateway SNAT as if it is a GUEST network, which works fine, except for the SNAT problem shown above. Here is the code from MASTER (line 479 is SNAT rule):
if self.get_type() in ["guest"]:
...
...
self.fw.append(["nat", "front",
"-A POSTROUTING -s %s -o %s -j SNAT --to-source %s" %
(guestNetworkCidr, self.dev, self.address['public_ip'])])
I am thinking we just change that to the following. I can’t think of any reason we need the source/guest CIDR specified:
if self.get_type() in ["guest"]:
...
...
self.fw.append(["nat", "front",
"-A POSTROUTING -o %s -j SNAT --to-source %s" %
(self.dev, self.address['public_ip'])])
More Information
Here is a link I found that further documents the Private Gateway Source NAT feature:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Source+NAT+on+private+gateway
Notice, how in the SNAT shown it does not specify a source guest network CIDR - my above fix matches the example in the link above.
Activity
ustcweizhou commentedon May 30, 2018
@Slair1 Thanks for you reply on jira.
You are correct. VPC connects the entire network in DC via private gateway, not entire internet. The routes (default and to static route) are good.
Looking at the code, I think the iptables rules should be added by
see line 584 to line 590 in
https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/cs/CsAddress.py
Could you post /etc/cloudstack/ips.json in the virtual router ?
yadvr commentedon May 31, 2018
@PaulAngus @DaanHoogland I've tentatively marked this against the 4.11.1.0 milestone, given the problem may have been in ACS since version 4.9, please triage if this should be moved to 4.11.2.0 instead.
Slair1 commentedon Jun 1, 2018
@ustcweizhou Could you take a look at this pull request?
#2681
I made the change in a different area of CsAddress.py. CloudStack treats the Private Gateway creation as a network type of Guest not Public. The section you updated updates if the type is Public. The code i updated is for when the type is Guest. We are currently running with our patch in place and it seems to work well, however, would like some additional review on the PR.
Thanks!!
ustcweizhou commentedon Jun 2, 2018
@Slair1 I saw your PR.
I am using 4.7.1. The ip in private gateway network is Public, not Guest. It seems to be a regression issue.
If the type is Guest, your PR should fix the issue. However, it will impact all guest networks (vpc tiers), I am not sure if it will lead to security issues.
Is it good to add a check before the code ?
the source_nat should be true for the ip in private gateway network, but false for ips in guest networks. This is why I asked you if you can post the /etc/cloudstack/ips.json .
yadvr commentedon Jun 7, 2018
@Slair1 can you reply to @ustcweizhou 's comment ^^ ?
yadvr commentedon Jun 8, 2018
I went ahead and applied @ustcweizhou 's advised fix on your PR @Slair1 /cc @PaulAngus @DaanHoogland
DaanHoogland commentedon Jun 8, 2018
noticed, lgtm
Slair1 commentedon Jun 8, 2018
@rhtyd @ustcweizhou @DaanHoogland
Actually, thinking about this some more, why are we even creating a SNAT for traffic destined towards a guest network? Looking at the existing functionality:
eth2is the interface connected to the guest network. That SNAT doesn't even make sense. Nothing should have a source address of 10.0.0.0/24 if it is destined to the guest network (10.0.0.0/24). The only traffic with a source of 10.0.0.0/24 would be traffic originating from the vRouter itself - which would already have a source IP of 10.0.0.1.Actually, we we shouldn't event have a SNAT at all, we should never be NAT'ing traffic destined towards a Guest network... The only exception to that is when using a Private Gateways with SNAT enabled - since a Private Gateway is treated as a Guest network.
Wouldn't this make more sense?
We don't even need this code at all:
Thoughts?
ustcweizhou commentedon Jun 9, 2018
I agree with @Slair1
yadvr commentedon Jun 11, 2018
@Slair1 /cc @ustcweizhou what if you want to acquire a public IP to attach to a VM directly (NAT), and in case one VM within the same network/tier want to access other VM by a public NAT-ed IP?
Slair1 commentedon Jun 11, 2018
@rhtyd /cc @ustcweizhou
I think you're right @rhtyd, it isn't obvious, but when a guest VM attempts to communicate with another guest VM's public IP, the packet flows into iptables and then hairpins back out. Without the SNAT in place, the reply from the destination VM would go straight to source VM. Which breaks any stateful, protocol such as TCP, since the source IP in the reply packet will be the private IP of the original destination VM while the original source VM is expecting the source IP to be the public IP... If that makes sense... if not let me know and i can explain more.
I'll take a look and update the code some!
Slair1 commentedon Jun 12, 2018
@rhtyd /cc @ustcweizhou
It looks like that old SNAT in
CsAddress.py(before this PR) was allowing the static NATs hair-pinning to work. However, that seems to only be because the actual static NAT hairpin code inconfigure.pywasn't working correctly because it hadeth0hard-coded in it instead of dynamic determining the guest network interface. So, I left ourCsAddress.pyas it (see current PR #2681) and modifiedconfigure.pyto fix the hairpin SNAT. Please see the following and let me know your thoughts. It may need some others to do some additional testing also.Before Changes to
configure.pyIP Address and Interfaces
OUTPUT DNAT (looks good)
PREROUTING DNAT (1st and 3rd line are not needed)
POSTROUTING SNAT (1st and 3rd line are incorrect. Last line isn't needed)
The 1st and 3rd line are close to getting hair-pinning with a static NAT working, but the interface is wrong. If you look above, eth0 was our 169.254.0.0/16 network...
After Changes to
configure.pyIP Address and Interfaces
OUTPUT DNAT
PREROUTING DNAT
POSTROUTING SNAT
That last line is still not needed, but not sure how/why that is there
Slair1 commentedon Jun 12, 2018
FYI, my test doesn't have PR #2579 - so i can't be 100% sure how my changes work with that PR. Unfortunately my test environment won't be available for a while to do more testing.
DaanHoogland commentedon Jun 14, 2018
did you add your changes to #2681, @Slair1 ? (or anywhere in this repo?)
Slair1 commentedon Jun 14, 2018
@DaanHoogland Yep, changes are in #2681
Source NAT option on Private Gateway (#2681)
yadvr commentedon Jun 20, 2018
PR was merged by Daan, closing this manually.