You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feature request: implement Endpoint-Independent Mapping for UDP #77
Firestack have to give up its reliance on gonet.UDPConn (which handles outgoing unconnected UDP sockets just fine).
This is not going to be trivial though a few projects like SagerNet did once implement this (unsure if it worked as they don't seem to use it anymore).
The problem isn't port forwarding (if I may call it that) using gonet.DialUDP, the problem is, the protocol handlers firestack sets up (for TCP and UDP) are not called for unconnected UDP (and listening TCP) sockets.
Firestack have to give up its reliance on gonet.UDPConn (which handles outgoing unconnected UDP sockets just fine).
Don't think even the SagerNet code I shared above would work. In that,
Either: Android does not forward packets from unconnected sockets to the VPN tunnel:
Testing this may be easy: Listening on unspecifiedIP:port (over both UDP, TCP) using nc+Termux, say, would not generate any relevant logs in firestack (in Rethink, Configure -> Settings -> Log level must be set to Very Verbose).
Or: firestack must handle UDP on its own and not rely on netstack at all. That is, instead of delivering the UDP packet (injecting it in) to netstack:
This is incorrect. nc -u -l -p 12345 does not send UDP packets until netcat receives one UDP packet.
Is there a way, then, for firestack to know if such a listener was started, so it can start an egress listener?
Also, I guess, I am at a loss as to what an "endpoint-independent mapping for UDP" would look like... Does it mean, instead of dialing to an (remote) endpoint, 1 always announce UDP instead 2?
Is there a way, then, for firestack to know if such a listener was started, so it can start an egress listener?
On Linux you can use inotify API to monitor /proc/net/udp and /proc/net/tcp, without some permissions this method may not work on Android.
Also, I guess, I am at a loss as to what an "endpoint-independent mapping for UDP" would look like... Does it mean, instead of dialing to an (remote) endpoint, 1 always announce UDP instead 2?
1. https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L352 [↩](#user-content-fnref-0-044726a13dad48d457abd69cf7035293)
2. https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L345 [↩](#user-content-fnref-1-044726a13dad48d457abd69cf7035293)
3. Proxies (severs) currently _have_ to be excluded for them to work: https://github.com/celzero/rethink-app/issues/1399 [↩](#user-content-fnref-2-044726a13dad48d457abd69cf7035293)
Allow users to write some port forwarding rules? ( e.g. TCP 0.0.0.0:12346 -> 127.0.0.1:12345 )
I guess Netlink cmds might work too (would be surprised if they worked on Android but it might).
Ah, I think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes don't have the same effective UID.
Allow users to write some port forwarding rules? (TCP 0.0.0.0:12346 -> 127.0.0.1:12345)
SagerNet seems to be doing some stuff with this, but I am not sure exactly what it does:
I think, firestack needs to create a new gonet.UDPConnwithout using udp.ForwarderRequest.CreateEndpoint + gonet.NewUDPConn (as that makes a connected UDP socket, per docs):
think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes
Believe apps part of the VPN tunnel in Android are in a different network namespace and as such must not be able to stomp on the VPN app's sockets?
Using the other option (unconnected UDP over sockopt+reuse) the VPN app NATs the traffic from its listening port (bound to actual underlying network) onto the same port bound to the TUN device, if any. Which is the original proposal, that would work, would it not?
Why does firestack send SYN+ACK packets immediately after receiving SYN packets
netstack's gonet API requires for its internal book-keeping (that is, routing subsequent packets from the same TCP flow to an already established gonet.TCPConn).
Does this break anything apart from confusing apps into thinking they've connected to remote while they may infact be blocked later due to a firewall rule?
think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes
Believe apps part of the VPN tunnel in Android are in a different network namespace and as such must not be able to stomp on the VPN app's sockets?
Using the other option (unconnected UDP over sockopt+reuse) the VPN app NATs the traffic from its listening port (bound to actual underlying network) onto the same port bound to the TUN device, if any. Which is the original proposal, that would work, would it not?
I don't know if apps on android are in different namespaces, 😂 but on android every app has its own user, you can check with adb shell sh -c "ps -d|cat -" . The reason why I mentioned the setsockopt trick a few days ago is just I thought you might not like unconnected sockets.
Please read https://man.voidlinux.org/man7/socket.7#SO_REUSEPORT
Possibly the biggest downside, though, is that a service listening on the wildcard INADDR_ANY address claims the port number exclusively for itself. Binding over the wildcard-listening socket with a specific IP and port fails miserably due to the address already being taken (EADDRINUSE).
Unless your service is UDP-only, setting the SO_REUSEADDR socket option, will not help you overcome this restriction. The only way out is to turn to SO_REUSEPORT, normally used to construct a load-balancing socket group. And that is only if you are lucky enough to run the port-conflicting services as the same user (UID). That is a story for another post.
The reason why I mentioned the setsockopt trick a few days ago is just I thought you might not like unconnected sockets.
I don't like (or understand much of) networking in general, but here we are ;)
(but no, I don't hold any special grudge on unconnected sockets)
I don't know if apps on android are in different namespaces
I am not sure, either (though this must be easy to find out...). Even so, the apps (by default) can't really bind to underlying interfaces (wifi, mobile, usb, etc) and so them listening on a wildcard addr:port (say tcp/udp [::]:8080) means listening on (tcp/udp) packets sent to 8080 to the TUN device only? I take what you're saying is, despite that, the Kernel wouldn't let both (the VPN app and the installed app) bind to the same port (8080, in this case) regardless of routing rules (as the effective UIDs are different), yeah?.
Btw, checking if you working on this change (so we both don't end up impl it)?
I am not sure, either (though this must be easy to find out...). Even so, the apps (by default) can't really bind to underlying interfaces (wifi, mobile, usb, etc) and so them listening on a wildcard addr:port (say tcp/udp [::]:8080) means listening on (tcp/udp) packets sent to 8080 to the TUN device only?
Not only the TUN device, they can also receive packets sent to [::1]:8080 and [ the Link-Local Address of dummy0 ]:8080.
I take what you're saying is, despite that, the Kernel wouldn't let both (the VPN app and the installed app) bind to the same port (8080, in this case) regardless of routing rules (as the effective UIDs are different), yeah?.
netstack's gonet API requires for its internal book-keeping (that is, routing subsequent packets from the same TCP flow to an already established gonet.TCPConn).
@ignoramous
Can't we call CreateEndpoint() and Complete() after Dial() return? Let me see what will happen.
🧐
It looks to me that by default netstack's fdbased runs a "single processor". The client may have to opt-in (via ProcessorsPerChannel) to run multiple of those. firestack runs 6.
unshare -U -r -n
$ ip link set lo up
$ ip tuntap add mode tun name tun0
$ ip link set tun0 mtu 1500
$ ip link set tun0 up
$ ip addr add 10.0.2.100/24 dev tun0
$ ip addr add fd00::100/64 dev tun0
$ ip route add 0.0.0.0/0 dev tun0
$ ip route add ::/0 dev tun0
$ echo $$
3000
In the second terminal
$ go build -trimpath ./linux-tool/toy
$ ./toy -mode main -tun tun0 -target 3000
And would you like to introduce a CLI "frontend" to test some functions on Linux
Nice. I'll merge this if you'll PR it or I can cherry-pick relevant commits from your branch (if it is public).
gvisor's built-in TCP Forwarder always runs the handler in a new goroutine, so a "blocking" handler can't block the netstack "processor.
iirc, in our testing (must be mid ~2023), handling TCP + ICMP in quick succession (or some such) stun netstack into submission & introduction of goroutines "fixed" it. It could very well have been that our handling of ICMP was problematic & TCP probably had nothing to do with what we observed.
Activity
ignoramous commentedon Jul 28, 2024
Firestack have to give up its reliance on gonet.UDPConn (which handles outgoing unconnected UDP sockets just fine).
This is not going to be trivial though a few projects like SagerNet did once implement this (unsure if it worked as they don't seem to use it anymore).
https://github.com/SagerNet/LibSagerNetCore/blob/1fce969ea5/gvisor/udp.go (old) / https://github.com/SagerNet/sing-tun/blob/aecfc190f4/stack_gvisor_udp.go (new).
Welcome PR, for sure. :D
Also: xjasonlyu/tun2socks#177
ignoramous commentedon Jul 29, 2024
Curious, how does this work?
Lanius-collaris commentedon Jul 29, 2024
Set two sockopt, SO_REUSEADDR and SO_REUSEPORT.
Lanius-collaris commentedon Jul 30, 2024
Why? I am not familiar with gvisor, but its document claims
gonet.DialUDP()can create unconnected UDPConn.https://pkg.go.dev/gvisor.dev/gvisor@v0.0.0-20240730175157-c671ffdf47b2/pkg/tcpip/adapters/gonet#DialUDP
ignoramous commentedon Jul 30, 2024
The problem isn't port forwarding (if I may call it that) using
gonet.DialUDP, the problem is, the protocol handlers firestack sets up (for TCP and UDP) are not called for unconnected UDP (and listening TCP) sockets.Don't think even the SagerNet code I shared above would work. In that,
Testing this may be easy: Listening on
unspecifiedIP:port(over both UDP, TCP) using nc+Termux, say, would not generate any relevant logs in firestack (in Rethink, Configure -> Settings -> Log level must be set to Very Verbose).firestack/intra/netstack/fdbased.go
Line 505 in 7ea0539
Lanius-collaris commentedon Jul 30, 2024
This is incorrect.
nc -u -l -p 12345does not send UDP packets until netcat receives one UDP packet.ignoramous commentedon Jul 30, 2024
Is there a way, then, for firestack to know if such a listener was started, so it can start an egress listener?
Also, I guess, I am at a loss as to what an "endpoint-independent mapping for UDP" would look like... Does it mean, instead of dialing to an (remote) endpoint, 1 always announce UDP instead 2?
What could we do for TCP 3? Port forwarding?
Footnotes
https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L352 ↩
https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L345 ↩
Proxies (severs) currently have to be excluded for them to work: https://github.com/celzero/rethink-app/issues/1399 ↩
Lanius-collaris commentedon Jul 30, 2024
On Linux you can use inotify API to monitor
/proc/net/udpand/proc/net/tcp, without some permissions this method may not work on Android.Yes.
Allow users to write some port forwarding rules? ( e.g. TCP 0.0.0.0:12346 -> 127.0.0.1:12345 )
ignoramous commentedon Jul 30, 2024
Guess,
Proxy.Announcewould need to be paired withgonet.DialUDPinstead ofgonet.NewUDPConn?firestack/intra/netstack/udp.go
Line 149 in 7ea0539
I guess Netlink cmds might work too (would be surprised if they worked on Android but it might).
SagerNet seems to be doing some stuff with this, but I am not sure exactly what it does:
Lanius-collaris commentedon Jul 31, 2024
gonet.DialUDP()callsgonet.NewUDPConn()internally.See https://github.com/google/gvisor/blob/4542eb5ba62b/pkg/tcpip/adapters/gonet/gonet.go#L588
Ah, I think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes don't have the same effective UID.
I guess these two commits allow packets having multicast or broadcast destination address to bypass the userspace netstack.
A off topic question: Why does firestack send SYN+ACK packets immediately after receiving SYN packets?
ignoramous commentedon Jul 31, 2024
I think, firestack needs to create a new
gonet.UDPConnwithout usingudp.ForwarderRequest.CreateEndpoint+gonet.NewUDPConn(as that makes a connected UDP socket, per docs):firestack/intra/netstack/udp.go
Line 143 in 7ea0539
Believe apps part of the VPN tunnel in Android are in a different network namespace and as such must not be able to stomp on the VPN app's sockets?
Using the other option (unconnected UDP over sockopt+reuse) the VPN app NATs the traffic from its listening port (bound to actual underlying network) onto the same port bound to the TUN device, if any. Which is the original proposal, that would work, would it not?
netstack's
gonetAPI requires for its internal book-keeping (that is, routing subsequent packets from the same TCP flow to an already establishedgonet.TCPConn).firestack/intra/netstack/tcp.go
Lines 116 to 117 in 7ea0539
Does this break anything apart from confusing apps into thinking they've connected to remote while they may infact be blocked later due to a firewall rule?
Lanius-collaris commentedon Jul 31, 2024
I don't know if apps on android are in different namespaces, 😂 but on android every app has its own user, you can check with
adb shell sh -c "ps -d|cat -". The reason why I mentioned the setsockopt trick a few days ago is just I thought you might not like unconnected sockets.Please read https://man.voidlinux.org/man7/socket.7#SO_REUSEPORT
ignoramous commentedon Jul 31, 2024
Thanks. Now, I recall reading a Cloudflare blog post about
so_reuseport(mirror).I don't like (or understand much of) networking in general, but here we are ;)
(but no, I don't hold any special grudge on unconnected sockets)
I am not sure, either (though this must be easy to find out...). Even so, the apps (by default) can't really bind to underlying interfaces (wifi, mobile, usb, etc) and so them listening on a wildcard addr:port (say tcp/udp
[::]:8080) means listening on (tcp/udp) packets sent to8080to the TUN device only? I take what you're saying is, despite that, the Kernel wouldn't let both (the VPN app and the installed app) bind to the same port (8080, in this case) regardless of routing rules (as the effective UIDs are different), yeah?.Btw, checking if you working on this change (so we both don't end up impl it)?
Lanius-collaris commentedon Jul 31, 2024
Not only the TUN device, they can also receive packets sent to
[::1]:8080and[ the Link-Local Address of dummy0 ]:8080.Try this:
I am not working on this change.
ignoramous commentedon Aug 6, 2024
Related:
10 remaining items
hussainmohd-a commentedon Aug 30, 2024
f78f98d522be5fc4is demuxed to1.1.1.1,8.8.8.8,9.9.9.9, and8.8.4.4endpoints from:37288.udpmux: rmv vend() fn
udpmux: vend new routes into netstack only on ingress
netstack/udp: fix cond for demuxing onto existing dst
Lanius-collaris commentedon Oct 4, 2024
@ignoramous
Can't we call CreateEndpoint() and Complete() after Dial() return? Let me see what will happen.
🧐
ignoramous commentedon Oct 4, 2024
May work if in "Single Threaded" mode.
firestack/intra/settings/config.go
Lines 82 to 84 in f3a671e
Lanius-collaris commentedon Oct 20, 2024
I don't know if gvisor is in "Single Threaded" mode by default, but it works.
https://github.com/Lanius-collaris/gvisor-playground
(I use built-in
fdbased.New()and built-instack.New())ignoramous commentedon Oct 20, 2024
https://github.com/Lanius-collaris/gvisor-playground/blob/ba027236b8a490a6c7c426338d4fa965a1f49708/app/libs/aegis/stack/tcp.go#L75 think that wait might block all other processing in netstack.
It looks to me that by default netstack's fdbased runs a "single processor". The client may have to opt-in (via
ProcessorsPerChannel) to run multiple of those. firestack runs 6.Lanius-collaris commentedon Oct 20, 2024
@ignoramous
Wait()is called by another goroutine, can you seego func(){ ... }()? I guess that you reachedmaxInFlight( It is 64 in demo_cli ). Try setting a smaller timeout inDialTCPFnand increasing maxInFlight. firestack calledComplete()too early, so it is unlikely that firestack will reachmaxInFlight.https://github.com/google/gvisor/blob/17edc7260e94/pkg/tcpip/transport/tcp/forwarder.go#L135
default ProcessorsPerChannel >= 1
https://github.com/google/gvisor/blob/17edc7260e94/pkg/tcpip/link/fdbased/endpoint.go#L333-L335
ignoramous commentedon Oct 23, 2024
Gotcha! Thanks.
Attempt: b139771
netstack: disable early connects for gonet conns
tcp,udp: use baseHandler.onInflow for ReverseProxy
Lanius-collaris commentedon Apr 26, 2025
@ignoramous
firestack/intra/netstack/tcp.go
Lines 121 to 134 in 18e9d90
This is not enough, you need to adjust
intra/tcp.goas well:https://github.com/google/gvisor/blob/60ec4e7a009d/pkg/tcpip/transport/tcp/forwarder.go#L101
gvisor's built-in TCP Forwarder always runs the handler in a new goroutine, so a "blocking" handler can't block the netstack "processor".🤣
And would you like to introduce a CLI "frontend" to test some functions on Linux? The Go mobile is too slow. My patch:
Usage:
In the first terminal
In the second terminal
ignoramous commentedon May 23, 2025
Nice. I'll merge this if you'll PR it or I can cherry-pick relevant commits from your branch (if it is public).
iirc, in our testing (must be mid ~2023), handling TCP + ICMP in quick succession (or some such) stun netstack into submission & introduction of goroutines "fixed" it. It could very well have been that our handling of ICMP was problematic & TCP probably had nothing to do with what we observed.
Lanius-collaris commentedon May 26, 2025
I uploaded the patch to the quick-test1 branch, it's not ready for opening a pull request.