mirror of
https://github.com/amnezia-vpn/amneziawg-go.git
synced 2026-10-12 22:24:18 +03:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
79576e3f2e | ||
|
|
423aa96f94 | ||
|
|
5fbb7fdb08 | ||
|
|
a27b1dba96 | ||
|
|
0e9186510e | ||
|
|
78b47a89a8 | ||
|
|
dda3a71aec | ||
|
|
df87fbfcd6 | ||
|
|
336282f299 | ||
|
|
36431519bd | ||
|
|
9329840b5a | ||
|
|
ba6bb0bb8f | ||
|
|
c1670b484d | ||
|
|
b9d4759cef | ||
|
|
6035a275d2 | ||
|
|
c3bc566975 | ||
|
|
0be1878d38 | ||
|
|
f0cc315e5b | ||
|
|
392767a203 | ||
|
|
6c2c04ebd4 | ||
|
|
999cc3212d | ||
|
|
84cb7b4b61 | ||
|
|
e6fdad372b | ||
|
|
1ab8729390 | ||
|
|
1f0f5d5b7c |
No files matched your search
@@ -1,27 +1,24 @@
|
||||
# Go Implementation of AmneziaWG
|
||||
# Go Implementation of [WireGuard](https://www.wireguard.com/)
|
||||
|
||||
AmneziaWG is a contemporary version of the WireGuard protocol. It's a fork of WireGuard-Go and offers protection against detection by Deep Packet Inspection (DPI) systems. At the same time, it retains the simplified architecture and high performance of the original.
|
||||
|
||||
The precursor, WireGuard, is known for its efficiency but had issues with detection due to its distinctive packet signatures.
|
||||
AmneziaWG addresses this problem by employing advanced obfuscation methods, allowing its traffic to blend seamlessly with regular internet traffic.
|
||||
As a result, AmneziaWG maintains high performance while adding an extra layer of stealth, making it a superb choice for those seeking a fast and discreet VPN connection.
|
||||
This is an implementation of WireGuard in Go.
|
||||
|
||||
## Usage
|
||||
|
||||
Simply run:
|
||||
Most Linux kernel WireGuard users are used to adding an interface with `ip link add wg0 type wireguard`. With wireguard-go, instead simply run:
|
||||
|
||||
```
|
||||
$ amnezia-wg wg0
|
||||
$ wireguard-go wg0
|
||||
```
|
||||
|
||||
This will create an interface and fork into the background. To remove the interface, use the usual `ip link del wg0`, or if your system does not support removing interfaces directly, you may instead remove the control socket via `rm -f /var/run/wireguard/wg0.sock`, which will result in wireguard-go shutting down.
|
||||
|
||||
To run amnezia-wg without forking to the background, pass `-f` or `--foreground`:
|
||||
To run wireguard-go without forking to the background, pass `-f` or `--foreground`:
|
||||
|
||||
```
|
||||
$ amnezia-wg -f wg0
|
||||
$ wireguard-go -f wg0
|
||||
```
|
||||
When an interface is running, you may use [`amnezia-wg-tools `](https://github.com/amnezia-vpn/amnezia-wg-tools) to configure it, as well as the usual `ip(8)` and `ifconfig(8)` commands.
|
||||
|
||||
When an interface is running, you may use [`wg(8)`](https://git.zx2c4.com/wireguard-tools/about/src/man/wg.8) to configure it, as well as the usual `ip(8)` and `ifconfig(8)` commands.
|
||||
|
||||
To run with more logging you may set the environment variable `LOG_LEVEL=debug`.
|
||||
|
||||
@@ -29,24 +26,52 @@ To run with more logging you may set the environment variable `LOG_LEVEL=debug`.
|
||||
|
||||
### Linux
|
||||
|
||||
This will run on Linux; you should run amnezia-wg instead of using default linux kernel module.
|
||||
This will run on Linux; however you should instead use the kernel module, which is faster and better integrated into the OS. See the [installation page](https://www.wireguard.com/install/) for instructions.
|
||||
|
||||
### macOS
|
||||
|
||||
This runs on macOS using the utun driver. It does not yet support sticky sockets, and won't support fwmarks because of Darwin limitations. Since the utun driver cannot have arbitrary interface names, you must either use `utun[0-9]+` for an explicit interface name or `utun` to have the kernel select one for you. If you choose `utun` as the interface name, and the environment variable `WG_TUN_NAME_FILE` is defined, then the actual name of the interface chosen by the kernel is written to the file specified by that variable.
|
||||
This runs on MacOS, you should use it from [awg-apple](https://github.com/amnezia-vpn/awg-apple)
|
||||
|
||||
### Windows
|
||||
|
||||
This runs on Windows, you should use it from [awg-windows](https://github.com/amnezia-vpn/awg-windows), which uses this as a module.
|
||||
This runs on Windows, but you should instead use it from the more [fully featured Windows app](https://git.zx2c4.com/wireguard-windows/about/), which uses this as a module.
|
||||
|
||||
### FreeBSD
|
||||
|
||||
This will run on FreeBSD. It does not yet support sticky sockets. Fwmark is mapped to `SO_USER_COOKIE`.
|
||||
|
||||
### OpenBSD
|
||||
|
||||
This will run on OpenBSD. It does not yet support sticky sockets. Fwmark is mapped to `SO_RTABLE`. Since the tun driver cannot have arbitrary interface names, you must either use `tun[0-9]+` for an explicit interface name or `tun` to have the program select one for you. If you choose `tun` as the interface name, and the environment variable `WG_TUN_NAME_FILE` is defined, then the actual name of the interface chosen by the kernel is written to the file specified by that variable.
|
||||
|
||||
## Building
|
||||
|
||||
This requires an installation of the latest version of [Go](https://go.dev/).
|
||||
|
||||
```
|
||||
$ git clone https://github.com/amnezia-vpn/amnezia-wg
|
||||
$ cd amnezia-wg
|
||||
$ git clone https://git.zx2c4.com/wireguard-go
|
||||
$ cd wireguard-go
|
||||
$ make
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Copyright (C) 2017-2023 WireGuard LLC. All Rights Reserved.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
of the Software, and to permit persons to whom the Software is furnished to do
|
||||
so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
+136
-272
@@ -8,7 +8,6 @@ package conn
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/netip"
|
||||
"runtime"
|
||||
@@ -30,19 +29,16 @@ var (
|
||||
// methods for sending and receiving multiple datagrams per-syscall. See the
|
||||
// proposal in https://github.com/golang/go/issues/45886#issuecomment-1218301564.
|
||||
type StdNetBind struct {
|
||||
mu sync.Mutex // protects all fields except as specified
|
||||
ipv4 *net.UDPConn
|
||||
ipv6 *net.UDPConn
|
||||
ipv4PC *ipv4.PacketConn // will be nil on non-Linux
|
||||
ipv6PC *ipv6.PacketConn // will be nil on non-Linux
|
||||
ipv4TxOffload bool
|
||||
ipv4RxOffload bool
|
||||
ipv6TxOffload bool
|
||||
ipv6RxOffload bool
|
||||
mu sync.Mutex // protects all fields except as specified
|
||||
ipv4 *net.UDPConn
|
||||
ipv6 *net.UDPConn
|
||||
ipv4PC *ipv4.PacketConn // will be nil on non-Linux
|
||||
ipv6PC *ipv6.PacketConn // will be nil on non-Linux
|
||||
|
||||
// these two fields are not guarded by mu
|
||||
udpAddrPool sync.Pool
|
||||
msgsPool sync.Pool
|
||||
// these three fields are not guarded by mu
|
||||
udpAddrPool sync.Pool
|
||||
ipv4MsgsPool sync.Pool
|
||||
ipv6MsgsPool sync.Pool
|
||||
|
||||
blackhole4 bool
|
||||
blackhole6 bool
|
||||
@@ -58,14 +54,23 @@ func NewStdNetBind() Bind {
|
||||
},
|
||||
},
|
||||
|
||||
msgsPool: sync.Pool{
|
||||
ipv4MsgsPool: sync.Pool{
|
||||
New: func() any {
|
||||
msgs := make([]ipv4.Message, IdealBatchSize)
|
||||
for i := range msgs {
|
||||
msgs[i].Buffers = make(net.Buffers, 1)
|
||||
msgs[i].OOB = make([]byte, srcControlSize)
|
||||
}
|
||||
return &msgs
|
||||
},
|
||||
},
|
||||
|
||||
ipv6MsgsPool: sync.Pool{
|
||||
New: func() any {
|
||||
// ipv6.Message and ipv4.Message are interchangeable as they are
|
||||
// both aliases for x/net/internal/socket.Message.
|
||||
msgs := make([]ipv6.Message, IdealBatchSize)
|
||||
for i := range msgs {
|
||||
msgs[i].Buffers = make(net.Buffers, 1)
|
||||
msgs[i].OOB = make([]byte, 0, stickyControlSize+gsoControlSize)
|
||||
msgs[i].OOB = make([]byte, srcControlSize)
|
||||
}
|
||||
return &msgs
|
||||
},
|
||||
@@ -108,7 +113,7 @@ func (e *StdNetEndpoint) DstIP() netip.Addr {
|
||||
return e.AddrPort.Addr()
|
||||
}
|
||||
|
||||
// See control_default,linux, etc for implementations of SrcIP and SrcIfidx.
|
||||
// See sticky_default,linux, etc for implementations of SrcIP and SrcIfidx.
|
||||
|
||||
func (e *StdNetEndpoint) DstToBytes() []byte {
|
||||
b, _ := e.AddrPort.MarshalBinary()
|
||||
@@ -174,21 +179,19 @@ again:
|
||||
}
|
||||
var fns []ReceiveFunc
|
||||
if v4conn != nil {
|
||||
s.ipv4TxOffload, s.ipv4RxOffload = supportsUDPOffload(v4conn)
|
||||
if runtime.GOOS == "linux" || runtime.GOOS == "android" {
|
||||
if runtime.GOOS == "linux" {
|
||||
v4pc = ipv4.NewPacketConn(v4conn)
|
||||
s.ipv4PC = v4pc
|
||||
}
|
||||
fns = append(fns, s.makeReceiveIPv4(v4pc, v4conn, s.ipv4RxOffload))
|
||||
fns = append(fns, s.makeReceiveIPv4(v4pc, v4conn))
|
||||
s.ipv4 = v4conn
|
||||
}
|
||||
if v6conn != nil {
|
||||
s.ipv6TxOffload, s.ipv6RxOffload = supportsUDPOffload(v6conn)
|
||||
if runtime.GOOS == "linux" || runtime.GOOS == "android" {
|
||||
if runtime.GOOS == "linux" {
|
||||
v6pc = ipv6.NewPacketConn(v6conn)
|
||||
s.ipv6PC = v6pc
|
||||
}
|
||||
fns = append(fns, s.makeReceiveIPv6(v6pc, v6conn, s.ipv6RxOffload))
|
||||
fns = append(fns, s.makeReceiveIPv6(v6pc, v6conn))
|
||||
s.ipv6 = v6conn
|
||||
}
|
||||
if len(fns) == 0 {
|
||||
@@ -198,101 +201,76 @@ again:
|
||||
return fns, uint16(port), nil
|
||||
}
|
||||
|
||||
func (s *StdNetBind) putMessages(msgs *[]ipv6.Message) {
|
||||
for i := range *msgs {
|
||||
(*msgs)[i].OOB = (*msgs)[i].OOB[:0]
|
||||
(*msgs)[i] = ipv6.Message{Buffers: (*msgs)[i].Buffers, OOB: (*msgs)[i].OOB}
|
||||
}
|
||||
s.msgsPool.Put(msgs)
|
||||
}
|
||||
|
||||
func (s *StdNetBind) getMessages() *[]ipv6.Message {
|
||||
return s.msgsPool.Get().(*[]ipv6.Message)
|
||||
}
|
||||
|
||||
var (
|
||||
// If compilation fails here these are no longer the same underlying type.
|
||||
_ ipv6.Message = ipv4.Message{}
|
||||
)
|
||||
|
||||
type batchReader interface {
|
||||
ReadBatch([]ipv6.Message, int) (int, error)
|
||||
}
|
||||
|
||||
type batchWriter interface {
|
||||
WriteBatch([]ipv6.Message, int) (int, error)
|
||||
}
|
||||
|
||||
func (s *StdNetBind) receiveIP(
|
||||
br batchReader,
|
||||
conn *net.UDPConn,
|
||||
rxOffload bool,
|
||||
bufs [][]byte,
|
||||
sizes []int,
|
||||
eps []Endpoint,
|
||||
) (n int, err error) {
|
||||
msgs := s.getMessages()
|
||||
for i := range bufs {
|
||||
(*msgs)[i].Buffers[0] = bufs[i]
|
||||
(*msgs)[i].OOB = (*msgs)[i].OOB[:cap((*msgs)[i].OOB)]
|
||||
}
|
||||
defer s.putMessages(msgs)
|
||||
var numMsgs int
|
||||
if runtime.GOOS == "linux" || runtime.GOOS == "android" {
|
||||
if rxOffload {
|
||||
readAt := len(*msgs) - (IdealBatchSize / udpSegmentMaxDatagrams)
|
||||
numMsgs, err = br.ReadBatch((*msgs)[readAt:], 0)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
numMsgs, err = splitCoalescedMessages(*msgs, readAt, getGSOSize)
|
||||
func (s *StdNetBind) makeReceiveIPv4(pc *ipv4.PacketConn, conn *net.UDPConn) ReceiveFunc {
|
||||
return func(bufs [][]byte, sizes []int, eps []Endpoint) (n int, err error) {
|
||||
msgs := s.ipv4MsgsPool.Get().(*[]ipv4.Message)
|
||||
defer s.ipv4MsgsPool.Put(msgs)
|
||||
for i := range bufs {
|
||||
(*msgs)[i].Buffers[0] = bufs[i]
|
||||
}
|
||||
var numMsgs int
|
||||
if runtime.GOOS == "linux" {
|
||||
numMsgs, err = pc.ReadBatch(*msgs, 0)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
} else {
|
||||
numMsgs, err = br.ReadBatch(*msgs, 0)
|
||||
msg := &(*msgs)[0]
|
||||
msg.N, msg.NN, _, msg.Addr, err = conn.ReadMsgUDP(msg.Buffers[0], msg.OOB)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
numMsgs = 1
|
||||
}
|
||||
} else {
|
||||
msg := &(*msgs)[0]
|
||||
msg.N, msg.NN, _, msg.Addr, err = conn.ReadMsgUDP(msg.Buffers[0], msg.OOB)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
for i := 0; i < numMsgs; i++ {
|
||||
msg := &(*msgs)[i]
|
||||
sizes[i] = msg.N
|
||||
addrPort := msg.Addr.(*net.UDPAddr).AddrPort()
|
||||
ep := &StdNetEndpoint{AddrPort: addrPort} // TODO: remove allocation
|
||||
getSrcFromControl(msg.OOB[:msg.NN], ep)
|
||||
eps[i] = ep
|
||||
}
|
||||
numMsgs = 1
|
||||
}
|
||||
for i := 0; i < numMsgs; i++ {
|
||||
msg := &(*msgs)[i]
|
||||
sizes[i] = msg.N
|
||||
if sizes[i] == 0 {
|
||||
continue
|
||||
}
|
||||
addrPort := msg.Addr.(*net.UDPAddr).AddrPort()
|
||||
ep := &StdNetEndpoint{AddrPort: addrPort} // TODO: remove allocation
|
||||
getSrcFromControl(msg.OOB[:msg.NN], ep)
|
||||
eps[i] = ep
|
||||
}
|
||||
return numMsgs, nil
|
||||
}
|
||||
|
||||
func (s *StdNetBind) makeReceiveIPv4(pc *ipv4.PacketConn, conn *net.UDPConn, rxOffload bool) ReceiveFunc {
|
||||
return func(bufs [][]byte, sizes []int, eps []Endpoint) (n int, err error) {
|
||||
return s.receiveIP(pc, conn, rxOffload, bufs, sizes, eps)
|
||||
return numMsgs, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (s *StdNetBind) makeReceiveIPv6(pc *ipv6.PacketConn, conn *net.UDPConn, rxOffload bool) ReceiveFunc {
|
||||
func (s *StdNetBind) makeReceiveIPv6(pc *ipv6.PacketConn, conn *net.UDPConn) ReceiveFunc {
|
||||
return func(bufs [][]byte, sizes []int, eps []Endpoint) (n int, err error) {
|
||||
return s.receiveIP(pc, conn, rxOffload, bufs, sizes, eps)
|
||||
msgs := s.ipv6MsgsPool.Get().(*[]ipv6.Message)
|
||||
defer s.ipv6MsgsPool.Put(msgs)
|
||||
for i := range bufs {
|
||||
(*msgs)[i].Buffers[0] = bufs[i]
|
||||
}
|
||||
var numMsgs int
|
||||
if runtime.GOOS == "linux" {
|
||||
numMsgs, err = pc.ReadBatch(*msgs, 0)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
} else {
|
||||
msg := &(*msgs)[0]
|
||||
msg.N, msg.NN, _, msg.Addr, err = conn.ReadMsgUDP(msg.Buffers[0], msg.OOB)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
numMsgs = 1
|
||||
}
|
||||
for i := 0; i < numMsgs; i++ {
|
||||
msg := &(*msgs)[i]
|
||||
sizes[i] = msg.N
|
||||
addrPort := msg.Addr.(*net.UDPAddr).AddrPort()
|
||||
ep := &StdNetEndpoint{AddrPort: addrPort} // TODO: remove allocation
|
||||
getSrcFromControl(msg.OOB[:msg.NN], ep)
|
||||
eps[i] = ep
|
||||
}
|
||||
return numMsgs, nil
|
||||
}
|
||||
}
|
||||
|
||||
// TODO: When all Binds handle IdealBatchSize, remove this dynamic function and
|
||||
// rename the IdealBatchSize constant to BatchSize.
|
||||
func (s *StdNetBind) BatchSize() int {
|
||||
if runtime.GOOS == "linux" || runtime.GOOS == "android" {
|
||||
if runtime.GOOS == "linux" {
|
||||
return IdealBatchSize
|
||||
}
|
||||
return 1
|
||||
@@ -315,42 +293,28 @@ func (s *StdNetBind) Close() error {
|
||||
}
|
||||
s.blackhole4 = false
|
||||
s.blackhole6 = false
|
||||
s.ipv4TxOffload = false
|
||||
s.ipv4RxOffload = false
|
||||
s.ipv6TxOffload = false
|
||||
s.ipv6RxOffload = false
|
||||
if err1 != nil {
|
||||
return err1
|
||||
}
|
||||
return err2
|
||||
}
|
||||
|
||||
type ErrUDPGSODisabled struct {
|
||||
onLaddr string
|
||||
RetryErr error
|
||||
}
|
||||
|
||||
func (e ErrUDPGSODisabled) Error() string {
|
||||
return fmt.Sprintf("disabled UDP GSO on %s, NIC(s) may not support checksum offload", e.onLaddr)
|
||||
}
|
||||
|
||||
func (e ErrUDPGSODisabled) Unwrap() error {
|
||||
return e.RetryErr
|
||||
}
|
||||
|
||||
func (s *StdNetBind) Send(bufs [][]byte, endpoint Endpoint) error {
|
||||
s.mu.Lock()
|
||||
blackhole := s.blackhole4
|
||||
conn := s.ipv4
|
||||
offload := s.ipv4TxOffload
|
||||
br := batchWriter(s.ipv4PC)
|
||||
var (
|
||||
pc4 *ipv4.PacketConn
|
||||
pc6 *ipv6.PacketConn
|
||||
)
|
||||
is6 := false
|
||||
if endpoint.DstIP().Is6() {
|
||||
blackhole = s.blackhole6
|
||||
conn = s.ipv6
|
||||
br = s.ipv6PC
|
||||
pc6 = s.ipv6PC
|
||||
is6 = true
|
||||
offload = s.ipv6TxOffload
|
||||
} else {
|
||||
pc4 = s.ipv4PC
|
||||
}
|
||||
s.mu.Unlock()
|
||||
|
||||
@@ -360,185 +324,85 @@ func (s *StdNetBind) Send(bufs [][]byte, endpoint Endpoint) error {
|
||||
if conn == nil {
|
||||
return syscall.EAFNOSUPPORT
|
||||
}
|
||||
|
||||
msgs := s.getMessages()
|
||||
defer s.putMessages(msgs)
|
||||
ua := s.udpAddrPool.Get().(*net.UDPAddr)
|
||||
defer s.udpAddrPool.Put(ua)
|
||||
if is6 {
|
||||
as16 := endpoint.DstIP().As16()
|
||||
copy(ua.IP, as16[:])
|
||||
ua.IP = ua.IP[:16]
|
||||
return s.send6(conn, pc6, endpoint, bufs)
|
||||
} else {
|
||||
as4 := endpoint.DstIP().As4()
|
||||
copy(ua.IP, as4[:])
|
||||
ua.IP = ua.IP[:4]
|
||||
return s.send4(conn, pc4, endpoint, bufs)
|
||||
}
|
||||
ua.Port = int(endpoint.(*StdNetEndpoint).Port())
|
||||
var (
|
||||
retried bool
|
||||
err error
|
||||
)
|
||||
retry:
|
||||
if offload {
|
||||
n := coalesceMessages(ua, endpoint.(*StdNetEndpoint), bufs, *msgs, setGSOSize)
|
||||
err = s.send(conn, br, (*msgs)[:n])
|
||||
if err != nil && offload && errShouldDisableUDPGSO(err) {
|
||||
offload = false
|
||||
s.mu.Lock()
|
||||
if is6 {
|
||||
s.ipv6TxOffload = false
|
||||
} else {
|
||||
s.ipv4TxOffload = false
|
||||
}
|
||||
s.mu.Unlock()
|
||||
retried = true
|
||||
goto retry
|
||||
}
|
||||
} else {
|
||||
for i := range bufs {
|
||||
(*msgs)[i].Addr = ua
|
||||
(*msgs)[i].Buffers[0] = bufs[i]
|
||||
setSrcControl(&(*msgs)[i].OOB, endpoint.(*StdNetEndpoint))
|
||||
}
|
||||
err = s.send(conn, br, (*msgs)[:len(bufs)])
|
||||
}
|
||||
if retried {
|
||||
return ErrUDPGSODisabled{onLaddr: conn.LocalAddr().String(), RetryErr: err}
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *StdNetBind) send(conn *net.UDPConn, pc batchWriter, msgs []ipv6.Message) error {
|
||||
func (s *StdNetBind) send4(conn *net.UDPConn, pc *ipv4.PacketConn, ep Endpoint, bufs [][]byte) error {
|
||||
ua := s.udpAddrPool.Get().(*net.UDPAddr)
|
||||
as4 := ep.DstIP().As4()
|
||||
copy(ua.IP, as4[:])
|
||||
ua.IP = ua.IP[:4]
|
||||
ua.Port = int(ep.(*StdNetEndpoint).Port())
|
||||
msgs := s.ipv4MsgsPool.Get().(*[]ipv4.Message)
|
||||
for i, buf := range bufs {
|
||||
(*msgs)[i].Buffers[0] = buf
|
||||
(*msgs)[i].Addr = ua
|
||||
setSrcControl(&(*msgs)[i].OOB, ep.(*StdNetEndpoint))
|
||||
}
|
||||
var (
|
||||
n int
|
||||
err error
|
||||
start int
|
||||
)
|
||||
if runtime.GOOS == "linux" || runtime.GOOS == "android" {
|
||||
if runtime.GOOS == "linux" {
|
||||
for {
|
||||
n, err = pc.WriteBatch(msgs[start:], 0)
|
||||
if err != nil || n == len(msgs[start:]) {
|
||||
n, err = pc.WriteBatch((*msgs)[start:len(bufs)], 0)
|
||||
if err != nil || n == len((*msgs)[start:len(bufs)]) {
|
||||
break
|
||||
}
|
||||
start += n
|
||||
}
|
||||
} else {
|
||||
for _, msg := range msgs {
|
||||
_, _, err = conn.WriteMsgUDP(msg.Buffers[0], msg.OOB, msg.Addr.(*net.UDPAddr))
|
||||
for i, buf := range bufs {
|
||||
_, _, err = conn.WriteMsgUDP(buf, (*msgs)[i].OOB, ua)
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
s.udpAddrPool.Put(ua)
|
||||
s.ipv4MsgsPool.Put(msgs)
|
||||
return err
|
||||
}
|
||||
|
||||
const (
|
||||
// Exceeding these values results in EMSGSIZE. They account for layer3 and
|
||||
// layer4 headers. IPv6 does not need to account for itself as the payload
|
||||
// length field is self excluding.
|
||||
maxIPv4PayloadLen = 1<<16 - 1 - 20 - 8
|
||||
maxIPv6PayloadLen = 1<<16 - 1 - 8
|
||||
|
||||
// This is a hard limit imposed by the kernel.
|
||||
udpSegmentMaxDatagrams = 64
|
||||
)
|
||||
|
||||
type setGSOFunc func(control *[]byte, gsoSize uint16)
|
||||
|
||||
func coalesceMessages(addr *net.UDPAddr, ep *StdNetEndpoint, bufs [][]byte, msgs []ipv6.Message, setGSO setGSOFunc) int {
|
||||
var (
|
||||
base = -1 // index of msg we are currently coalescing into
|
||||
gsoSize int // segmentation size of msgs[base]
|
||||
dgramCnt int // number of dgrams coalesced into msgs[base]
|
||||
endBatch bool // tracking flag to start a new batch on next iteration of bufs
|
||||
)
|
||||
maxPayloadLen := maxIPv4PayloadLen
|
||||
if ep.DstIP().Is6() {
|
||||
maxPayloadLen = maxIPv6PayloadLen
|
||||
}
|
||||
func (s *StdNetBind) send6(conn *net.UDPConn, pc *ipv6.PacketConn, ep Endpoint, bufs [][]byte) error {
|
||||
ua := s.udpAddrPool.Get().(*net.UDPAddr)
|
||||
as16 := ep.DstIP().As16()
|
||||
copy(ua.IP, as16[:])
|
||||
ua.IP = ua.IP[:16]
|
||||
ua.Port = int(ep.(*StdNetEndpoint).Port())
|
||||
msgs := s.ipv6MsgsPool.Get().(*[]ipv6.Message)
|
||||
for i, buf := range bufs {
|
||||
if i > 0 {
|
||||
msgLen := len(buf)
|
||||
baseLenBefore := len(msgs[base].Buffers[0])
|
||||
freeBaseCap := cap(msgs[base].Buffers[0]) - baseLenBefore
|
||||
if msgLen+baseLenBefore <= maxPayloadLen &&
|
||||
msgLen <= gsoSize &&
|
||||
msgLen <= freeBaseCap &&
|
||||
dgramCnt < udpSegmentMaxDatagrams &&
|
||||
!endBatch {
|
||||
msgs[base].Buffers[0] = append(msgs[base].Buffers[0], buf...)
|
||||
if i == len(bufs)-1 {
|
||||
setGSO(&msgs[base].OOB, uint16(gsoSize))
|
||||
}
|
||||
dgramCnt++
|
||||
if msgLen < gsoSize {
|
||||
// A smaller than gsoSize packet on the tail is legal, but
|
||||
// it must end the batch.
|
||||
endBatch = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
}
|
||||
if dgramCnt > 1 {
|
||||
setGSO(&msgs[base].OOB, uint16(gsoSize))
|
||||
}
|
||||
// Reset prior to incrementing base since we are preparing to start a
|
||||
// new potential batch.
|
||||
endBatch = false
|
||||
base++
|
||||
gsoSize = len(buf)
|
||||
setSrcControl(&msgs[base].OOB, ep)
|
||||
msgs[base].Buffers[0] = buf
|
||||
msgs[base].Addr = addr
|
||||
dgramCnt = 1
|
||||
(*msgs)[i].Buffers[0] = buf
|
||||
(*msgs)[i].Addr = ua
|
||||
setSrcControl(&(*msgs)[i].OOB, ep.(*StdNetEndpoint))
|
||||
}
|
||||
return base + 1
|
||||
}
|
||||
|
||||
type getGSOFunc func(control []byte) (int, error)
|
||||
|
||||
func splitCoalescedMessages(msgs []ipv6.Message, firstMsgAt int, getGSO getGSOFunc) (n int, err error) {
|
||||
for i := firstMsgAt; i < len(msgs); i++ {
|
||||
msg := &msgs[i]
|
||||
if msg.N == 0 {
|
||||
return n, err
|
||||
}
|
||||
var (
|
||||
gsoSize int
|
||||
start int
|
||||
end = msg.N
|
||||
numToSplit = 1
|
||||
)
|
||||
gsoSize, err = getGSO(msg.OOB[:msg.NN])
|
||||
if err != nil {
|
||||
return n, err
|
||||
}
|
||||
if gsoSize > 0 {
|
||||
numToSplit = (msg.N + gsoSize - 1) / gsoSize
|
||||
end = gsoSize
|
||||
}
|
||||
for j := 0; j < numToSplit; j++ {
|
||||
if n > i {
|
||||
return n, errors.New("splitting coalesced packet resulted in overflow")
|
||||
var (
|
||||
n int
|
||||
err error
|
||||
start int
|
||||
)
|
||||
if runtime.GOOS == "linux" {
|
||||
for {
|
||||
n, err = pc.WriteBatch((*msgs)[start:len(bufs)], 0)
|
||||
if err != nil || n == len((*msgs)[start:len(bufs)]) {
|
||||
break
|
||||
}
|
||||
copied := copy(msgs[n].Buffers[0], msg.Buffers[0][start:end])
|
||||
msgs[n].N = copied
|
||||
msgs[n].Addr = msg.Addr
|
||||
start = end
|
||||
end += gsoSize
|
||||
if end > msg.N {
|
||||
end = msg.N
|
||||
}
|
||||
n++
|
||||
start += n
|
||||
}
|
||||
if i != n-1 {
|
||||
// It is legal for bytes to move within msg.Buffers[0] as a result
|
||||
// of splitting, so we only zero the source msg len when it is not
|
||||
// the destination of the last split operation above.
|
||||
msg.N = 0
|
||||
} else {
|
||||
for i, buf := range bufs {
|
||||
_, _, err = conn.WriteMsgUDP(buf, (*msgs)[i].OOB, ua)
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return n, nil
|
||||
s.udpAddrPool.Put(ua)
|
||||
s.ipv6MsgsPool.Put(msgs)
|
||||
return err
|
||||
}
|
||||
+1
-229
@@ -1,12 +1,6 @@
|
||||
package conn
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"net"
|
||||
"testing"
|
||||
|
||||
"golang.org/x/net/ipv6"
|
||||
)
|
||||
import "testing"
|
||||
|
||||
func TestStdNetBindReceiveFuncAfterClose(t *testing.T) {
|
||||
bind := NewStdNetBind().(*StdNetBind)
|
||||
@@ -26,225 +20,3 @@ func TestStdNetBindReceiveFuncAfterClose(t *testing.T) {
|
||||
fn(bufs, sizes, eps)
|
||||
}
|
||||
}
|
||||
|
||||
func mockSetGSOSize(control *[]byte, gsoSize uint16) {
|
||||
*control = (*control)[:cap(*control)]
|
||||
binary.LittleEndian.PutUint16(*control, gsoSize)
|
||||
}
|
||||
|
||||
func Test_coalesceMessages(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
buffs [][]byte
|
||||
wantLens []int
|
||||
wantGSO []int
|
||||
}{
|
||||
{
|
||||
name: "one message no coalesce",
|
||||
buffs: [][]byte{
|
||||
make([]byte, 1, 1),
|
||||
},
|
||||
wantLens: []int{1},
|
||||
wantGSO: []int{0},
|
||||
},
|
||||
{
|
||||
name: "two messages equal len coalesce",
|
||||
buffs: [][]byte{
|
||||
make([]byte, 1, 2),
|
||||
make([]byte, 1, 1),
|
||||
},
|
||||
wantLens: []int{2},
|
||||
wantGSO: []int{1},
|
||||
},
|
||||
{
|
||||
name: "two messages unequal len coalesce",
|
||||
buffs: [][]byte{
|
||||
make([]byte, 2, 3),
|
||||
make([]byte, 1, 1),
|
||||
},
|
||||
wantLens: []int{3},
|
||||
wantGSO: []int{2},
|
||||
},
|
||||
{
|
||||
name: "three messages second unequal len coalesce",
|
||||
buffs: [][]byte{
|
||||
make([]byte, 2, 3),
|
||||
make([]byte, 1, 1),
|
||||
make([]byte, 2, 2),
|
||||
},
|
||||
wantLens: []int{3, 2},
|
||||
wantGSO: []int{2, 0},
|
||||
},
|
||||
{
|
||||
name: "three messages limited cap coalesce",
|
||||
buffs: [][]byte{
|
||||
make([]byte, 2, 4),
|
||||
make([]byte, 2, 2),
|
||||
make([]byte, 2, 2),
|
||||
},
|
||||
wantLens: []int{4, 2},
|
||||
wantGSO: []int{2, 0},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range cases {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
addr := &net.UDPAddr{
|
||||
IP: net.ParseIP("127.0.0.1").To4(),
|
||||
Port: 1,
|
||||
}
|
||||
msgs := make([]ipv6.Message, len(tt.buffs))
|
||||
for i := range msgs {
|
||||
msgs[i].Buffers = make([][]byte, 1)
|
||||
msgs[i].OOB = make([]byte, 0, 2)
|
||||
}
|
||||
got := coalesceMessages(addr, &StdNetEndpoint{AddrPort: addr.AddrPort()}, tt.buffs, msgs, mockSetGSOSize)
|
||||
if got != len(tt.wantLens) {
|
||||
t.Fatalf("got len %d want: %d", got, len(tt.wantLens))
|
||||
}
|
||||
for i := 0; i < got; i++ {
|
||||
if msgs[i].Addr != addr {
|
||||
t.Errorf("msgs[%d].Addr != passed addr", i)
|
||||
}
|
||||
gotLen := len(msgs[i].Buffers[0])
|
||||
if gotLen != tt.wantLens[i] {
|
||||
t.Errorf("len(msgs[%d].Buffers[0]) %d != %d", i, gotLen, tt.wantLens[i])
|
||||
}
|
||||
gotGSO, err := mockGetGSOSize(msgs[i].OOB)
|
||||
if err != nil {
|
||||
t.Fatalf("msgs[%d] getGSOSize err: %v", i, err)
|
||||
}
|
||||
if gotGSO != tt.wantGSO[i] {
|
||||
t.Errorf("msgs[%d] gsoSize %d != %d", i, gotGSO, tt.wantGSO[i])
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func mockGetGSOSize(control []byte) (int, error) {
|
||||
if len(control) < 2 {
|
||||
return 0, nil
|
||||
}
|
||||
return int(binary.LittleEndian.Uint16(control)), nil
|
||||
}
|
||||
|
||||
func Test_splitCoalescedMessages(t *testing.T) {
|
||||
newMsg := func(n, gso int) ipv6.Message {
|
||||
msg := ipv6.Message{
|
||||
Buffers: [][]byte{make([]byte, 1<<16-1)},
|
||||
N: n,
|
||||
OOB: make([]byte, 2),
|
||||
}
|
||||
binary.LittleEndian.PutUint16(msg.OOB, uint16(gso))
|
||||
if gso > 0 {
|
||||
msg.NN = 2
|
||||
}
|
||||
return msg
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
msgs []ipv6.Message
|
||||
firstMsgAt int
|
||||
wantNumEval int
|
||||
wantMsgLens []int
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "second last split last empty",
|
||||
msgs: []ipv6.Message{
|
||||
newMsg(0, 0),
|
||||
newMsg(0, 0),
|
||||
newMsg(3, 1),
|
||||
newMsg(0, 0),
|
||||
},
|
||||
firstMsgAt: 2,
|
||||
wantNumEval: 3,
|
||||
wantMsgLens: []int{1, 1, 1, 0},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "second last no split last empty",
|
||||
msgs: []ipv6.Message{
|
||||
newMsg(0, 0),
|
||||
newMsg(0, 0),
|
||||
newMsg(1, 0),
|
||||
newMsg(0, 0),
|
||||
},
|
||||
firstMsgAt: 2,
|
||||
wantNumEval: 1,
|
||||
wantMsgLens: []int{1, 0, 0, 0},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "second last no split last no split",
|
||||
msgs: []ipv6.Message{
|
||||
newMsg(0, 0),
|
||||
newMsg(0, 0),
|
||||
newMsg(1, 0),
|
||||
newMsg(1, 0),
|
||||
},
|
||||
firstMsgAt: 2,
|
||||
wantNumEval: 2,
|
||||
wantMsgLens: []int{1, 1, 0, 0},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "second last no split last split",
|
||||
msgs: []ipv6.Message{
|
||||
newMsg(0, 0),
|
||||
newMsg(0, 0),
|
||||
newMsg(1, 0),
|
||||
newMsg(3, 1),
|
||||
},
|
||||
firstMsgAt: 2,
|
||||
wantNumEval: 4,
|
||||
wantMsgLens: []int{1, 1, 1, 1},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "second last split last split",
|
||||
msgs: []ipv6.Message{
|
||||
newMsg(0, 0),
|
||||
newMsg(0, 0),
|
||||
newMsg(2, 1),
|
||||
newMsg(2, 1),
|
||||
},
|
||||
firstMsgAt: 2,
|
||||
wantNumEval: 4,
|
||||
wantMsgLens: []int{1, 1, 1, 1},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "second last no split last split overflow",
|
||||
msgs: []ipv6.Message{
|
||||
newMsg(0, 0),
|
||||
newMsg(0, 0),
|
||||
newMsg(1, 0),
|
||||
newMsg(4, 1),
|
||||
},
|
||||
firstMsgAt: 2,
|
||||
wantNumEval: 4,
|
||||
wantMsgLens: []int{1, 1, 1, 1},
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range cases {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := splitCoalescedMessages(tt.msgs, 2, mockGetGSOSize)
|
||||
if err != nil && !tt.wantErr {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
if got != tt.wantNumEval {
|
||||
t.Fatalf("got to eval: %d want: %d", got, tt.wantNumEval)
|
||||
}
|
||||
for i, msg := range tt.msgs {
|
||||
if msg.N != tt.wantMsgLens[i] {
|
||||
t.Fatalf("msg[%d].N: %d want: %d", i, msg.N, tt.wantMsgLens[i])
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -57,13 +57,5 @@ func init() {
|
||||
}
|
||||
return err
|
||||
},
|
||||
|
||||
// Attempt to enable UDP_GRO
|
||||
func(network, address string, c syscall.RawConn) error {
|
||||
c.Control(func(fd uintptr) {
|
||||
_ = unix.SetsockoptInt(int(fd), unix.IPPROTO_UDP, unix.UDP_GRO, 1)
|
||||
})
|
||||
return nil
|
||||
},
|
||||
)
|
||||
}
|
||||
@@ -1,12 +0,0 @@
|
||||
//go:build !linux
|
||||
|
||||
/* SPDX-License-Identifier: MIT
|
||||
*
|
||||
* Copyright (C) 2017-2023 WireGuard LLC. All Rights Reserved.
|
||||
*/
|
||||
|
||||
package conn
|
||||
|
||||
func errShouldDisableUDPGSO(err error) bool {
|
||||
return false
|
||||
}
|
||||
@@ -1,26 +0,0 @@
|
||||
/* SPDX-License-Identifier: MIT
|
||||
*
|
||||
* Copyright (C) 2017-2023 WireGuard LLC. All Rights Reserved.
|
||||
*/
|
||||
|
||||
package conn
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
func errShouldDisableUDPGSO(err error) bool {
|
||||
var serr *os.SyscallError
|
||||
if errors.As(err, &serr) {
|
||||
// EIO is returned by udp_send_skb() if the device driver does not have
|
||||
// tx checksumming enabled, which is a hard requirement of UDP_SEGMENT.
|
||||
// See:
|
||||
// https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man7/udp.7?id=806eabd74910447f21005160e90957bde4db0183#n228
|
||||
// https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/ipv4/udp.c?h=v6.2&id=c9c3395d5e3dcc6daee66c6908354d47bf98cb0c#n942
|
||||
return serr.Err == unix.EIO
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -1,15 +0,0 @@
|
||||
//go:build !linux
|
||||
// +build !linux
|
||||
|
||||
/* SPDX-License-Identifier: MIT
|
||||
*
|
||||
* Copyright (C) 2017-2023 WireGuard LLC. All Rights Reserved.
|
||||
*/
|
||||
|
||||
package conn
|
||||
|
||||
import "net"
|
||||
|
||||
func supportsUDPOffload(conn *net.UDPConn) (txOffload, rxOffload bool) {
|
||||
return
|
||||
}
|
||||
@@ -1,29 +0,0 @@
|
||||
/* SPDX-License-Identifier: MIT
|
||||
*
|
||||
* Copyright (C) 2017-2023 WireGuard LLC. All Rights Reserved.
|
||||
*/
|
||||
|
||||
package conn
|
||||
|
||||
import (
|
||||
"net"
|
||||
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
func supportsUDPOffload(conn *net.UDPConn) (txOffload, rxOffload bool) {
|
||||
rc, err := conn.SyscallConn()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
err = rc.Control(func(fd uintptr) {
|
||||
_, errSyscall := unix.GetsockoptInt(int(fd), unix.IPPROTO_UDP, unix.UDP_SEGMENT)
|
||||
txOffload = errSyscall == nil
|
||||
opt, errSyscall := unix.GetsockoptInt(int(fd), unix.IPPROTO_UDP, unix.UDP_GRO)
|
||||
rxOffload = errSyscall == nil && opt == 1
|
||||
})
|
||||
if err != nil {
|
||||
return false, false
|
||||
}
|
||||
return txOffload, rxOffload
|
||||
}
|
||||
@@ -1,21 +0,0 @@
|
||||
//go:build !linux
|
||||
|
||||
/* SPDX-License-Identifier: MIT
|
||||
*
|
||||
* Copyright (C) 2017-2023 WireGuard LLC. All Rights Reserved.
|
||||
*/
|
||||
|
||||
package conn
|
||||
|
||||
// getGSOSize parses control for UDP_GRO and if found returns its GSO size data.
|
||||
func getGSOSize(control []byte) (int, error) {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
// setGSOSize sets a UDP_SEGMENT in control based on gsoSize.
|
||||
func setGSOSize(control *[]byte, gsoSize uint16) {
|
||||
}
|
||||
|
||||
// gsoControlSize returns the recommended buffer size for pooling sticky and UDP
|
||||
// offloading control data.
|
||||
const gsoControlSize = 0
|
||||
@@ -1,65 +0,0 @@
|
||||
//go:build linux
|
||||
|
||||
/* SPDX-License-Identifier: MIT
|
||||
*
|
||||
* Copyright (C) 2017-2023 WireGuard LLC. All Rights Reserved.
|
||||
*/
|
||||
|
||||
package conn
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"unsafe"
|
||||
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
const (
|
||||
sizeOfGSOData = 2
|
||||
)
|
||||
|
||||
// getGSOSize parses control for UDP_GRO and if found returns its GSO size data.
|
||||
func getGSOSize(control []byte) (int, error) {
|
||||
var (
|
||||
hdr unix.Cmsghdr
|
||||
data []byte
|
||||
rem = control
|
||||
err error
|
||||
)
|
||||
|
||||
for len(rem) > unix.SizeofCmsghdr {
|
||||
hdr, data, rem, err = unix.ParseOneSocketControlMessage(rem)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("error parsing socket control message: %w", err)
|
||||
}
|
||||
if hdr.Level == unix.SOL_UDP && hdr.Type == unix.UDP_GRO && len(data) >= sizeOfGSOData {
|
||||
var gso uint16
|
||||
copy(unsafe.Slice((*byte)(unsafe.Pointer(&gso)), sizeOfGSOData), data[:sizeOfGSOData])
|
||||
return int(gso), nil
|
||||
}
|
||||
}
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
// setGSOSize sets a UDP_SEGMENT in control based on gsoSize. It leaves existing
|
||||
// data in control untouched.
|
||||
func setGSOSize(control *[]byte, gsoSize uint16) {
|
||||
existingLen := len(*control)
|
||||
avail := cap(*control) - existingLen
|
||||
space := unix.CmsgSpace(sizeOfGSOData)
|
||||
if avail < space {
|
||||
return
|
||||
}
|
||||
*control = (*control)[:cap(*control)]
|
||||
gsoControl := (*control)[existingLen:]
|
||||
hdr := (*unix.Cmsghdr)(unsafe.Pointer(&(gsoControl)[0]))
|
||||
hdr.Level = unix.SOL_UDP
|
||||
hdr.Type = unix.UDP_SEGMENT
|
||||
hdr.SetLen(unix.CmsgLen(sizeOfGSOData))
|
||||
copy((gsoControl)[unix.CmsgLen(0):], unsafe.Slice((*byte)(unsafe.Pointer(&gsoSize)), sizeOfGSOData))
|
||||
*control = (*control)[:existingLen+space]
|
||||
}
|
||||
|
||||
// gsoControlSize returns the recommended buffer size for pooling UDP
|
||||
// offloading control data.
|
||||
var gsoControlSize = unix.CmsgSpace(sizeOfGSOData)
|
||||
@@ -21,9 +21,8 @@ func (e *StdNetEndpoint) SrcToString() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// TODO: macOS, FreeBSD and other BSDs likely do support the sticky sockets
|
||||
// {get,set}srcControl feature set, but use alternatively named flags and need
|
||||
// ports and require testing.
|
||||
// TODO: macOS, FreeBSD and other BSDs likely do support this feature set, but
|
||||
// use alternatively named flags and need ports and require testing.
|
||||
|
||||
// getSrcFromControl parses the control for PKTINFO and if found updates ep with
|
||||
// the source information found.
|
||||
@@ -35,8 +34,8 @@ func getSrcFromControl(control []byte, ep *StdNetEndpoint) {
|
||||
func setSrcControl(control *[]byte, ep *StdNetEndpoint) {
|
||||
}
|
||||
|
||||
// stickyControlSize returns the recommended buffer size for pooling sticky
|
||||
// offloading control data.
|
||||
const stickyControlSize = 0
|
||||
// srcControlSize returns the recommended buffer size for pooling sticky control
|
||||
// data.
|
||||
const srcControlSize = 0
|
||||
|
||||
const StdNetSupportsStickySockets = false
|
||||
@@ -105,8 +105,6 @@ func setSrcControl(control *[]byte, ep *StdNetEndpoint) {
|
||||
*control = append(*control, ep.src...)
|
||||
}
|
||||
|
||||
// stickyControlSize returns the recommended buffer size for pooling sticky
|
||||
// offloading control data.
|
||||
var stickyControlSize = unix.CmsgSpace(unix.SizeofInet6Pktinfo)
|
||||
var srcControlSize = unix.CmsgSpace(unix.SizeofInet6Pktinfo)
|
||||
|
||||
const StdNetSupportsStickySockets = true
|
||||
@@ -60,7 +60,7 @@ func Test_setSrcControl(t *testing.T) {
|
||||
}
|
||||
setSrc(ep, netip.MustParseAddr("127.0.0.1"), 5)
|
||||
|
||||
control := make([]byte, stickyControlSize)
|
||||
control := make([]byte, srcControlSize)
|
||||
|
||||
setSrcControl(&control, ep)
|
||||
|
||||
@@ -89,7 +89,7 @@ func Test_setSrcControl(t *testing.T) {
|
||||
}
|
||||
setSrc(ep, netip.MustParseAddr("::1"), 5)
|
||||
|
||||
control := make([]byte, stickyControlSize)
|
||||
control := make([]byte, srcControlSize)
|
||||
|
||||
setSrcControl(&control, ep)
|
||||
|
||||
@@ -113,7 +113,7 @@ func Test_setSrcControl(t *testing.T) {
|
||||
})
|
||||
|
||||
t.Run("ClearOnNoSrc", func(t *testing.T) {
|
||||
control := make([]byte, stickyControlSize)
|
||||
control := make([]byte, unix.CmsgLen(0))
|
||||
hdr := (*unix.Cmsghdr)(unsafe.Pointer(&control[0]))
|
||||
hdr.Level = 1
|
||||
hdr.Type = 2
|
||||
@@ -129,7 +129,7 @@ func Test_setSrcControl(t *testing.T) {
|
||||
|
||||
func Test_getSrcFromControl(t *testing.T) {
|
||||
t.Run("IPv4", func(t *testing.T) {
|
||||
control := make([]byte, stickyControlSize)
|
||||
control := make([]byte, unix.CmsgSpace(unix.SizeofInet4Pktinfo))
|
||||
hdr := (*unix.Cmsghdr)(unsafe.Pointer(&control[0]))
|
||||
hdr.Level = unix.IPPROTO_IP
|
||||
hdr.Type = unix.IP_PKTINFO
|
||||
@@ -149,7 +149,7 @@ func Test_getSrcFromControl(t *testing.T) {
|
||||
}
|
||||
})
|
||||
t.Run("IPv6", func(t *testing.T) {
|
||||
control := make([]byte, stickyControlSize)
|
||||
control := make([]byte, unix.CmsgSpace(unix.SizeofInet6Pktinfo))
|
||||
hdr := (*unix.Cmsghdr)(unsafe.Pointer(&control[0]))
|
||||
hdr.Level = unix.IPPROTO_IPV6
|
||||
hdr.Type = unix.IPV6_PKTINFO
|
||||
|
||||
+16
-16
@@ -19,13 +19,13 @@ import (
|
||||
// call wg.Done to remove the initial reference.
|
||||
// When the refcount hits 0, the queue's channel is closed.
|
||||
type outboundQueue struct {
|
||||
c chan *QueueOutboundElementsContainer
|
||||
c chan *QueueOutboundElement
|
||||
wg sync.WaitGroup
|
||||
}
|
||||
|
||||
func newOutboundQueue() *outboundQueue {
|
||||
q := &outboundQueue{
|
||||
c: make(chan *QueueOutboundElementsContainer, QueueOutboundSize),
|
||||
c: make(chan *QueueOutboundElement, QueueOutboundSize),
|
||||
}
|
||||
q.wg.Add(1)
|
||||
go func() {
|
||||
@@ -37,13 +37,13 @@ func newOutboundQueue() *outboundQueue {
|
||||
|
||||
// A inboundQueue is similar to an outboundQueue; see those docs.
|
||||
type inboundQueue struct {
|
||||
c chan *QueueInboundElementsContainer
|
||||
c chan *QueueInboundElement
|
||||
wg sync.WaitGroup
|
||||
}
|
||||
|
||||
func newInboundQueue() *inboundQueue {
|
||||
q := &inboundQueue{
|
||||
c: make(chan *QueueInboundElementsContainer, QueueInboundSize),
|
||||
c: make(chan *QueueInboundElement, QueueInboundSize),
|
||||
}
|
||||
q.wg.Add(1)
|
||||
go func() {
|
||||
@@ -72,7 +72,7 @@ func newHandshakeQueue() *handshakeQueue {
|
||||
}
|
||||
|
||||
type autodrainingInboundQueue struct {
|
||||
c chan *QueueInboundElementsContainer
|
||||
c chan *[]*QueueInboundElement
|
||||
}
|
||||
|
||||
// newAutodrainingInboundQueue returns a channel that will be drained when it gets GC'd.
|
||||
@@ -81,7 +81,7 @@ type autodrainingInboundQueue struct {
|
||||
// some other means, such as sending a sentinel nil values.
|
||||
func newAutodrainingInboundQueue(device *Device) *autodrainingInboundQueue {
|
||||
q := &autodrainingInboundQueue{
|
||||
c: make(chan *QueueInboundElementsContainer, QueueInboundSize),
|
||||
c: make(chan *[]*QueueInboundElement, QueueInboundSize),
|
||||
}
|
||||
runtime.SetFinalizer(q, device.flushInboundQueue)
|
||||
return q
|
||||
@@ -90,13 +90,13 @@ func newAutodrainingInboundQueue(device *Device) *autodrainingInboundQueue {
|
||||
func (device *Device) flushInboundQueue(q *autodrainingInboundQueue) {
|
||||
for {
|
||||
select {
|
||||
case elemsContainer := <-q.c:
|
||||
elemsContainer.Lock()
|
||||
for _, elem := range elemsContainer.elems {
|
||||
case elems := <-q.c:
|
||||
for _, elem := range *elems {
|
||||
elem.Lock()
|
||||
device.PutMessageBuffer(elem.buffer)
|
||||
device.PutInboundElement(elem)
|
||||
}
|
||||
device.PutInboundElementsContainer(elemsContainer)
|
||||
device.PutInboundElementsSlice(elems)
|
||||
default:
|
||||
return
|
||||
}
|
||||
@@ -104,7 +104,7 @@ func (device *Device) flushInboundQueue(q *autodrainingInboundQueue) {
|
||||
}
|
||||
|
||||
type autodrainingOutboundQueue struct {
|
||||
c chan *QueueOutboundElementsContainer
|
||||
c chan *[]*QueueOutboundElement
|
||||
}
|
||||
|
||||
// newAutodrainingOutboundQueue returns a channel that will be drained when it gets GC'd.
|
||||
@@ -114,7 +114,7 @@ type autodrainingOutboundQueue struct {
|
||||
// All sends to the channel must be best-effort, because there may be no receivers.
|
||||
func newAutodrainingOutboundQueue(device *Device) *autodrainingOutboundQueue {
|
||||
q := &autodrainingOutboundQueue{
|
||||
c: make(chan *QueueOutboundElementsContainer, QueueOutboundSize),
|
||||
c: make(chan *[]*QueueOutboundElement, QueueOutboundSize),
|
||||
}
|
||||
runtime.SetFinalizer(q, device.flushOutboundQueue)
|
||||
return q
|
||||
@@ -123,13 +123,13 @@ func newAutodrainingOutboundQueue(device *Device) *autodrainingOutboundQueue {
|
||||
func (device *Device) flushOutboundQueue(q *autodrainingOutboundQueue) {
|
||||
for {
|
||||
select {
|
||||
case elemsContainer := <-q.c:
|
||||
elemsContainer.Lock()
|
||||
for _, elem := range elemsContainer.elems {
|
||||
case elems := <-q.c:
|
||||
for _, elem := range *elems {
|
||||
elem.Lock()
|
||||
device.PutMessageBuffer(elem.buffer)
|
||||
device.PutOutboundElement(elem)
|
||||
}
|
||||
device.PutOutboundElementsContainer(elemsContainer)
|
||||
device.PutOutboundElementsSlice(elems)
|
||||
default:
|
||||
return
|
||||
}
|
||||
|
||||
+28
-34
@@ -70,11 +70,11 @@ type Device struct {
|
||||
cookieChecker CookieChecker
|
||||
|
||||
pool struct {
|
||||
inboundElementsContainer *WaitPool
|
||||
outboundElementsContainer *WaitPool
|
||||
messageBuffers *WaitPool
|
||||
inboundElements *WaitPool
|
||||
outboundElements *WaitPool
|
||||
outboundElementsSlice *WaitPool
|
||||
inboundElementsSlice *WaitPool
|
||||
messageBuffers *WaitPool
|
||||
inboundElements *WaitPool
|
||||
outboundElements *WaitPool
|
||||
}
|
||||
|
||||
queue struct {
|
||||
@@ -98,7 +98,6 @@ type Device struct {
|
||||
}
|
||||
|
||||
type aSecCfgType struct {
|
||||
isSet bool
|
||||
junkPacketCount int
|
||||
junkPacketMinSize int
|
||||
junkPacketMaxSize int
|
||||
@@ -546,7 +545,7 @@ func (device *Device) BindUpdate() error {
|
||||
// start receiving routines
|
||||
device.net.stopping.Add(len(recvFns))
|
||||
device.queue.decryption.wg.Add(len(recvFns)) // each RoutineReceiveIncoming goroutine writes to device.queue.decryption
|
||||
device.queue.handshake.wg.Add(len(recvFns)) // each RoutineReceiveIncoming goroutine writes to device.queue.handshake
|
||||
device.queue.handshake.wg.Add(len(recvFns)) // each RoutineReceiveIncoming goroutine writes to device.queue.handshake
|
||||
batchSize := netc.bind.BatchSize()
|
||||
for _, fn := range recvFns {
|
||||
go device.RoutineReceiveIncoming(batchSize, fn)
|
||||
@@ -566,17 +565,12 @@ func (device *Device) isAdvancedSecurityOn() bool {
|
||||
return device.isASecOn.IsSet()
|
||||
}
|
||||
|
||||
func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
|
||||
if !tempASecCfg.isSet {
|
||||
return err
|
||||
}
|
||||
|
||||
func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
isASecOn := false
|
||||
device.aSecMux.Lock()
|
||||
if tempASecCfg.junkPacketCount < 0 {
|
||||
err = ipcErrorf(
|
||||
ipc.IpcErrorInvalid,
|
||||
ipc.IpcErrorInvalid,
|
||||
"JunkPacketCount should be non negative",
|
||||
)
|
||||
}
|
||||
@@ -584,24 +578,24 @@ func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
if tempASecCfg.junkPacketCount != 0 {
|
||||
isASecOn = true
|
||||
}
|
||||
|
||||
|
||||
device.aSecCfg.junkPacketMinSize = tempASecCfg.junkPacketMinSize
|
||||
if tempASecCfg.junkPacketMinSize != 0 {
|
||||
isASecOn = true
|
||||
}
|
||||
|
||||
if device.aSecCfg.junkPacketCount > 0 &&
|
||||
if device.aSecCfg.junkPacketCount > 0 &&
|
||||
tempASecCfg.junkPacketMaxSize == tempASecCfg.junkPacketMinSize {
|
||||
|
||||
|
||||
tempASecCfg.junkPacketMaxSize++ // to make rand gen work
|
||||
}
|
||||
|
||||
if tempASecCfg.junkPacketMaxSize >= MaxSegmentSize {
|
||||
if tempASecCfg.junkPacketMaxSize >= MaxSegmentSize{
|
||||
device.aSecCfg.junkPacketMinSize = 0
|
||||
device.aSecCfg.junkPacketMaxSize = 1
|
||||
if err != nil {
|
||||
err = ipcErrorf(
|
||||
ipc.IpcErrorInvalid,
|
||||
ipc.IpcErrorInvalid,
|
||||
"JunkPacketMaxSize: %d; should be smaller than maxSegmentSize: %d; %w",
|
||||
tempASecCfg.junkPacketMaxSize,
|
||||
MaxSegmentSize,
|
||||
@@ -609,7 +603,7 @@ func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
)
|
||||
} else {
|
||||
err = ipcErrorf(
|
||||
ipc.IpcErrorInvalid,
|
||||
ipc.IpcErrorInvalid,
|
||||
"JunkPacketMaxSize: %d; should be smaller than maxSegmentSize: %d",
|
||||
tempASecCfg.junkPacketMaxSize,
|
||||
MaxSegmentSize,
|
||||
@@ -618,18 +612,18 @@ func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
} else if tempASecCfg.junkPacketMaxSize < tempASecCfg.junkPacketMinSize {
|
||||
if err != nil {
|
||||
err = ipcErrorf(
|
||||
ipc.IpcErrorInvalid,
|
||||
ipc.IpcErrorInvalid,
|
||||
"maxSize: %d; should be greater than minSize: %d; %w",
|
||||
tempASecCfg.junkPacketMaxSize,
|
||||
tempASecCfg.junkPacketMinSize,
|
||||
tempASecCfg.junkPacketMinSize,
|
||||
err,
|
||||
)
|
||||
} else {
|
||||
err = ipcErrorf(
|
||||
ipc.IpcErrorInvalid,
|
||||
ipc.IpcErrorInvalid,
|
||||
"maxSize: %d; should be greater than minSize: %d",
|
||||
tempASecCfg.junkPacketMaxSize,
|
||||
tempASecCfg.junkPacketMinSize,
|
||||
tempASecCfg.junkPacketMinSize,
|
||||
)
|
||||
}
|
||||
} else {
|
||||
@@ -657,10 +651,10 @@ func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
MaxSegmentSize,
|
||||
)
|
||||
}
|
||||
} else {
|
||||
} else {
|
||||
device.aSecCfg.initPacketJunkSize = tempASecCfg.initPacketJunkSize
|
||||
}
|
||||
|
||||
|
||||
if tempASecCfg.initPacketJunkSize != 0 {
|
||||
isASecOn = true
|
||||
}
|
||||
@@ -682,7 +676,7 @@ func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
MaxSegmentSize,
|
||||
)
|
||||
}
|
||||
} else {
|
||||
} else {
|
||||
device.aSecCfg.responsePacketJunkSize = tempASecCfg.responsePacketJunkSize
|
||||
}
|
||||
|
||||
@@ -699,7 +693,7 @@ func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
device.log.Verbosef("UAPI: Using default init type")
|
||||
MessageInitiationType = 1
|
||||
}
|
||||
|
||||
|
||||
if tempASecCfg.responsePacketMagicHeader > 4 {
|
||||
isASecOn = true
|
||||
device.log.Verbosef("UAPI: Updating response_packet_magic_header")
|
||||
@@ -709,7 +703,7 @@ func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
device.log.Verbosef("UAPI: Using default response type")
|
||||
MessageResponseType = 2
|
||||
}
|
||||
|
||||
|
||||
if tempASecCfg.underloadPacketMagicHeader > 4 {
|
||||
isASecOn = true
|
||||
device.log.Verbosef("UAPI: Updating underload_packet_magic_header")
|
||||
@@ -780,14 +774,14 @@ func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
newResponseSize,
|
||||
)
|
||||
}
|
||||
} else {
|
||||
} else {
|
||||
packetSizeToMsgType = map[int]uint32{
|
||||
newInitSize: MessageInitiationType,
|
||||
newResponseSize: MessageResponseType,
|
||||
newInitSize: MessageInitiationType,
|
||||
newResponseSize: MessageResponseType,
|
||||
MessageCookieReplySize: MessageCookieReplyType,
|
||||
MessageTransportSize: MessageTransportType,
|
||||
}
|
||||
|
||||
|
||||
msgTypeToJunkSize = map[uint32]int{
|
||||
MessageInitiationType: device.aSecCfg.initPacketJunkSize,
|
||||
MessageResponseType: device.aSecCfg.responsePacketJunkSize,
|
||||
@@ -798,6 +792,6 @@ func (device *Device) handlePostConfig(tempASecCfg *aSecCfgType) (err error) {
|
||||
|
||||
device.isASecOn.SetTo(isASecOn)
|
||||
device.aSecMux.Unlock()
|
||||
|
||||
|
||||
return err
|
||||
}
|
||||
+4
-4
@@ -45,9 +45,9 @@ type Peer struct {
|
||||
}
|
||||
|
||||
queue struct {
|
||||
staged chan *QueueOutboundElementsContainer // staged packets before a handshake is available
|
||||
outbound *autodrainingOutboundQueue // sequential ordering of udp transmission
|
||||
inbound *autodrainingInboundQueue // sequential ordering of tun writing
|
||||
staged chan *[]*QueueOutboundElement // staged packets before a handshake is available
|
||||
outbound *autodrainingOutboundQueue // sequential ordering of udp transmission
|
||||
inbound *autodrainingInboundQueue // sequential ordering of tun writing
|
||||
}
|
||||
|
||||
cookieGenerator CookieGenerator
|
||||
@@ -81,7 +81,7 @@ func (device *Device) NewPeer(pk NoisePublicKey) (*Peer, error) {
|
||||
peer.device = device
|
||||
peer.queue.outbound = newAutodrainingOutboundQueue(device)
|
||||
peer.queue.inbound = newAutodrainingInboundQueue(device)
|
||||
peer.queue.staged = make(chan *QueueOutboundElementsContainer, QueueStagedSize)
|
||||
peer.queue.staged = make(chan *[]*QueueOutboundElement, QueueStagedSize)
|
||||
|
||||
// map public key
|
||||
_, ok := device.peers.keyMap[pk]
|
||||
|
||||
+20
-24
@@ -46,13 +46,13 @@ func (p *WaitPool) Put(x any) {
|
||||
}
|
||||
|
||||
func (device *Device) PopulatePools() {
|
||||
device.pool.inboundElementsContainer = NewWaitPool(PreallocatedBuffersPerPool, func() any {
|
||||
s := make([]*QueueInboundElement, 0, device.BatchSize())
|
||||
return &QueueInboundElementsContainer{elems: s}
|
||||
})
|
||||
device.pool.outboundElementsContainer = NewWaitPool(PreallocatedBuffersPerPool, func() any {
|
||||
device.pool.outboundElementsSlice = NewWaitPool(PreallocatedBuffersPerPool, func() any {
|
||||
s := make([]*QueueOutboundElement, 0, device.BatchSize())
|
||||
return &QueueOutboundElementsContainer{elems: s}
|
||||
return &s
|
||||
})
|
||||
device.pool.inboundElementsSlice = NewWaitPool(PreallocatedBuffersPerPool, func() any {
|
||||
s := make([]*QueueInboundElement, 0, device.BatchSize())
|
||||
return &s
|
||||
})
|
||||
device.pool.messageBuffers = NewWaitPool(PreallocatedBuffersPerPool, func() any {
|
||||
return new([MaxMessageSize]byte)
|
||||
@@ -65,32 +65,28 @@ func (device *Device) PopulatePools() {
|
||||
})
|
||||
}
|
||||
|
||||
func (device *Device) GetInboundElementsContainer() *QueueInboundElementsContainer {
|
||||
c := device.pool.inboundElementsContainer.Get().(*QueueInboundElementsContainer)
|
||||
c.Mutex = sync.Mutex{}
|
||||
return c
|
||||
func (device *Device) GetOutboundElementsSlice() *[]*QueueOutboundElement {
|
||||
return device.pool.outboundElementsSlice.Get().(*[]*QueueOutboundElement)
|
||||
}
|
||||
|
||||
func (device *Device) PutInboundElementsContainer(c *QueueInboundElementsContainer) {
|
||||
for i := range c.elems {
|
||||
c.elems[i] = nil
|
||||
func (device *Device) PutOutboundElementsSlice(s *[]*QueueOutboundElement) {
|
||||
for i := range *s {
|
||||
(*s)[i] = nil
|
||||
}
|
||||
c.elems = c.elems[:0]
|
||||
device.pool.inboundElementsContainer.Put(c)
|
||||
*s = (*s)[:0]
|
||||
device.pool.outboundElementsSlice.Put(s)
|
||||
}
|
||||
|
||||
func (device *Device) GetOutboundElementsContainer() *QueueOutboundElementsContainer {
|
||||
c := device.pool.outboundElementsContainer.Get().(*QueueOutboundElementsContainer)
|
||||
c.Mutex = sync.Mutex{}
|
||||
return c
|
||||
func (device *Device) GetInboundElementsSlice() *[]*QueueInboundElement {
|
||||
return device.pool.inboundElementsSlice.Get().(*[]*QueueInboundElement)
|
||||
}
|
||||
|
||||
func (device *Device) PutOutboundElementsContainer(c *QueueOutboundElementsContainer) {
|
||||
for i := range c.elems {
|
||||
c.elems[i] = nil
|
||||
func (device *Device) PutInboundElementsSlice(s *[]*QueueInboundElement) {
|
||||
for i := range *s {
|
||||
(*s)[i] = nil
|
||||
}
|
||||
c.elems = c.elems[:0]
|
||||
device.pool.outboundElementsContainer.Put(c)
|
||||
*s = (*s)[:0]
|
||||
device.pool.inboundElementsSlice.Put(s)
|
||||
}
|
||||
|
||||
func (device *Device) GetMessageBuffer() *[MaxMessageSize]byte {
|
||||
|
||||
@@ -14,6 +14,6 @@ const (
|
||||
QueueOutboundSize = 1024
|
||||
QueueInboundSize = 1024
|
||||
QueueHandshakeSize = 1024
|
||||
MaxSegmentSize = (1 << 16) - 1 // largest possible UDP datagram
|
||||
MaxSegmentSize = 2200
|
||||
PreallocatedBuffersPerPool = 4096
|
||||
)
|
||||
+39
-42
@@ -27,6 +27,7 @@ type QueueHandshakeElement struct {
|
||||
}
|
||||
|
||||
type QueueInboundElement struct {
|
||||
sync.Mutex
|
||||
buffer *[MaxMessageSize]byte
|
||||
packet []byte
|
||||
counter uint64
|
||||
@@ -34,11 +35,6 @@ type QueueInboundElement struct {
|
||||
endpoint conn.Endpoint
|
||||
}
|
||||
|
||||
type QueueInboundElementsContainer struct {
|
||||
sync.Mutex
|
||||
elems []*QueueInboundElement
|
||||
}
|
||||
|
||||
// clearPointers clears elem fields that contain pointers.
|
||||
// This makes the garbage collector's life easier and
|
||||
// avoids accidentally keeping other objects around unnecessarily.
|
||||
@@ -94,7 +90,7 @@ func (device *Device) RoutineReceiveIncoming(
|
||||
count int
|
||||
endpoints = make([]conn.Endpoint, maxBatchSize)
|
||||
deathSpiral int
|
||||
elemsByPeer = make(map[*Peer]*QueueInboundElementsContainer, maxBatchSize)
|
||||
elemsByPeer = make(map[*Peer]*[]*QueueInboundElement, maxBatchSize)
|
||||
)
|
||||
|
||||
for i := range bufsArrs {
|
||||
@@ -143,7 +139,7 @@ func (device *Device) RoutineReceiveIncoming(
|
||||
if device.isAdvancedSecurityOn() {
|
||||
if assumedMsgType, ok := packetSizeToMsgType[size]; ok {
|
||||
junkSize := msgTypeToJunkSize[assumedMsgType]
|
||||
// transport size can align with other header types;
|
||||
// transport size can align with other header types;
|
||||
// making sure we have the right msgType
|
||||
msgType = binary.LittleEndian.Uint32(packet[junkSize:junkSize+4])
|
||||
if msgType == assumedMsgType {
|
||||
@@ -157,7 +153,7 @@ func (device *Device) RoutineReceiveIncoming(
|
||||
if msgType != MessageTransportType {
|
||||
device.log.Verbosef("ASec: Received message with unknown type")
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
msgType = binary.LittleEndian.Uint32(packet[:4])
|
||||
@@ -199,14 +195,15 @@ func (device *Device) RoutineReceiveIncoming(
|
||||
elem.keypair = keypair
|
||||
elem.endpoint = endpoints[i]
|
||||
elem.counter = 0
|
||||
elem.Mutex = sync.Mutex{}
|
||||
elem.Lock()
|
||||
|
||||
elemsForPeer, ok := elemsByPeer[peer]
|
||||
if !ok {
|
||||
elemsForPeer = device.GetInboundElementsContainer()
|
||||
elemsForPeer.Lock()
|
||||
elemsForPeer = device.GetInboundElementsSlice()
|
||||
elemsByPeer[peer] = elemsForPeer
|
||||
}
|
||||
elemsForPeer.elems = append(elemsForPeer.elems, elem)
|
||||
*elemsForPeer = append(*elemsForPeer, elem)
|
||||
bufsArrs[i] = device.GetMessageBuffer()
|
||||
bufs[i] = bufsArrs[i][:]
|
||||
continue
|
||||
@@ -246,16 +243,18 @@ func (device *Device) RoutineReceiveIncoming(
|
||||
}
|
||||
}
|
||||
device.aSecMux.RUnlock()
|
||||
for peer, elemsContainer := range elemsByPeer {
|
||||
for peer, elems := range elemsByPeer {
|
||||
if peer.isRunning.Load() {
|
||||
peer.queue.inbound.c <- elemsContainer
|
||||
device.queue.decryption.c <- elemsContainer
|
||||
peer.queue.inbound.c <- elems
|
||||
for _, elem := range *elems {
|
||||
device.queue.decryption.c <- elem
|
||||
}
|
||||
} else {
|
||||
for _, elem := range elemsContainer.elems {
|
||||
for _, elem := range *elems {
|
||||
device.PutMessageBuffer(elem.buffer)
|
||||
device.PutInboundElement(elem)
|
||||
}
|
||||
device.PutInboundElementsContainer(elemsContainer)
|
||||
device.PutInboundElementsSlice(elems)
|
||||
}
|
||||
delete(elemsByPeer, peer)
|
||||
}
|
||||
@@ -268,28 +267,26 @@ func (device *Device) RoutineDecryption(id int) {
|
||||
defer device.log.Verbosef("Routine: decryption worker %d - stopped", id)
|
||||
device.log.Verbosef("Routine: decryption worker %d - started", id)
|
||||
|
||||
for elemsContainer := range device.queue.decryption.c {
|
||||
for _, elem := range elemsContainer.elems {
|
||||
// split message into fields
|
||||
counter := elem.packet[MessageTransportOffsetCounter:MessageTransportOffsetContent]
|
||||
content := elem.packet[MessageTransportOffsetContent:]
|
||||
for elem := range device.queue.decryption.c {
|
||||
// split message into fields
|
||||
counter := elem.packet[MessageTransportOffsetCounter:MessageTransportOffsetContent]
|
||||
content := elem.packet[MessageTransportOffsetContent:]
|
||||
|
||||
// decrypt and release to consumer
|
||||
var err error
|
||||
elem.counter = binary.LittleEndian.Uint64(counter)
|
||||
// copy counter to nonce
|
||||
binary.LittleEndian.PutUint64(nonce[0x4:0xc], elem.counter)
|
||||
elem.packet, err = elem.keypair.receive.Open(
|
||||
content[:0],
|
||||
nonce[:],
|
||||
content,
|
||||
nil,
|
||||
)
|
||||
if err != nil {
|
||||
elem.packet = nil
|
||||
}
|
||||
// decrypt and release to consumer
|
||||
var err error
|
||||
elem.counter = binary.LittleEndian.Uint64(counter)
|
||||
// copy counter to nonce
|
||||
binary.LittleEndian.PutUint64(nonce[0x4:0xc], elem.counter)
|
||||
elem.packet, err = elem.keypair.receive.Open(
|
||||
content[:0],
|
||||
nonce[:],
|
||||
content,
|
||||
nil,
|
||||
)
|
||||
if err != nil {
|
||||
elem.packet = nil
|
||||
}
|
||||
elemsContainer.Unlock()
|
||||
elem.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -471,12 +468,12 @@ func (peer *Peer) RoutineSequentialReceiver(maxBatchSize int) {
|
||||
|
||||
bufs := make([][]byte, 0, maxBatchSize)
|
||||
|
||||
for elemsContainer := range peer.queue.inbound.c {
|
||||
if elemsContainer == nil {
|
||||
for elems := range peer.queue.inbound.c {
|
||||
if elems == nil {
|
||||
return
|
||||
}
|
||||
elemsContainer.Lock()
|
||||
for _, elem := range elemsContainer.elems {
|
||||
for _, elem := range *elems {
|
||||
elem.Lock()
|
||||
if elem.packet == nil {
|
||||
// decryption failed
|
||||
continue
|
||||
@@ -555,11 +552,11 @@ func (peer *Peer) RoutineSequentialReceiver(maxBatchSize int) {
|
||||
device.log.Errorf("Failed to write packets to TUN device: %v", err)
|
||||
}
|
||||
}
|
||||
for _, elem := range elemsContainer.elems {
|
||||
for _, elem := range *elems {
|
||||
device.PutMessageBuffer(elem.buffer)
|
||||
device.PutInboundElement(elem)
|
||||
}
|
||||
bufs = bufs[:0]
|
||||
device.PutInboundElementsContainer(elemsContainer)
|
||||
device.PutInboundElementsSlice(elems)
|
||||
}
|
||||
}
|
||||
+70
-92
@@ -15,7 +15,6 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/amnezia-vpn/amnezia-wg/conn"
|
||||
"github.com/amnezia-vpn/amnezia-wg/tun"
|
||||
"golang.org/x/crypto/chacha20poly1305"
|
||||
"golang.org/x/net/ipv4"
|
||||
@@ -47,6 +46,7 @@ import (
|
||||
*/
|
||||
|
||||
type QueueOutboundElement struct {
|
||||
sync.Mutex
|
||||
buffer *[MaxMessageSize]byte // slice holding the packet data
|
||||
packet []byte // slice of "buffer" (always!)
|
||||
nonce uint64 // nonce for encryption
|
||||
@@ -54,14 +54,10 @@ type QueueOutboundElement struct {
|
||||
peer *Peer // related peer
|
||||
}
|
||||
|
||||
type QueueOutboundElementsContainer struct {
|
||||
sync.Mutex
|
||||
elems []*QueueOutboundElement
|
||||
}
|
||||
|
||||
func (device *Device) NewOutboundElement() *QueueOutboundElement {
|
||||
elem := device.GetOutboundElement()
|
||||
elem.buffer = device.GetMessageBuffer()
|
||||
elem.Mutex = sync.Mutex{}
|
||||
elem.nonce = 0
|
||||
// keypair and peer were cleared (if necessary) by clearPointers.
|
||||
return elem
|
||||
@@ -83,15 +79,15 @@ func (elem *QueueOutboundElement) clearPointers() {
|
||||
func (peer *Peer) SendKeepalive() {
|
||||
if len(peer.queue.staged) == 0 && peer.isRunning.Load() {
|
||||
elem := peer.device.NewOutboundElement()
|
||||
elemsContainer := peer.device.GetOutboundElementsContainer()
|
||||
elemsContainer.elems = append(elemsContainer.elems, elem)
|
||||
elems := peer.device.GetOutboundElementsSlice()
|
||||
*elems = append(*elems, elem)
|
||||
select {
|
||||
case peer.queue.staged <- elemsContainer:
|
||||
case peer.queue.staged <- elems:
|
||||
peer.device.log.Verbosef("%v - Sending keepalive packet", peer)
|
||||
default:
|
||||
peer.device.PutMessageBuffer(elem.buffer)
|
||||
peer.device.PutOutboundElement(elem)
|
||||
peer.device.PutOutboundElementsContainer(elemsContainer)
|
||||
peer.device.PutOutboundElementsSlice(elems)
|
||||
}
|
||||
}
|
||||
peer.SendStagedPackets()
|
||||
@@ -130,34 +126,25 @@ func (peer *Peer) SendHandshakeInitiation(isRetry bool) error {
|
||||
if peer.device.isAdvancedSecurityOn() {
|
||||
peer.device.aSecMux.RLock()
|
||||
junks, err := peer.createJunkPackets()
|
||||
peer.device.aSecMux.RUnlock()
|
||||
|
||||
if err != nil {
|
||||
peer.device.aSecMux.RUnlock()
|
||||
peer.device.log.Errorf("%v - %v", peer, err)
|
||||
return err
|
||||
}
|
||||
|
||||
err = peer.SendBuffers(junks)
|
||||
if err != nil {
|
||||
peer.device.log.Errorf("%v - Failed to send junk packets: %v", peer, err)
|
||||
return err
|
||||
}
|
||||
|
||||
peer.device.aSecMux.RLock()
|
||||
sendBuffer = append(sendBuffer, junks...)
|
||||
if peer.device.aSecCfg.initPacketJunkSize != 0 {
|
||||
buf := make([]byte, 0, peer.device.aSecCfg.initPacketJunkSize)
|
||||
writer := bytes.NewBuffer(buf[:0])
|
||||
err = appendJunk(writer, peer.device.aSecCfg.initPacketJunkSize)
|
||||
if err != nil {
|
||||
peer.device.log.Errorf("%v - %v", peer, err)
|
||||
peer.device.aSecMux.RUnlock()
|
||||
peer.device.log.Errorf("%v - %v", peer, err)
|
||||
return err
|
||||
}
|
||||
junkedHeader = writer.Bytes()
|
||||
}
|
||||
peer.device.aSecMux.RUnlock()
|
||||
}
|
||||
|
||||
var buf [MessageInitiationSize]byte
|
||||
writer := bytes.NewBuffer(buf[:0])
|
||||
binary.Write(writer, binary.LittleEndian, msg)
|
||||
@@ -167,9 +154,9 @@ func (peer *Peer) SendHandshakeInitiation(isRetry bool) error {
|
||||
|
||||
peer.timersAnyAuthenticatedPacketTraversal()
|
||||
peer.timersAnyAuthenticatedPacketSent()
|
||||
|
||||
|
||||
sendBuffer = append(sendBuffer, junkedHeader)
|
||||
|
||||
|
||||
err = peer.SendBuffers(sendBuffer)
|
||||
if err != nil {
|
||||
peer.device.log.Errorf("%v - Failed to send handshake initiation: %v", peer, err)
|
||||
@@ -204,7 +191,7 @@ func (peer *Peer) SendHandshakeResponse() error {
|
||||
return err
|
||||
}
|
||||
junkedHeader = writer.Bytes()
|
||||
}
|
||||
}
|
||||
peer.device.aSecMux.RUnlock()
|
||||
}
|
||||
var buf [MessageResponseSize]byte
|
||||
@@ -282,7 +269,7 @@ func (device *Device) RoutineReadFromTUN() {
|
||||
readErr error
|
||||
elems = make([]*QueueOutboundElement, batchSize)
|
||||
bufs = make([][]byte, batchSize)
|
||||
elemsByPeer = make(map[*Peer]*QueueOutboundElementsContainer, batchSize)
|
||||
elemsByPeer = make(map[*Peer]*[]*QueueOutboundElement, batchSize)
|
||||
count = 0
|
||||
sizes = make([]int, batchSize)
|
||||
offset = MessageTransportHeaderSize
|
||||
@@ -339,10 +326,10 @@ func (device *Device) RoutineReadFromTUN() {
|
||||
}
|
||||
elemsForPeer, ok := elemsByPeer[peer]
|
||||
if !ok {
|
||||
elemsForPeer = device.GetOutboundElementsContainer()
|
||||
elemsForPeer = device.GetOutboundElementsSlice()
|
||||
elemsByPeer[peer] = elemsForPeer
|
||||
}
|
||||
elemsForPeer.elems = append(elemsForPeer.elems, elem)
|
||||
*elemsForPeer = append(*elemsForPeer, elem)
|
||||
elems[i] = device.NewOutboundElement()
|
||||
bufs[i] = elems[i].buffer[:]
|
||||
}
|
||||
@@ -352,11 +339,11 @@ func (device *Device) RoutineReadFromTUN() {
|
||||
peer.StagePackets(elemsForPeer)
|
||||
peer.SendStagedPackets()
|
||||
} else {
|
||||
for _, elem := range elemsForPeer.elems {
|
||||
for _, elem := range *elemsForPeer {
|
||||
device.PutMessageBuffer(elem.buffer)
|
||||
device.PutOutboundElement(elem)
|
||||
}
|
||||
device.PutOutboundElementsContainer(elemsForPeer)
|
||||
device.PutOutboundElementsSlice(elemsForPeer)
|
||||
}
|
||||
delete(elemsByPeer, peer)
|
||||
}
|
||||
@@ -380,7 +367,7 @@ func (device *Device) RoutineReadFromTUN() {
|
||||
}
|
||||
}
|
||||
|
||||
func (peer *Peer) StagePackets(elems *QueueOutboundElementsContainer) {
|
||||
func (peer *Peer) StagePackets(elems *[]*QueueOutboundElement) {
|
||||
for {
|
||||
select {
|
||||
case peer.queue.staged <- elems:
|
||||
@@ -389,11 +376,11 @@ func (peer *Peer) StagePackets(elems *QueueOutboundElementsContainer) {
|
||||
}
|
||||
select {
|
||||
case tooOld := <-peer.queue.staged:
|
||||
for _, elem := range tooOld.elems {
|
||||
for _, elem := range *tooOld {
|
||||
peer.device.PutMessageBuffer(elem.buffer)
|
||||
peer.device.PutOutboundElement(elem)
|
||||
}
|
||||
peer.device.PutOutboundElementsContainer(tooOld)
|
||||
peer.device.PutOutboundElementsSlice(tooOld)
|
||||
default:
|
||||
}
|
||||
}
|
||||
@@ -412,52 +399,54 @@ top:
|
||||
}
|
||||
|
||||
for {
|
||||
var elemsContainerOOO *QueueOutboundElementsContainer
|
||||
var elemsOOO *[]*QueueOutboundElement
|
||||
select {
|
||||
case elemsContainer := <-peer.queue.staged:
|
||||
case elems := <-peer.queue.staged:
|
||||
i := 0
|
||||
for _, elem := range elemsContainer.elems {
|
||||
for _, elem := range *elems {
|
||||
elem.peer = peer
|
||||
elem.nonce = keypair.sendNonce.Add(1) - 1
|
||||
if elem.nonce >= RejectAfterMessages {
|
||||
keypair.sendNonce.Store(RejectAfterMessages)
|
||||
if elemsContainerOOO == nil {
|
||||
elemsContainerOOO = peer.device.GetOutboundElementsContainer()
|
||||
if elemsOOO == nil {
|
||||
elemsOOO = peer.device.GetOutboundElementsSlice()
|
||||
}
|
||||
elemsContainerOOO.elems = append(elemsContainerOOO.elems, elem)
|
||||
*elemsOOO = append(*elemsOOO, elem)
|
||||
continue
|
||||
} else {
|
||||
elemsContainer.elems[i] = elem
|
||||
(*elems)[i] = elem
|
||||
i++
|
||||
}
|
||||
|
||||
elem.keypair = keypair
|
||||
elem.Lock()
|
||||
}
|
||||
elemsContainer.Lock()
|
||||
elemsContainer.elems = elemsContainer.elems[:i]
|
||||
*elems = (*elems)[:i]
|
||||
|
||||
if elemsContainerOOO != nil {
|
||||
peer.StagePackets(elemsContainerOOO) // XXX: Out of order, but we can't front-load go chans
|
||||
if elemsOOO != nil {
|
||||
peer.StagePackets(elemsOOO) // XXX: Out of order, but we can't front-load go chans
|
||||
}
|
||||
|
||||
if len(elemsContainer.elems) == 0 {
|
||||
peer.device.PutOutboundElementsContainer(elemsContainer)
|
||||
if len(*elems) == 0 {
|
||||
peer.device.PutOutboundElementsSlice(elems)
|
||||
goto top
|
||||
}
|
||||
|
||||
// add to parallel and sequential queue
|
||||
if peer.isRunning.Load() {
|
||||
peer.queue.outbound.c <- elemsContainer
|
||||
peer.device.queue.encryption.c <- elemsContainer
|
||||
peer.queue.outbound.c <- elems
|
||||
for _, elem := range *elems {
|
||||
peer.device.queue.encryption.c <- elem
|
||||
}
|
||||
} else {
|
||||
for _, elem := range elemsContainer.elems {
|
||||
for _, elem := range *elems {
|
||||
peer.device.PutMessageBuffer(elem.buffer)
|
||||
peer.device.PutOutboundElement(elem)
|
||||
}
|
||||
peer.device.PutOutboundElementsContainer(elemsContainer)
|
||||
peer.device.PutOutboundElementsSlice(elems)
|
||||
}
|
||||
|
||||
if elemsContainerOOO != nil {
|
||||
if elemsOOO != nil {
|
||||
goto top
|
||||
}
|
||||
default:
|
||||
@@ -494,12 +483,12 @@ func (peer *Peer) createJunkPackets() ([][]byte, error) {
|
||||
func (peer *Peer) FlushStagedPackets() {
|
||||
for {
|
||||
select {
|
||||
case elemsContainer := <-peer.queue.staged:
|
||||
for _, elem := range elemsContainer.elems {
|
||||
case elems := <-peer.queue.staged:
|
||||
for _, elem := range *elems {
|
||||
peer.device.PutMessageBuffer(elem.buffer)
|
||||
peer.device.PutOutboundElement(elem)
|
||||
}
|
||||
peer.device.PutOutboundElementsContainer(elemsContainer)
|
||||
peer.device.PutOutboundElementsSlice(elems)
|
||||
default:
|
||||
return
|
||||
}
|
||||
@@ -533,34 +522,30 @@ func (device *Device) RoutineEncryption(id int) {
|
||||
defer device.log.Verbosef("Routine: encryption worker %d - stopped", id)
|
||||
device.log.Verbosef("Routine: encryption worker %d - started", id)
|
||||
|
||||
for elemsContainer := range device.queue.encryption.c {
|
||||
for _, elem := range elemsContainer.elems {
|
||||
// populate header fields
|
||||
header := elem.buffer[:MessageTransportHeaderSize]
|
||||
for elem := range device.queue.encryption.c {
|
||||
// populate header fields
|
||||
header := elem.buffer[:MessageTransportHeaderSize]
|
||||
|
||||
fieldType := header[0:4]
|
||||
fieldReceiver := header[4:8]
|
||||
fieldNonce := header[8:16]
|
||||
fieldType := header[0:4]
|
||||
fieldReceiver := header[4:8]
|
||||
fieldNonce := header[8:16]
|
||||
|
||||
binary.LittleEndian.PutUint32(fieldType, MessageTransportType)
|
||||
binary.LittleEndian.PutUint32(fieldReceiver, elem.keypair.remoteIndex)
|
||||
binary.LittleEndian.PutUint64(fieldNonce, elem.nonce)
|
||||
binary.LittleEndian.PutUint32(fieldType, MessageTransportType)
|
||||
binary.LittleEndian.PutUint32(fieldReceiver, elem.keypair.remoteIndex)
|
||||
binary.LittleEndian.PutUint64(fieldNonce, elem.nonce)
|
||||
|
||||
// pad content to multiple of 16
|
||||
paddingSize := calculatePaddingSize(len(elem.packet), int(device.tun.mtu.Load()))
|
||||
elem.packet = append(elem.packet, paddingZeros[:paddingSize]...)
|
||||
// pad content to multiple of 16
|
||||
paddingSize := calculatePaddingSize(
|
||||
len(elem.packet),
|
||||
int(device.tun.mtu.Load()),
|
||||
)
|
||||
elem.packet = append(elem.packet, paddingZeros[:paddingSize]...)
|
||||
|
||||
// encrypt content and release to consumer
|
||||
// encrypt content and release to consumer
|
||||
|
||||
binary.LittleEndian.PutUint64(nonce[4:], elem.nonce)
|
||||
elem.packet = elem.keypair.send.Seal(
|
||||
header,
|
||||
nonce[:],
|
||||
elem.packet,
|
||||
nil,
|
||||
)
|
||||
}
|
||||
elemsContainer.Unlock()
|
||||
binary.LittleEndian.PutUint64(nonce[4:], elem.nonce)
|
||||
elem.packet = elem.keypair.send.Seal(header, nonce[:], elem.packet, nil)
|
||||
elem.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -574,9 +559,9 @@ func (peer *Peer) RoutineSequentialSender(maxBatchSize int) {
|
||||
|
||||
bufs := make([][]byte, 0, maxBatchSize)
|
||||
|
||||
for elemsContainer := range peer.queue.outbound.c {
|
||||
for elems := range peer.queue.outbound.c {
|
||||
bufs = bufs[:0]
|
||||
if elemsContainer == nil {
|
||||
if elems == nil {
|
||||
return
|
||||
}
|
||||
if !peer.isRunning.Load() {
|
||||
@@ -586,16 +571,16 @@ func (peer *Peer) RoutineSequentialSender(maxBatchSize int) {
|
||||
// The timers and SendBuffers code are resilient to a few stragglers.
|
||||
// TODO: rework peer shutdown order to ensure
|
||||
// that we never accidentally keep timers alive longer than necessary.
|
||||
elemsContainer.Lock()
|
||||
for _, elem := range elemsContainer.elems {
|
||||
for _, elem := range *elems {
|
||||
elem.Lock()
|
||||
device.PutMessageBuffer(elem.buffer)
|
||||
device.PutOutboundElement(elem)
|
||||
}
|
||||
continue
|
||||
}
|
||||
dataSent := false
|
||||
elemsContainer.Lock()
|
||||
for _, elem := range elemsContainer.elems {
|
||||
for _, elem := range *elems {
|
||||
elem.Lock()
|
||||
if len(elem.packet) != MessageKeepaliveSize {
|
||||
dataSent = true
|
||||
}
|
||||
@@ -609,18 +594,11 @@ func (peer *Peer) RoutineSequentialSender(maxBatchSize int) {
|
||||
if dataSent {
|
||||
peer.timersDataSent()
|
||||
}
|
||||
for _, elem := range elemsContainer.elems {
|
||||
for _, elem := range *elems {
|
||||
device.PutMessageBuffer(elem.buffer)
|
||||
device.PutOutboundElement(elem)
|
||||
}
|
||||
device.PutOutboundElementsContainer(elemsContainer)
|
||||
if err != nil {
|
||||
var errGSO conn.ErrUDPGSODisabled
|
||||
if errors.As(err, &errGSO) {
|
||||
device.log.Verbosef(err.Error())
|
||||
err = errGSO.RetryErr
|
||||
}
|
||||
}
|
||||
device.PutOutboundElementsSlice(elems)
|
||||
if err != nil {
|
||||
device.log.Errorf("%v - Failed to send data packets: %v", peer, err)
|
||||
continue
|
||||
|
||||
+2
-12
@@ -295,7 +295,6 @@ func (device *Device) handleDeviceLine(key, value string, tempASecCfg *aSecCfgTy
|
||||
}
|
||||
device.log.Verbosef("UAPI: Updating junk_packet_count")
|
||||
tempASecCfg.junkPacketCount = junkPacketCount
|
||||
tempASecCfg.isSet = true
|
||||
|
||||
case "jmin":
|
||||
junkPacketMinSize, err := strconv.Atoi(value)
|
||||
@@ -304,7 +303,6 @@ func (device *Device) handleDeviceLine(key, value string, tempASecCfg *aSecCfgTy
|
||||
}
|
||||
device.log.Verbosef("UAPI: Updating junk_packet_min_size")
|
||||
tempASecCfg.junkPacketMinSize = junkPacketMinSize
|
||||
tempASecCfg.isSet = true
|
||||
|
||||
case "jmax":
|
||||
junkPacketMaxSize, err := strconv.Atoi(value)
|
||||
@@ -313,7 +311,6 @@ func (device *Device) handleDeviceLine(key, value string, tempASecCfg *aSecCfgTy
|
||||
}
|
||||
device.log.Verbosef("UAPI: Updating junk_packet_max_size")
|
||||
tempASecCfg.junkPacketMaxSize = junkPacketMaxSize
|
||||
tempASecCfg.isSet = true
|
||||
|
||||
case "s1":
|
||||
initPacketJunkSize, err := strconv.Atoi(value)
|
||||
@@ -322,7 +319,6 @@ func (device *Device) handleDeviceLine(key, value string, tempASecCfg *aSecCfgTy
|
||||
}
|
||||
device.log.Verbosef("UAPI: Updating init_packet_junk_size")
|
||||
tempASecCfg.initPacketJunkSize = initPacketJunkSize
|
||||
tempASecCfg.isSet = true
|
||||
|
||||
case "s2":
|
||||
responsePacketJunkSize, err := strconv.Atoi(value)
|
||||
@@ -331,7 +327,6 @@ func (device *Device) handleDeviceLine(key, value string, tempASecCfg *aSecCfgTy
|
||||
}
|
||||
device.log.Verbosef("UAPI: Updating response_packet_junk_size")
|
||||
tempASecCfg.responsePacketJunkSize = responsePacketJunkSize
|
||||
tempASecCfg.isSet = true
|
||||
|
||||
case "h1":
|
||||
initPacketMagicHeader, err := strconv.ParseUint(value, 10, 32)
|
||||
@@ -339,7 +334,6 @@ func (device *Device) handleDeviceLine(key, value string, tempASecCfg *aSecCfgTy
|
||||
return ipcErrorf(ipc.IpcErrorInvalid, "faield to parse init_packet_magic_header %w", err)
|
||||
}
|
||||
tempASecCfg.initPacketMagicHeader = uint32(initPacketMagicHeader)
|
||||
tempASecCfg.isSet = true
|
||||
|
||||
case "h2":
|
||||
responsePacketMagicHeader, err := strconv.ParseUint(value, 10, 32)
|
||||
@@ -347,7 +341,6 @@ func (device *Device) handleDeviceLine(key, value string, tempASecCfg *aSecCfgTy
|
||||
return ipcErrorf(ipc.IpcErrorInvalid, "faield to parse response_packet_magic_header %w", err)
|
||||
}
|
||||
tempASecCfg.responsePacketMagicHeader = uint32(responsePacketMagicHeader)
|
||||
tempASecCfg.isSet = true
|
||||
|
||||
case "h3":
|
||||
underloadPacketMagicHeader, err := strconv.ParseUint(value, 10, 32)
|
||||
@@ -355,7 +348,6 @@ func (device *Device) handleDeviceLine(key, value string, tempASecCfg *aSecCfgTy
|
||||
return ipcErrorf(ipc.IpcErrorInvalid, "faield to parse underload_packet_magic_header %w", err)
|
||||
}
|
||||
tempASecCfg.underloadPacketMagicHeader = uint32(underloadPacketMagicHeader)
|
||||
tempASecCfg.isSet = true
|
||||
|
||||
case "h4":
|
||||
transportPacketMagicHeader, err := strconv.ParseUint(value, 10, 32)
|
||||
@@ -363,10 +355,8 @@ func (device *Device) handleDeviceLine(key, value string, tempASecCfg *aSecCfgTy
|
||||
return ipcErrorf(ipc.IpcErrorInvalid, "faield to parse transport_packet_magic_header %w", err)
|
||||
}
|
||||
tempASecCfg.transportPacketMagicHeader = uint32(transportPacketMagicHeader)
|
||||
tempASecCfg.isSet = true
|
||||
|
||||
default:
|
||||
return ipcErrorf(ipc.IpcErrorInvalid, "invalid UAPI device key: %v", key)
|
||||
return ipcErrorf(ipc.IpcErrorInvalid, "invalid UAPI device key: %v",key)
|
||||
}
|
||||
|
||||
return nil
|
||||
@@ -473,7 +463,7 @@ func (device *Device) handlePeerLine(
|
||||
device.log.Verbosef("%v - UAPI: Updating endpoint", peer.Peer)
|
||||
endpoint, err := device.net.bind.ParseEndpoint(value)
|
||||
if err != nil {
|
||||
return ipcErrorf(ipc.IpcErrorInvalid, "failed to set endpoint %v: %w", value, err)
|
||||
return ipcErrorf(ipc.IpcErrorInvalid, "failed to set endpoint %v: %w", value, err)
|
||||
}
|
||||
peer.Lock()
|
||||
defer peer.Unlock()
|
||||
|
||||
@@ -4,14 +4,14 @@ go 1.20
|
||||
|
||||
require (
|
||||
github.com/tevino/abool/v2 v2.1.0
|
||||
golang.org/x/crypto v0.13.0
|
||||
golang.org/x/net v0.15.0
|
||||
golang.org/x/sys v0.12.0
|
||||
golang.org/x/crypto v0.6.0
|
||||
golang.org/x/net v0.7.0
|
||||
golang.org/x/sys v0.5.1-0.20230222185716-a3b23cc77e89
|
||||
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2
|
||||
gvisor.dev/gvisor v0.0.0-20230927004350-cbd86285d259
|
||||
gvisor.dev/gvisor v0.0.0-20221203005347-703fd9b7fbc0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/google/btree v1.0.1 // indirect
|
||||
golang.org/x/time v0.0.0-20220210224613-90d013bbcef8 // indirect
|
||||
golang.org/x/time v0.0.0-20191024005414-555d28b269f0 // indirect
|
||||
)
|
||||
@@ -2,15 +2,15 @@ github.com/google/btree v1.0.1 h1:gK4Kx5IaGY9CD5sPJ36FHiBJ6ZXl0kilRiiCj+jdYp4=
|
||||
github.com/google/btree v1.0.1/go.mod h1:xXMiIv4Fb/0kKde4SpL7qlzvu5cMJDRkFDxJfI9uaxA=
|
||||
github.com/tevino/abool/v2 v2.1.0 h1:7w+Vf9f/5gmKT4m4qkayb33/92M+Um45F2BkHOR+L/c=
|
||||
github.com/tevino/abool/v2 v2.1.0/go.mod h1:+Lmlqk6bHDWHqN1cbxqhwEAwMPXgc8I1SDEamtseuXY=
|
||||
golang.org/x/crypto v0.13.0 h1:mvySKfSWJ+UKUii46M40LOvyWfN0s2U+46/jDd0e6Ck=
|
||||
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
|
||||
golang.org/x/net v0.15.0 h1:ugBLEUaxABaB5AJqW9enI0ACdci2RUd4eP51NTBvuJ8=
|
||||
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
|
||||
golang.org/x/sys v0.12.0 h1:CM0HF96J0hcLAwsHPJZjfdNzs0gftsLfgKt57wWHJ0o=
|
||||
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/time v0.0.0-20220210224613-90d013bbcef8 h1:vVKdlvoWBphwdxWKrFZEuM0kGgGLxUOYcY4U/2Vjg44=
|
||||
golang.org/x/time v0.0.0-20220210224613-90d013bbcef8/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc=
|
||||
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
|
||||
golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g=
|
||||
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
|
||||
golang.org/x/sys v0.5.1-0.20230222185716-a3b23cc77e89 h1:260HNjMTPDya+jq5AM1zZLgG9pv9GASPAGiEEJUbRg4=
|
||||
golang.org/x/sys v0.5.1-0.20230222185716-a3b23cc77e89/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/time v0.0.0-20191024005414-555d28b269f0 h1:/5xXl8Y5W96D+TtHSlonuFqGHIWVuyCkGJLwGh9JJFs=
|
||||
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 h1:B82qJJgjvYKsXS9jeunTOisW56dUokqW/FOteYJJ/yg=
|
||||
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2/go.mod h1:deeaetjYA+DHMHg+sMSMI58GrEteJUUzzw7en6TJQcI=
|
||||
gvisor.dev/gvisor v0.0.0-20230927004350-cbd86285d259 h1:TbRPT0HtzFP3Cno1zZo7yPzEEnfu8EjLfl6IU9VfqkQ=
|
||||
gvisor.dev/gvisor v0.0.0-20230927004350-cbd86285d259/go.mod h1:AVgIgHMwK63XvmAzWG9vLQ41YnVHN0du0tEC46fI7yY=
|
||||
gvisor.dev/gvisor v0.0.0-20221203005347-703fd9b7fbc0 h1:Wobr37noukisGxpKo5jAsLREcpj61RxrWYzD8uwveOY=
|
||||
gvisor.dev/gvisor v0.0.0-20221203005347-703fd9b7fbc0/go.mod h1:Dn5idtptoW1dIos9U6A2rpebLs/MtTwFacjKb8jLdQA=
|
||||
+12
-88
@@ -3,99 +3,23 @@ package tun
|
||||
import "encoding/binary"
|
||||
|
||||
// TODO: Explore SIMD and/or other assembly optimizations.
|
||||
// TODO: Test native endian loads. See RFC 1071 section 2 part B.
|
||||
func checksumNoFold(b []byte, initial uint64) uint64 {
|
||||
ac := initial
|
||||
|
||||
for len(b) >= 128 {
|
||||
ac += uint64(binary.BigEndian.Uint32(b[:4]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[4:8]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[8:12]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[12:16]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[16:20]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[20:24]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[24:28]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[28:32]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[32:36]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[36:40]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[40:44]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[44:48]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[48:52]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[52:56]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[56:60]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[60:64]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[64:68]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[68:72]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[72:76]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[76:80]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[80:84]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[84:88]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[88:92]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[92:96]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[96:100]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[100:104]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[104:108]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[108:112]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[112:116]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[116:120]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[120:124]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[124:128]))
|
||||
b = b[128:]
|
||||
i := 0
|
||||
n := len(b)
|
||||
for n >= 4 {
|
||||
ac += uint64(binary.BigEndian.Uint32(b[i : i+4]))
|
||||
n -= 4
|
||||
i += 4
|
||||
}
|
||||
if len(b) >= 64 {
|
||||
ac += uint64(binary.BigEndian.Uint32(b[:4]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[4:8]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[8:12]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[12:16]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[16:20]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[20:24]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[24:28]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[28:32]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[32:36]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[36:40]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[40:44]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[44:48]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[48:52]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[52:56]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[56:60]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[60:64]))
|
||||
b = b[64:]
|
||||
for n >= 2 {
|
||||
ac += uint64(binary.BigEndian.Uint16(b[i : i+2]))
|
||||
n -= 2
|
||||
i += 2
|
||||
}
|
||||
if len(b) >= 32 {
|
||||
ac += uint64(binary.BigEndian.Uint32(b[:4]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[4:8]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[8:12]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[12:16]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[16:20]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[20:24]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[24:28]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[28:32]))
|
||||
b = b[32:]
|
||||
if n == 1 {
|
||||
ac += uint64(b[i]) << 8
|
||||
}
|
||||
if len(b) >= 16 {
|
||||
ac += uint64(binary.BigEndian.Uint32(b[:4]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[4:8]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[8:12]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[12:16]))
|
||||
b = b[16:]
|
||||
}
|
||||
if len(b) >= 8 {
|
||||
ac += uint64(binary.BigEndian.Uint32(b[:4]))
|
||||
ac += uint64(binary.BigEndian.Uint32(b[4:8]))
|
||||
b = b[8:]
|
||||
}
|
||||
if len(b) >= 4 {
|
||||
ac += uint64(binary.BigEndian.Uint32(b))
|
||||
b = b[4:]
|
||||
}
|
||||
if len(b) >= 2 {
|
||||
ac += uint64(binary.BigEndian.Uint16(b))
|
||||
b = b[2:]
|
||||
}
|
||||
if len(b) == 1 {
|
||||
ac += uint64(b[0]) << 8
|
||||
}
|
||||
|
||||
return ac
|
||||
}
|
||||
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
package tun
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func BenchmarkChecksum(b *testing.B) {
|
||||
lengths := []int{
|
||||
64,
|
||||
128,
|
||||
256,
|
||||
512,
|
||||
1024,
|
||||
1500,
|
||||
2048,
|
||||
4096,
|
||||
8192,
|
||||
9000,
|
||||
9001,
|
||||
}
|
||||
|
||||
for _, length := range lengths {
|
||||
b.Run(fmt.Sprintf("%d", length), func(b *testing.B) {
|
||||
buf := make([]byte, length)
|
||||
rng := rand.New(rand.NewSource(1))
|
||||
rng.Read(buf)
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
checksum(buf, 0)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+7
-7
@@ -25,7 +25,7 @@ import (
|
||||
"github.com/amnezia-vpn/amnezia-wg/tun"
|
||||
|
||||
"golang.org/x/net/dns/dnsmessage"
|
||||
"gvisor.dev/gvisor/pkg/buffer"
|
||||
"gvisor.dev/gvisor/pkg/bufferv2"
|
||||
"gvisor.dev/gvisor/pkg/tcpip"
|
||||
"gvisor.dev/gvisor/pkg/tcpip/adapters/gonet"
|
||||
"gvisor.dev/gvisor/pkg/tcpip/header"
|
||||
@@ -43,7 +43,7 @@ type netTun struct {
|
||||
ep *channel.Endpoint
|
||||
stack *stack.Stack
|
||||
events chan tun.Event
|
||||
incomingPacket chan *buffer.View
|
||||
incomingPacket chan *bufferv2.View
|
||||
mtu int
|
||||
dnsServers []netip.Addr
|
||||
hasV4, hasV6 bool
|
||||
@@ -61,7 +61,7 @@ func CreateNetTUN(localAddresses, dnsServers []netip.Addr, mtu int) (tun.Device,
|
||||
ep: channel.New(1024, uint32(mtu), ""),
|
||||
stack: stack.New(opts),
|
||||
events: make(chan tun.Event, 10),
|
||||
incomingPacket: make(chan *buffer.View),
|
||||
incomingPacket: make(chan *bufferv2.View),
|
||||
dnsServers: dnsServers,
|
||||
mtu: mtu,
|
||||
}
|
||||
@@ -84,7 +84,7 @@ func CreateNetTUN(localAddresses, dnsServers []netip.Addr, mtu int) (tun.Device,
|
||||
}
|
||||
protoAddr := tcpip.ProtocolAddress{
|
||||
Protocol: protoNumber,
|
||||
AddressWithPrefix: tcpip.AddrFromSlice(ip.AsSlice()).WithPrefix(),
|
||||
AddressWithPrefix: tcpip.Address(ip.AsSlice()).WithPrefix(),
|
||||
}
|
||||
tcpipErr := dev.stack.AddProtocolAddress(1, protoAddr, stack.AddressProperties{})
|
||||
if tcpipErr != nil {
|
||||
@@ -140,7 +140,7 @@ func (tun *netTun) Write(buf [][]byte, offset int) (int, error) {
|
||||
continue
|
||||
}
|
||||
|
||||
pkb := stack.NewPacketBuffer(stack.PacketBufferOptions{Payload: buffer.MakeWithData(packet)})
|
||||
pkb := stack.NewPacketBuffer(stack.PacketBufferOptions{Payload: bufferv2.MakeWithData(packet)})
|
||||
switch packet[0] >> 4 {
|
||||
case 4:
|
||||
tun.ep.InjectInbound(header.IPv4ProtocolNumber, pkb)
|
||||
@@ -198,7 +198,7 @@ func convertToFullAddr(endpoint netip.AddrPort) (tcpip.FullAddress, tcpip.Networ
|
||||
}
|
||||
return tcpip.FullAddress{
|
||||
NIC: 1,
|
||||
Addr: tcpip.AddrFromSlice(endpoint.Addr().AsSlice()),
|
||||
Addr: tcpip.Address(endpoint.Addr().AsSlice()),
|
||||
Port: endpoint.Port(),
|
||||
}, protoNumber
|
||||
}
|
||||
@@ -453,7 +453,7 @@ func (pc *PingConn) ReadFrom(p []byte) (n int, addr net.Addr, err error) {
|
||||
return 0, nil, fmt.Errorf("ping read: %s", tcpipErr)
|
||||
}
|
||||
|
||||
remoteAddr, _ := netip.AddrFromSlice(res.RemoteAddr.Addr.AsSlice())
|
||||
remoteAddr, _ := netip.AddrFromSlice([]byte(res.RemoteAddr.Addr))
|
||||
return res.Count, &PingAddr{remoteAddr}, nil
|
||||
}
|
||||
|
||||
|
||||
+63
-99
@@ -269,11 +269,11 @@ func tcpChecksumValid(pkt []byte, iphLen uint8, isV6 bool) bool {
|
||||
type coalesceResult int
|
||||
|
||||
const (
|
||||
coalesceInsufficientCap coalesceResult = iota
|
||||
coalescePSHEnding
|
||||
coalesceItemInvalidCSum
|
||||
coalescePktInvalidCSum
|
||||
coalesceSuccess
|
||||
coalesceInsufficientCap coalesceResult = 0
|
||||
coalescePSHEnding coalesceResult = 1
|
||||
coalesceItemInvalidCSum coalesceResult = 2
|
||||
coalescePktInvalidCSum coalesceResult = 3
|
||||
coalesceSuccess coalesceResult = 4
|
||||
)
|
||||
|
||||
// coalesceTCPPackets attempts to coalesce pkt with the packet described by
|
||||
@@ -339,6 +339,42 @@ func coalesceTCPPackets(mode canCoalesce, pkt []byte, pktBuffsIndex int, gsoSize
|
||||
if gsoSize > item.gsoSize {
|
||||
item.gsoSize = gsoSize
|
||||
}
|
||||
hdr := virtioNetHdr{
|
||||
flags: unix.VIRTIO_NET_HDR_F_NEEDS_CSUM, // this turns into CHECKSUM_PARTIAL in the skb
|
||||
hdrLen: uint16(headersLen),
|
||||
gsoSize: uint16(item.gsoSize),
|
||||
csumStart: uint16(item.iphLen),
|
||||
csumOffset: 16,
|
||||
}
|
||||
|
||||
// Recalculate the total len (IPv4) or payload len (IPv6). Recalculate the
|
||||
// (IPv4) header checksum.
|
||||
if isV6 {
|
||||
hdr.gsoType = unix.VIRTIO_NET_HDR_GSO_TCPV6
|
||||
binary.BigEndian.PutUint16(pktHead[4:], uint16(coalescedLen)-uint16(item.iphLen)) // set new payload len
|
||||
} else {
|
||||
hdr.gsoType = unix.VIRTIO_NET_HDR_GSO_TCPV4
|
||||
pktHead[10], pktHead[11] = 0, 0 // clear checksum field
|
||||
binary.BigEndian.PutUint16(pktHead[2:], uint16(coalescedLen)) // set new total length
|
||||
iphCSum := ^checksum(pktHead[:item.iphLen], 0) // compute checksum
|
||||
binary.BigEndian.PutUint16(pktHead[10:], iphCSum) // set checksum field
|
||||
}
|
||||
hdr.encode(bufs[item.bufsIndex][bufsOffset-virtioNetHdrLen:])
|
||||
|
||||
// Calculate the pseudo header checksum and place it at the TCP checksum
|
||||
// offset. Downstream checksum offloading will combine this with computation
|
||||
// of the tcp header and payload checksum.
|
||||
addrLen := 4
|
||||
addrOffset := ipv4SrcAddrOffset
|
||||
if isV6 {
|
||||
addrLen = 16
|
||||
addrOffset = ipv6SrcAddrOffset
|
||||
}
|
||||
srcAddrAt := bufsOffset + addrOffset
|
||||
srcAddr := bufs[item.bufsIndex][srcAddrAt : srcAddrAt+addrLen]
|
||||
dstAddr := bufs[item.bufsIndex][srcAddrAt+addrLen : srcAddrAt+addrLen*2]
|
||||
psum := pseudoHeaderChecksumNoFold(unix.IPPROTO_TCP, srcAddr, dstAddr, uint16(coalescedLen-int(item.iphLen)))
|
||||
binary.BigEndian.PutUint16(pktHead[hdr.csumStart+hdr.csumOffset:], checksum([]byte{}, psum))
|
||||
|
||||
item.numMerged++
|
||||
return coalesceSuccess
|
||||
@@ -354,52 +390,43 @@ const (
|
||||
maxUint16 = 1<<16 - 1
|
||||
)
|
||||
|
||||
type tcpGROResult int
|
||||
|
||||
const (
|
||||
tcpGROResultNoop tcpGROResult = iota
|
||||
tcpGROResultTableInsert
|
||||
tcpGROResultCoalesced
|
||||
)
|
||||
|
||||
// tcpGRO evaluates the TCP packet at pktI in bufs for coalescing with
|
||||
// existing packets tracked in table. It returns a tcpGROResultNoop when no
|
||||
// action was taken, tcpGROResultTableInsert when the evaluated packet was
|
||||
// inserted into table, and tcpGROResultCoalesced when the evaluated packet was
|
||||
// coalesced with another packet in table.
|
||||
func tcpGRO(bufs [][]byte, offset int, pktI int, table *tcpGROTable, isV6 bool) tcpGROResult {
|
||||
// existing packets tracked in table. It will return false when pktI is not
|
||||
// coalesced, otherwise true. This indicates to the caller if bufs[pktI]
|
||||
// should be written to the Device.
|
||||
func tcpGRO(bufs [][]byte, offset int, pktI int, table *tcpGROTable, isV6 bool) (pktCoalesced bool) {
|
||||
pkt := bufs[pktI][offset:]
|
||||
if len(pkt) > maxUint16 {
|
||||
// A valid IPv4 or IPv6 packet will never exceed this.
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
iphLen := int((pkt[0] & 0x0F) * 4)
|
||||
if isV6 {
|
||||
iphLen = 40
|
||||
ipv6HPayloadLen := int(binary.BigEndian.Uint16(pkt[4:]))
|
||||
if ipv6HPayloadLen != len(pkt)-iphLen {
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
} else {
|
||||
totalLen := int(binary.BigEndian.Uint16(pkt[2:]))
|
||||
if totalLen != len(pkt) {
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
}
|
||||
if len(pkt) < iphLen {
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
tcphLen := int((pkt[iphLen+12] >> 4) * 4)
|
||||
if tcphLen < 20 || tcphLen > 60 {
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
if len(pkt) < iphLen+tcphLen {
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
if !isV6 {
|
||||
if pkt[6]&ipv4FlagMoreFragments != 0 || pkt[6]<<3 != 0 || pkt[7] != 0 {
|
||||
// no GRO support for fragmented segments for now
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
}
|
||||
tcpFlags := pkt[iphLen+tcpFlagsOffset]
|
||||
@@ -407,14 +434,14 @@ func tcpGRO(bufs [][]byte, offset int, pktI int, table *tcpGROTable, isV6 bool)
|
||||
// not a candidate if any non-ACK flags (except PSH+ACK) are set
|
||||
if tcpFlags != tcpFlagACK {
|
||||
if pkt[iphLen+tcpFlagsOffset] != tcpFlagACK|tcpFlagPSH {
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
pshSet = true
|
||||
}
|
||||
gsoSize := uint16(len(pkt) - tcphLen - iphLen)
|
||||
// not a candidate if payload len is 0
|
||||
if gsoSize < 1 {
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
seq := binary.BigEndian.Uint32(pkt[iphLen+4:])
|
||||
srcAddrOffset := ipv4SrcAddrOffset
|
||||
@@ -425,7 +452,7 @@ func tcpGRO(bufs [][]byte, offset int, pktI int, table *tcpGROTable, isV6 bool)
|
||||
}
|
||||
items, existing := table.lookupOrInsert(pkt, srcAddrOffset, srcAddrOffset+addrLen, iphLen, tcphLen, pktI)
|
||||
if !existing {
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
}
|
||||
for i := len(items) - 1; i >= 0; i-- {
|
||||
// In the best case of packets arriving in order iterating in reverse is
|
||||
@@ -443,20 +470,20 @@ func tcpGRO(bufs [][]byte, offset int, pktI int, table *tcpGROTable, isV6 bool)
|
||||
switch result {
|
||||
case coalesceSuccess:
|
||||
table.updateAt(item, i)
|
||||
return tcpGROResultCoalesced
|
||||
return true
|
||||
case coalesceItemInvalidCSum:
|
||||
// delete the item with an invalid csum
|
||||
table.deleteAt(item.key, i)
|
||||
case coalescePktInvalidCSum:
|
||||
// no point in inserting an item that we can't coalesce
|
||||
return tcpGROResultNoop
|
||||
return false
|
||||
default:
|
||||
}
|
||||
}
|
||||
}
|
||||
// failed to coalesce with any other packets; store the item in the flow
|
||||
table.insert(pkt, srcAddrOffset, srcAddrOffset+addrLen, iphLen, tcphLen, pktI)
|
||||
return tcpGROResultTableInsert
|
||||
return false
|
||||
}
|
||||
|
||||
func isTCP4NoIPOptions(b []byte) bool {
|
||||
@@ -488,64 +515,6 @@ func isTCP6NoEH(b []byte) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// applyCoalesceAccounting updates bufs to account for coalescing based on the
|
||||
// metadata found in table.
|
||||
func applyCoalesceAccounting(bufs [][]byte, offset int, table *tcpGROTable, isV6 bool) error {
|
||||
for _, items := range table.itemsByFlow {
|
||||
for _, item := range items {
|
||||
if item.numMerged > 0 {
|
||||
hdr := virtioNetHdr{
|
||||
flags: unix.VIRTIO_NET_HDR_F_NEEDS_CSUM, // this turns into CHECKSUM_PARTIAL in the skb
|
||||
hdrLen: uint16(item.iphLen + item.tcphLen),
|
||||
gsoSize: item.gsoSize,
|
||||
csumStart: uint16(item.iphLen),
|
||||
csumOffset: 16,
|
||||
}
|
||||
pkt := bufs[item.bufsIndex][offset:]
|
||||
|
||||
// Recalculate the total len (IPv4) or payload len (IPv6).
|
||||
// Recalculate the (IPv4) header checksum.
|
||||
if isV6 {
|
||||
hdr.gsoType = unix.VIRTIO_NET_HDR_GSO_TCPV6
|
||||
binary.BigEndian.PutUint16(pkt[4:], uint16(len(pkt))-uint16(item.iphLen)) // set new IPv6 header payload len
|
||||
} else {
|
||||
hdr.gsoType = unix.VIRTIO_NET_HDR_GSO_TCPV4
|
||||
pkt[10], pkt[11] = 0, 0
|
||||
binary.BigEndian.PutUint16(pkt[2:], uint16(len(pkt))) // set new total length
|
||||
iphCSum := ^checksum(pkt[:item.iphLen], 0) // compute IPv4 header checksum
|
||||
binary.BigEndian.PutUint16(pkt[10:], iphCSum) // set IPv4 header checksum field
|
||||
}
|
||||
err := hdr.encode(bufs[item.bufsIndex][offset-virtioNetHdrLen:])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Calculate the pseudo header checksum and place it at the TCP
|
||||
// checksum offset. Downstream checksum offloading will combine
|
||||
// this with computation of the tcp header and payload checksum.
|
||||
addrLen := 4
|
||||
addrOffset := ipv4SrcAddrOffset
|
||||
if isV6 {
|
||||
addrLen = 16
|
||||
addrOffset = ipv6SrcAddrOffset
|
||||
}
|
||||
srcAddrAt := offset + addrOffset
|
||||
srcAddr := bufs[item.bufsIndex][srcAddrAt : srcAddrAt+addrLen]
|
||||
dstAddr := bufs[item.bufsIndex][srcAddrAt+addrLen : srcAddrAt+addrLen*2]
|
||||
psum := pseudoHeaderChecksumNoFold(unix.IPPROTO_TCP, srcAddr, dstAddr, uint16(len(pkt)-int(item.iphLen)))
|
||||
binary.BigEndian.PutUint16(pkt[hdr.csumStart+hdr.csumOffset:], checksum([]byte{}, psum))
|
||||
} else {
|
||||
hdr := virtioNetHdr{}
|
||||
err := hdr.encode(bufs[item.bufsIndex][offset-virtioNetHdrLen:])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// handleGRO evaluates bufs for GRO, and writes the indices of the resulting
|
||||
// packets into toWrite. toWrite, tcp4Table, and tcp6Table should initially be
|
||||
// empty (but non-nil), and are passed in to save allocs as the caller may reset
|
||||
@@ -555,28 +524,23 @@ func handleGRO(bufs [][]byte, offset int, tcp4Table, tcp6Table *tcpGROTable, toW
|
||||
if offset < virtioNetHdrLen || offset > len(bufs[i])-1 {
|
||||
return errors.New("invalid offset")
|
||||
}
|
||||
var result tcpGROResult
|
||||
var coalesced bool
|
||||
switch {
|
||||
case isTCP4NoIPOptions(bufs[i][offset:]): // ipv4 packets w/IP options do not coalesce
|
||||
result = tcpGRO(bufs, offset, i, tcp4Table, false)
|
||||
coalesced = tcpGRO(bufs, offset, i, tcp4Table, false)
|
||||
case isTCP6NoEH(bufs[i][offset:]): // ipv6 packets w/extension headers do not coalesce
|
||||
result = tcpGRO(bufs, offset, i, tcp6Table, true)
|
||||
coalesced = tcpGRO(bufs, offset, i, tcp6Table, true)
|
||||
}
|
||||
switch result {
|
||||
case tcpGROResultNoop:
|
||||
if !coalesced {
|
||||
hdr := virtioNetHdr{}
|
||||
err := hdr.encode(bufs[i][offset-virtioNetHdrLen:])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fallthrough
|
||||
case tcpGROResultTableInsert:
|
||||
*toWrite = append(*toWrite, i)
|
||||
}
|
||||
}
|
||||
err4 := applyCoalesceAccounting(bufs, offset, tcp4Table, false)
|
||||
err6 := applyCoalesceAccounting(bufs, offset, tcp6Table, true)
|
||||
return errors.Join(err4, err6)
|
||||
return nil
|
||||
}
|
||||
|
||||
// tcpTSO splits packets from in into outBuffs, writing the size of each
|
||||
|
||||
@@ -35,8 +35,8 @@ func tcp4PacketMutateIPFields(srcIPPort, dstIPPort netip.AddrPort, flags header.
|
||||
srcAs4 := srcIPPort.Addr().As4()
|
||||
dstAs4 := dstIPPort.Addr().As4()
|
||||
ipFields := &header.IPv4Fields{
|
||||
SrcAddr: tcpip.AddrFromSlice(srcAs4[:]),
|
||||
DstAddr: tcpip.AddrFromSlice(dstAs4[:]),
|
||||
SrcAddr: tcpip.Address(srcAs4[:]),
|
||||
DstAddr: tcpip.Address(dstAs4[:]),
|
||||
Protocol: unix.IPPROTO_TCP,
|
||||
TTL: 64,
|
||||
TotalLength: uint16(totalLen),
|
||||
@@ -72,8 +72,8 @@ func tcp6PacketMutateIPFields(srcIPPort, dstIPPort netip.AddrPort, flags header.
|
||||
srcAs16 := srcIPPort.Addr().As16()
|
||||
dstAs16 := dstIPPort.Addr().As16()
|
||||
ipFields := &header.IPv6Fields{
|
||||
SrcAddr: tcpip.AddrFromSlice(srcAs16[:]),
|
||||
DstAddr: tcpip.AddrFromSlice(dstAs16[:]),
|
||||
SrcAddr: tcpip.Address(srcAs16[:]),
|
||||
DstAddr: tcpip.Address(dstAs16[:]),
|
||||
TransportProtocol: unix.IPPROTO_TCP,
|
||||
HopLimit: 64,
|
||||
PayloadLength: uint16(segmentSize + 20),
|
||||
|
||||
@@ -127,9 +127,6 @@ func (tun *NativeTun) MTU() (int, error) {
|
||||
|
||||
// TODO: This is a temporary hack. We really need to be monitoring the interface in real time and adapting to MTU changes.
|
||||
func (tun *NativeTun) ForceMTU(mtu int) {
|
||||
if tun.close.Load() {
|
||||
return
|
||||
}
|
||||
update := tun.forcedMTU != mtu
|
||||
tun.forcedMTU = mtu
|
||||
if update {
|
||||
|
||||
Reference in new issue
Block a user