VPN Server on Synology: L2TP/IPsec

2019-01-31 · synology, vpn, l2tp

Every few months someone asks me to just open SMB to the internet so they can grab a file from home. No. The right way in is a VPN, and the VPN Server package on DSM 6.x offers PPTP, OpenVPN and L2TP/IPsec. PPTP is broken; don't use it. OpenVPN is the most flexible but needs a client on every device. L2TP/IPsec is the pragmatic middle: Windows, macOS, iOS and Android all have it built in, so users only need a server address, a pre-shared key and their DSM login.

I'll cover the setup on DSM 6.2, the ports, and the Windows-behind-NAT problem that catches almost everyone. You need VPN Server installed, a public hostname (DDNS is fine), and router access for port forwarding. Also check your subnets: if the office is 192.168.1.0/24 and users are at home on the same range, routing fails. I renumber the office to something like 10.20.30.0/24 before deploying a VPN.

Enabling L2TP/IPsec

Open VPN Server → L2TP/IPsec → tick "Enable L2TP/IPsec VPN server":

Dynamic IP address:      10.99.0.0
Maximum connections:     10
Max connections per account: 2
Authentication:          MS-CHAP v2
MTU:                     1400
Use manual DNS:          10.20.30.1 (your internal DNS)
Enable SHA2-256 compatible mode: on
Pre-shared key:          long random string
Enable L2TP-over-IPsec:  (this is what the whole page is)

The dynamic range is what remote clients receive; keep it disjoint from every real subnet. The pre-shared key is shared by all users and only authenticates the IPsec layer; user identity comes from the DSM login on top. SHA2-256 compatible mode is needed for recent Windows and macOS clients that refuse SHA1.

Lower the MTU to 1400 from the start. L2TP inside IPsec adds overhead, and the default 1500 leads to hanging SMB transfers and web pages that never finish loading.

Who may connect

VPN Server → Privilege. Every DSM user is listed with a checkbox per protocol. Tick L2TP for the people who need it and nobody else. Users in the administrators group are not automatically allowed; the checkbox is what matters. Auto Block covers VPN logins, so a user typing the wrong password repeatedly will be locked out at the IP level just like on the web interface.

Ports

L2TP/IPsec uses three UDP ports:

UDP 500    IKE
UDP 4500   NAT-T
UDP 1701   L2TP

On the DSM firewall (Control Panel → Security → Firewall → Edit Rules) add an Allow rule using the built-in application "VPN Server (L2TP)", which covers all three. On the router, forward the three ports to the NAS. Some routers have an "IPsec passthrough" option; enable it. Confirm from outside that at least IKE answers:

nmap -sU -p 500,4500 nas.example.com

Expect open|filtered on both; UDP scans cannot say more than that.

Windows client

Settings → Network & Internet → VPN → Add a VPN connection:

VPN provider:        Windows (built-in)
Server name:         nas.example.com
VPN type:            L2TP/IPsec with pre-shared key
Pre-shared key:      as set on the NAS
Sign-in info:        User name and password

If the client PC is behind its own NAT router, which is nearly always, Windows refuses to establish the tunnel until a registry value is set. Run in an elevated PowerShell and reboot:

New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\PolicyAgent" -Name AssumeUDPEncapsulationContextOnSendRule -PropertyType DWord -Value 2 -Force
Restart-Computer

Value 2 allows both server and client behind NAT. Without it the connection fails with error 809.

On macOS the connection is System Preferences → Network → plus → VPN → L2TP over IPsec; on iOS it is Settings → General → VPN → Add VPN Configuration → L2TP. Both accept the shared secret under Authentication Settings.

Split tunnelling

By default Windows sends all traffic through the VPN once connected. For access to internal resources only, untick "Use default gateway on remote network" in the adapter's IPv4 advanced settings, or in PowerShell:

Set-VpnConnection -Name "Office NAS" -SplitTunneling $true
Add-VpnConnectionRoute -ConnectionName "Office NAS" -DestinationPrefix 10.20.30.0/24

I turn split tunnelling on for almost everyone. It's faster for them and easier on the NAS uplink.

Trying it out

Connect from a phone on mobile data first, because it avoids the home-router NAT variable. Once connected:

VPN Server → Connection List shows the user, the assigned 10.99.0.x address and the protocol.
From the client, open https://10.20.30.5:5001 (the NAS LAN address) and \\10.20.30.5 for SMB.
On the NAS, the tunnel interface exists: ip addr show ppp0 and sudo cat /var/log/messages | grep -i pppd for the negotiation lines.

When it fails

Windows error 809 "The network connection between your computer and the VPN server could not be established": the registry value above is missing, or UDP 500/4500 is not forwarded. Check the registry first.
Windows error 691 "The remote connection was denied because the user name and password combination you provided is not recognized": the user is not ticked under Privilege, or the account is locked by Auto Block. Look at VPN Server → Log.
Connection succeeds but nothing on the LAN is reachable. The client's home subnet overlaps the office subnet, or "Allow clients to access server's LAN" is unticked under General Settings in VPN Server.
Large files stall while small ones work: MTU. Lower it to 1400 on the server if you left the default, and on the Windows side netsh interface ipv4 set subinterface "Office NAS" mtu=1400 store=persistent.

If the user count grows or you want per-user certificates, move people to OpenVPN. Both can run on the same package at once, so there's no rush.