What you need to know about Internet Protocol Security (IPsec)
Internet Protocol Security is a secure network protocol suite that authenticates and encrypts packets of data to provide secure communication between two endpoints. In this article, we’ll explain what IPsec is, how it works, some of its limitations, and compare it with an open source VPN alternative, WireGuard®.
Internet Protocol Security (IPsec) is a network protocol suite for enabling secure IP connections between devices. It authenticates and encrypts packets of data to provide secure encrypted communication between two endpoints over an IP network.
IPsec is commonly used to set up VPNs for connecting branch offices and remote workers to a corporate network. Because it ensures the security of data in transit, IPsec is considered more suitable than SSL-based VPNs for encrypting corporate VPN traffic.
An IPsec VPN may not always be the best solution, though. In this article, we’ll discuss what IPsec is, how it works, and what its limitations are, and suggest an alternative that might better suit your needs.
What is IPsec?
IPsec is a secure network connection protocol used to set up encrypted connections between devices to ensure the security of data transmitted over the internet or any public network. First developed in 1995, IPsec is an Internet Engineering Task Force (IETF) standard that adds encryption and authentication on top of the IP routing process to make it more secure.
IPsec provides a protocol for secure network connections by encrypting IP packets and authenticating packet origins. IPsec operates at the network layer, enabling it to handle all traffic that passes through the network. Because of this, it’s not dependent on the application being used and rarely requires individual application configuration.
How IPsec works
IPsec defines protocols for secure key exchange and management. The suite comprises several protocols, including Authentication Header, Encapsulating Security Payload, and protocols for creating security associations such as the Internet Key Exchange (IKE) protocol.
Authentication Header
The Authentication Header (AH) protocol authenticates data and is commonly used for VPN configurations where encryption is not allowed. It’s used to confirm that packets have originated from a trusted source and ensures data integrity. It also provides authentication for IP headers and upper layer protocols and protects against replay attacks. AH is rarely used in modern VPN configurations, and Encapsulating Security Payload is the preferred encryption/integrity method.
Encapsulating Security Payload
Encapsulating Security Payload (ESP) encrypts data and performs checks to guarantee the confidentiality and integrity of the data. ESP adds its own header and trailer to each data packet. ESP encrypts both the IP header and payload for each packet when in IPsec tunnel mode and only the payload when in transport mode.
Security Association
A set of IPsec specifications called a security association (SA) is negotiated between communicating devices. SAs provide the encryption and decryption processes needed to negotiate security levels between two devices. During this process, connected devices establish a key exchange that allows them to decrypt each other’s messages.
SAs are stored in the Security Associations Database (SADB), and a unique identification tag, the Security Parameter Index (SPI), is assigned to each SA entry in the database. The SPI is added to the header of outgoing packets so that the receiving device can use it to identify the SA associated with the packet and look it up in the SADB.
Configuring IPsec
The IPsec protocol is supported natively on major platforms including Windows, macOS, iOS, and Android, with no additional software required. There are two modes of configuration — transport mode and tunnel mode — for both AH and ESP protocols.
Configuring IPsec in transport mode
IPsec transport mode authenticates and/or encrypts data packets but does not modify the IP header. Encrypted traffic is sent between hosts that have a previously established IPsec connection. Transport mode for the AH protocol signs the entire packet for integrity and authentication. ESP transport mode signs the original packet payload, but not the IP header.
Configuring IPsec transport mode is a two-step process: Phase one negotiates a secure channel between two devices by creating an SA using the Internet Security Association and Key Management Protocol (ISAKMP). In phase two, peers use Internet Key Exchange or a similar protocol to negotiate authentication and encryption. Because IPsec transport mode does not encrypt headers, it’s less secure than tunnel mode.
Configuring IPsec in tunnel mode
IPsec tunnel mode creates virtual tunnels between two connection points on a VPN and is the preferred mode in most VPNs because it can be used to connect two different networks securely.
Tunnel mode for the AH protocol ensures the integrity of data packets but does not encrypt data. IPsec tunnels set up using AH do not support NAT (more on this below). For ESP, tunnel mode encrypts both the IP header and the data packet to ensure data authenticity and integrity when routing VPN connections through untrusted networks.
To configure IPsec tunnel mode, you need a protocol such as IKE to negotiate secure communication between the devices. There are two phases: Phase one authenticates the IPsec peers and negotiates SAs to establish a secure tunnel between channels. Then phase two negotiates authentication and encryption, encrypts the entire packet, and adds it to the payload.
IPsec limitations
While IPsec is popular and widely used, its large overheads and conflicts with network address translation (NAT) can prove challenging. Let’s look at some of these issues in detail.
Packets can exceed router limits
Because of the overhead associated with adding new protocol headers and packet encapsulation, IPsec packets can often become larger than the maximum transmission unit (MTU) — the top limit for the size of a packet supported by the local network. When this occurs, the package is fragmented or broken down into packets that are smaller than or equal to the allowable size.
If one fragment is lost, all fragments must be resent, and since only the first fragment contains the header, the firewall or router may discard the other fragments because they can’t be validated. Fragmentation can also result in packets being delivered in the incorrect order, resulting in them having to be retransmitted.
Issues with NAT
Network address translation (NAT) maps an IP address space into another by changing information in packet headers while in transit through a router. This can cause problems with the IPsec protocol:
- Changed IP identifiers causing dropped or discarded packets: When using methods such as preshared key authentication, IPsec IKE uses IP addresses as identifiers and IKE embeds the IP address of the source device into its payload. When a packet goes through a NAT device, NAT changes the source IP address in the packet to the IP address of the NAT device. Since this address does not match the source address, the receiving device will drop the packet, causing all packets to be resent.
- Failed integrity checks and invalid checksums: The IPsec AH protocol uses an integrity check value (ICV) on immutable IP header fields such as the source and destination address. These fields are also used to calculate TCP and UDP checksums for verifying packets. In typical NAT communication, the NAT device replaces the source IP address and port with its IP address and port. These changes can cause failed integrity checks and invalid checksums.
Now that we’ve discussed some of the challenges with using IPsec to secure your communications, you may be wondering if there is a suitable alternative. WireGuard® is an open source, modern VPN protocol that’s simple to use and easy to implement. Let’s do a quick comparison.
IPsec vs. WireGuard
IPsec and WireGuard are both VPN protocols used to connect remote networks and access private infrastructure. You can read more in our in-depth comparison of IPsec and WireGuard on platform availability, user experience, and security, but for the purposes of this article, we’ll focus on a brief comparison of key benefits of the two protocols.
Advantages of IPsec
IPsec is commonly used for secure communication on corporate VPNs for several reasons:
- Wide adoption: IPsec is widely adopted with native clients on all major operating systems — i.e., Windows, Linux, iOS, and Android. Supported IPsec implementations are also available for major router manufacturers including Cisco and Juniper, as well as IoT devices.
- Ease of use and familiarity: IPsec’s connection management approach is familiar to users and easy to implement.
- Compatibility with legacy systems: IPsec offers many encryption options, including older encryption methods such as the RSA algorithm and preshared keys that allow you to add a legacy system to an existing IPsec VPN. However, these legacy encryption methods can be more complicated to set up and can create security vulnerabilities in the VPN.
Advantages of WireGuard
WireGuard is a relatively new protocol that’s fast becoming popular in the VPN industry because of its high-speed performance and advanced encryption methods. Here are some of its best features:
- Smaller, more secure code base: IPsec includes legacy protocols resulting in a very large code base that has not been formally verified. WireGuard’s code base is smaller and simpler, with little legacy functionality, making it easier for the open source community to audit for security. WireGuard’s codebase has also been formally verified and documented in a cryptographic analysis.
- Faster than IPsec: WireGuard is a Linux kernel module, which improves performance compared to a process running in user space. WireGuard has conducted performance benchmarks that show about a 15% higher throughput compared to IPsec.
- Built with firewalls in mind: For IPsec to work with a firewall, you need to open UDP port 500 for ISAKMP traffic and allow IP protocols 50 and 51 on firewall filters for ESP and AH traffic, respectively. WireGuard doesn’t need open ports. It uses the UDP protocol and can be configured to use any port.
- Wide platform support: There are Wireguard implementations available for the major operating systems. Note, however, that the protocol is not yet natively supported on Android or iOS; you’ll need WireGuard iOS and Android apps.
- Reduced likelihood of misconfiguration: While WireGuard has fewer encryption options than IPsec, it uses more modern encryption solutions with more secure default configurations. Clients and servers cannot specify insecure encryption options, ensuring that end users are all on a recent encryption standard.
Conclusion
Though a commonly used VPN protocol, IPsec has several drawbacks, including packet fragmentation, failed data integrity checks, and the propensity to create security vulnerabilities when incorrectly configured. WireGuard, built with firewalls in mind, offers modern encryption solutions that allow you to create a more secure VPN.
As the most advanced WireGuard VPN on the market, Tailscale allows you to create encrypted point-to-point connections for remote access to your devices and applications from anywhere in the world. Tailscale is easy to configure, use, and audit, and is suitable for business environments of any size. Download Tailscale to get started.
FAQs
How does IPsec work?
IPsec defines several protocols for secure key exchange and management, including Authentication Header, which confirms that packets have originated from a trusted source and ensures data integrity; Encapsulating Security Payload, which encrypts data and performs checks to guarantee the confidentiality and integrity of the data; and security association, a set of IPsec specifications that provide the encryption and decryption processes needed to negotiate security levels between two devices.
What are some of IPsec’s challenges?
IPsec requires additional overhead for new protocol headers and packet encapsulation, which can cause packets to exceed the size supported by local networks. When this occurs, the package is fragmented into smaller units, which can lead to lost or discarded data.
NAT traversal can also cause complications with IPsec when a NAT device changes a packet’s source IP address, which can lead to dropped packets or failed integrity checks and invalid checksums.
