What is IPsec? Comprehensive Guide to Internet Protocol Security

What is IPsec?

Internet Protocol Security (IPsec) is a network protocol suite for enabling secure IP connections between devices. It authenticates and encrypts packets of data to provide secure encrypted communication between two endpoints over an IP network, ensuring data confidentiality.

IPsec is commonly used to set up VPNs for connecting branch offices and remote workers to a corporate network. Because it ensures the security of data in transit, IPsec is considered more suitable than SSL-based VPNs for encrypting corporate VPN traffic.

An IPsec VPN may not always be the best solution, though. In this article, we’ll discuss what IPsec is, how it works, and what its limitations are, and suggest an alternative that might better suit your needs.

A brief IPsec history

First developed in 1995, IPsec is an Internet Engineering Task Force (IETF) standard that adds encryption and authentication on top of the IP routing process to make it more secure.

IPsec provides a protocol for secure network connections by encrypting IP packets and authenticating packet origins. IPsec operates at the network layer, enabling it to handle all traffic that passes through the network. Because of this, it’s not dependent on the application being used and rarely requires individual application configuration.

How does IPsec work?

IPsec defines protocols for secure key exchange and management. The suite comprises several protocols, including Authentication Header, Encapsulating Security Payload, and protocols for creating security associations such as the Internet Key Exchange (IKE) protocol. IPsec provides a protocol for secure network connections by encrypting the entire IP packet and authenticating packet origins.

The Authentication Header (AH) protocol authenticates data and is commonly used for VPN configurations where encryption is not allowed. It’s used to confirm that packets have originated from a trusted source and ensures data integrity. It also provides authentication for IP headers and upper layer protocols and protects against replay attacks. AH is rarely used in modern VPN configurations, and Encapsulating Security Payload is the preferred encryption/integrity method.

IPsec Connection Establishment Process

The IPsec connection establishment process involves several steps that ensure secure communication between two devices. This process is divided into two phases: Phase 1 and Phase 2.

Phase 1: Internet Key Exchange (IKE) Tunnel Establishment

In this phase, the two devices establish a secure tunnel using the Internet Key Exchange (IKE) protocol. IKE allows the devices to negotiate the encryption, authentication, and other protocols necessary for secure communication.

The steps involved are:

Negotiation: The devices negotiate the encryption, authentication, and other protocols to be used.
Diffie-Hellman Key Exchange: The devices exchange keying material using the Diffie-Hellman key exchange algorithm.
Authentication: The devices authenticate each other using digital certificates or pre-shared keys.

Phase 2: IPsec Tunnel Establishment

In this phase, the devices establish the IPsec tunnel using the protocols negotiated in Phase 1.

The steps involved are:

Quick Mode: The devices negotiate the encryption, authentication, and other protocols for the IPsec tunnel.
IPsec SA Establishment: The devices establish the IPsec Security Association (SA) using the protocols negotiated in Quick Mode.

IPsec Transmission

Once the IPsec tunnel is established, the devices can start transmitting data securely. The IPsec transmission process involves several steps:

Data Encryption: The data is encrypted using the encryption algorithm negotiated in Phase 1.
Authentication Header (AH) or Encapsulating Security Payload (ESP) Addition: The Authentication Header (AH) or Encapsulating Security Payload (ESP) is added to the encrypted data.
IP Header Addition: The IP header is added to the encrypted data.
Transmission: The encrypted data is transmitted over the IP network.

Encapsulating Security Payload

Encapsulating Security Payload (ESP) encrypts data and performs checks to guarantee the confidentiality and integrity of the data. ESP adds its own header and trailer to each data packet.

ESP encrypts both the IP header and payload for each packet when in IPsec tunnel mode and only the payload when in transport mode.

Security Association

A set of IPsec specifications called a security association (SA) is negotiated between communicating devices. SAs provide the encryption and decryption processes needed to negotiate security levels between two devices.

During this process, connected devices establish a key exchange that allows them to decrypt each other's messages.

SAs are stored in the Security Associations Database (SADB), and a unique identification tag, the Security Parameter Index (SPI), is assigned to each SA entry in the database.

The SPI is added to the header of outgoing packets so that the receiving device can use it to identify the SA associated with the packet and look it up in the SADB.

Configuring IPsec

The IPsec protocol is supported natively on major platforms including Windows, macOS, iOS, and Android, with no additional software required. There are two modes of configuration — transport mode and tunnel mode — for both AH and ESP protocols.

Configuring IPsec in transport mode

IPsec transport mode authenticates and/or encrypts sensitive data packets but does not modify the IP header. Encrypted traffic is sent between hosts that have a previously established IPsec connection. Transport mode for the AH protocol signs the entire packet for integrity and authentication. ESP transport mode signs the original packet payload, but not the IP header.

Configuring IPsec transport mode is a two-step process: Phase one negotiates a secure channel between two devices by creating an SA using the Internet Security Association and Key Management Protocol (ISAKMP). In phase two, peers use Internet Key Exchange or a similar protocol to negotiate authentication and encryption. Because IPsec transport mode does not encrypt headers, it’s less secure than tunnel mode.

Configuring IPsec in tunnel mode

IPsec tunnel mode creates virtual private network tunnels between two connection points on a VPN and is the preferred mode in most VPNs because it can be used to connect two different networks securely.

Tunnel mode for the AH protocol ensures the integrity of data packets but does not encrypt data. IPsec tunnels set up using AH do not support Network Address Translator (NAT). For ESP, tunnel mode encrypts both the IP header and the data packet to ensure data authenticity and integrity when routing VPN connections through untrusted networks.

To configure IPsec tunnel mode, you need a protocol such as IKE to negotiate secure communication between the devices. There are two phases: Phase one authenticates the IPsec peers and negotiates SAs to establish a secure tunnel between channels. Then phase two negotiates authentication and encryption, encrypts the entire packet, and adds it to the payload.

IPsec limitations

While IPsec is popular and widely used, its limitations compared to other security protocols, such as its large overheads and conflicts with network address translation (NAT), can prove challenging. Let’s look at some of these issues in detail.

Packets can exceed router limits

Because of the overhead associated with adding new protocol headers and packet encapsulation, IPsec packets can often become larger than the maximum transmission unit (MTU) — the top limit for the size of a packet supported by the local network. When this occurs, the package is fragmented or broken down into packets that are smaller than or equal to the allowable size.

If one fragment is lost, all fragments must be resent, and since only the first fragment contains the header, the firewall or router may discard the other fragments because they can't be validated. Fragmentation can also result in packets being delivered in the incorrect order, resulting in them having to be retransmitted.

Issues with Network Address Translation (NAT)

NAT maps an IP address space into another by changing information in packet headers while in transit through a router. This can cause problems with the IPsec protocol:

Changed IP identifiers causing dropped or discarded packets: When using methods such as preshared key authentication, IPsec IKE uses IP addresses as identifiers and IKE embeds the IP address of the source device into its payload. When a packet goes through a NAT device, NAT changes the source IP address in the packet to the IP address of the NAT device. Since this address does not match the source address, the receiving device will drop the packet, causing all packets to be resent.
Failed integrity checks and invalid checksums: The IPsec AH protocol uses an integrity check value (ICV) on immutable IP header fields such as the source and destination address. These fields are also used to calculate TCP and UDP checksums for verifying packets. In typical NAT communication, the NAT device replaces the source IP address and port with its IP address and port. These changes can cause failed integrity checks and invalid checksums.

Now that we've discussed some of the challenges with using IPsec to secure your communications, you may be wondering if there is a suitable alternative. WireGuard® is an open source, modern VPN protocol that's simple to use and easy to implement. Let's do a quick comparison.

IPsec vs. WireGuard

IPsec and WireGuard are both VPN protocols used to connect remote networks and access private infrastructure. You can read more in our in-depth comparison of IPsec and WireGuard on platform availability, user experience, and security, but for the purposes of this article, we'll focus on a brief comparison of key benefits of the two protocols.

Advantages of IPsec

IPsec is commonly used for secure communication on corporate VPNs for several reasons:

Wide adoption: IPsec is widely adopted with native clients on all major operating systems — i.e., Windows, Linux, iOS, and Android. Supported IPsec implementations are also available for major router manufacturers including Cisco and Juniper, as well as IoT devices.
Ease of use and familiarity: IPsec connection management approach is familiar to users and easy to implement.
Compatibility with legacy systems: IPsec offers many encryption options, including older encryption methods such as the RSA algorithm and preshared keys that allow you to add a legacy system to an existing IPsec VPN. However, these legacy encryption methods can be more complicated to set up and can create security vulnerabilities in the VPN.

Advantages of WireGuard

WireGuard is a relatively new protocol that's fast becoming popular in the VPN industry because of its high-speed performance and advanced encryption methods. Here are some of its best features:

Smaller, more secure code base: IPsec includes legacy protocols resulting in a very large code base that has not been formally verified. WireGuard's code base is smaller and simpler, with little legacy functionality, making it easier for the open source community to audit for security. WireGuard's codebase has also been formally verified and documented in a cryptographic analysis.
Faster than IPsec: WireGuard is a Linux kernel module, which improves performance compared to a process running in user space. WireGuard has conducted performance benchmarks that show about a 15% higher throughput compared to IPsec.
Built with firewalls in mind: For IPsec to work with a firewall, you need to open UDP port 500 for ISAKMP traffic and allow IP protocols 50 and 51 on firewall filters for ESP and AH traffic, respectively. WireGuard doesn't need open ports. It uses the UDP protocol and can be configured to use any port.
Wide platform support: There are Wireguard implementations available for the major operating systems. Note, however, that the protocol is not yet natively supported on Android or iOS; you'll need WireGuard iOS and Android apps.
Reduced likelihood of misconfiguration: While WireGuard has fewer encryption options than IPsec, it uses more modern encryption solutions with more secure default configurations. Clients and servers cannot specify insecure encryption options, ensuring that end users are all on a recent encryption standard.

Emerging Trends and Applications

IPsec is widely used in various applications, including:

Virtual Private Networks (VPNs): IPsec is used to establish secure tunnels between two devices over the internet.
Network Security: IPsec is used to protect network data from unauthorized access.
Cloud Security: IPsec is used to protect data in cloud environments.
Internet of Things (IoT): IPsec is used to protect IoT devices from unauthorized access.

Emerging trends in IPsec include:

Increased use of IPsec in cloud environments: As more businesses move to the cloud, the use of IPsec to secure cloud-based data and applications is on the rise.
Increased use of IPsec in IoT devices: With the proliferation of IoT devices, IPsec is being increasingly used to secure communications between these devices and other network components.
Development of new IPsec protocols and algorithms: Ongoing research and development are leading to the creation of new IPsec protocols and algorithms that offer improved security and performance.
Increased focus on IPsec security and performance: As cyber threats continue to evolve, there is a growing emphasis on enhancing the security and performance of IPsec implementations.

How Tailscale can help

Though a commonly used VPN protocol, IPsec has several drawbacks, including packet fragmentation, failed data integrity checks, and the propensity to create security vulnerabilities when incorrectly configured.

As the most advanced WireGuard VPN on the market, Tailscale allows you to create encrypted point-to-point connections for remote access to your devices and applications from anywhere in the world.

Tailscale is easy to configure, use, and audit, and is suitable for business environments of any size.

Download Tailscale for free to get started.