If you’ve ever worked from a coffee shop and accessed your company’s server, or if two bank branches securely share data across the country, you’ve likely benefited from a digital guardian known as IPsec (Internet Protocol Security). It’s the protocol suite that forms the backbone of most enterprise VPNs.
Think of IPsec as the silent, unsung hero of the corporate and government world. While it doesn’t make headlines, this foundational protocol suite works tirelessly behind the scenes to encrypt and authenticate the vast majority of secure online communications.
Sure, newcomers like WireGuard are praised for their speed, but IPsec is the battle-tested veteran. It’s the enterprise-grade standard for security that you can count on, built into the very fabric of the network itself.
This guide will pull back the curtain. We’ll demystify how IPsec works (from its complex cryptographic handshakes to the crucial difference between its two main modes). The article shows you exactly why IPsec remains an indispensable tool for securing the internet in 2025.
What is IPsec (Internet Protocol Security)?
IPsec comes from Internet Protocol Security. It is a suite of protocols that secures internet protocol (IP) communication by authenticating and encrypting each packet. In layman’s terms, it’s a collection of open, consensus-driven standards created to protect information sent over Internet Protocol (IP) networks by the Internet Engineering Task Force (IETF).
IPsec works on the network layer of the OSI model (Layer 3) and is responsible for securing traffic at the IP packet level, thus providing a safe layer of encryption for applications that are on a higher layer.
It offers security focused on three main services:
- Authentication: Provides sender authentication to verify the sender is who they say they are.
- Integrity: It ensures that the data does not change or nobody is able to tamper it.
- Confidentiality (encryption): It prevents unauthorized third parties from viewing the contents of the data.
IPsec secures all network traffic between two endpoints; this could be lot-to-lot connections (router to router), a couple of servers, or, of course, a client to a gateway – and while there are specific security protocols that secure applications like web browsers (HTTPS://) – IPsec is a blanket security experience for networks.
How does IPsec work?
Think of sending a top-secret physical document. IPsec doesn’t just put it in a safe (encryption); it also puts the safe in a tamper-proof crate (authentication), ships it via a trusted courier with a specific route (the tunnel), and verifies the recipient’s identity before handing it over. The following steps detail this digital equivalent.
Key exchange (IKE)
Key Exchange is the first stage and is part of the protocol referred to as the Internet Key Exchange (IKE) protocol. It applies to the concept of agreeing on some cryptographic keys that the two peer devices can use securely without the attacker being able to know those keys.
- IKE Phase 1 (main mode/aggressive mode): The peers are able to authenticate one another. They negotiate and agree upon the initial cryptographic algorithms and supporting security, and use the Diffie-Hellman algorithm to establish an encrypted or secure channel (the Phase 1 Security Association (SA)). Once this Phase 1 SA security channel is established, it will protect further key negotiation.
- IKE Phase 2 (quick mode): Using the secure tunnel they built in Phase 1, the peers now negotiate the final IPsec connection (Phase 2 SA), deciding exactly how they will encrypt and authenticate the data packets.
Note: The process described above is for IKEv1. The modern standard, IKEv2, simplifies and strengthens this process. IKEv2 combines the initial exchanges into a more efficient four-message handshake, making connections faster and more resistant to denial-of-service (DoS) attacks. It also includes built-in support for mobility (MOBIKE), which is why it’s the preferred protocol for VPNs on smartphones that switch between Wi-Fi and cellular data.
For maximum security, Phase 2 (or its equivalent in IKEv2) should be configured with Perfect Forward Secrecy (PFS). When PFS is enabled, a new Diffie-Hellman key exchange is performed for every session. This means that even if an attacker were to compromise the long-term secret key of your VPN gateway, they could not decrypt any previously recorded VPN sessions. PFS is a critical defense against mass surveillance and data harvesting.
Packet headers and trailers
After the establishment of the Security Association (SA), IPsec modifies the original IP packet by adding its headers and trailers.
- Authentication Header (AH): A fixed-size AH header that is added after the IP header.
- The Encapsulating Security Payload (ESP): Adds an ESP header, an ESP trailer, and an ESP Authentication Data field, all of which contain information useful for the cryptographic processing and identification of the Security Association (SA).
Authentication
Authentication of the sender’s origin verifies the sender. The sender can prove their identity by computing a cryptographic hash of key information from the packet (plus a shared secret to produce a hash) and attaching that additional information directly to the packet.
The receiver computes the same value from the received packet, and if both values match, it confirms the sender’s authenticity. IPsec adopts procedures like HMAC-SHA-256 to support credential authentication.
Encryption
This process encrypts the data payload (and potentially the transport header), making it unreadable to anyone who intercepts it.
IPsec utilizes the most commonly used encryption standard, Advanced Encryption Standard (AES), which supports advanced cryptography in a variety of modes. The work of encryption is a responsibility of the Encapsulating Security Payload (a protocol within the IPsec that provides encryption, authentication for IP packets, and data integrity) protocol alone.
Transmission
After securing (encrypting and authenticating) the original packet, the system encapsulates it in IPsec Tunnel Mode or Transport Mode, adds new headers, and sends it over the public network as a regular IP packet.
Decryption
The reverse process will take place immediately the packet arrives at the receiving peer:
- SA lookup: The receiver looks up the correct SA (keys, algorithms) for that particular connection using the Security Parameter Index (SPI) in the IPsec header.
- Verification: The receiver verifies the authentication and integrity of the packet using the shared keys.
- Decapsulation and decryption: After verifying the packet, the receiver decrypts the payload, removes the IPsec headers and trailer, and restores it as a normal packet before sending it to the application layer.
Why is IPsec important?
The importance of IPsec originates from the insecurity built into the Internet Protocol (IP). The original design of IP focused solely on routing efficiency, which made it vulnerable to common attacks such as IP spoofing, packet sniffing, and session hijacking.
Therefore, IPsec addresses these fundamental flaws and is thus necessary for modern networks, such as:
- Trust and verification: It incorporates mandatory authentication so that during transmission, trusted devices can authenticate the source of the data.
- `Secure infrastructure: It is the premier technology for creating Virtual Private Networks (VPNs) to securely connect remote offices or users back to an organization over the public internet. For example, with an IPsec site-to-site VPN explained type of application, this technology is the backbone of a secure corporate infrastructure.
- Protection against snooping: The ESP protocol uses mandatory encryption to ensure that no intermediate device or malicious agent can read the contents of the data packets.
- Standardization: IPsec is an IETF standard and therefore, with its use, security devices from different manufacturers will communicate and interoperate (which is important for establishing complex enterprise networks composed of multi-vendor elements).
What is the port for IPsec?
IPsec isn’t just a simple application layer protocol with one TCP or UDP port like HTTP (80) or HTTPS (443). And because it is a network layer protocol, its components use multiple ports and protocols.
| IPsec component | Protocol type | Port/Protocol number | Purpose |
|---|---|---|---|
| IKE (Key Exchange) | UDP | 500 | Negotiates and creates the Security Association (SA). |
| IKE with NAT-T | UDP | 4500 | Incorporates Network Address Translation (NAT) to avoid any complications in regards to firewall configuration. |
| ESP (Encapsulation) | IP Protocol | 50 | Encapsulates, encrypts, and authenticates data. |
| AH (Authentication) | IP Protocol | 51 | Implements data integrity and origin authentication. |
Pro tip: In a firewall configuration of IPsec VPN, the administrator must open UDP ports 500 and 4500 to allow IKE traffic and for protocols 50 and 51 to pass through the firewall.
What protocols IPsec uses and follows
IPsec is a modular and complex protocol suite and relies on a variety of protocols to perform certain tasks:
- Internet Key Exchange (IKE): This is the primary protocol that sets up the secure channel as well as the protocol that is responsible for negotiation, key management, and the establishment of a Security Association (SA) between the two devices. IKE enables the two peers to agree upon encryption algorithms and share a secret key, without the need to pre-configure each security configuration item manually.
- Authentication Header (AH): This protocol provides connectionless data integrity and data origin authentication for the entire IP packet header information and the payload data. AH accomplishes this by adding a cryptographic hash value to the packet. AH does not provide encryption (i.e., confidentiality).
- Encapsulating Security Payload (ESP): This is the more widely used protocol. ESP provides encryption (i.e., confidentiality) for the data payload, and optional authentication and integrity checks on the payload itself. Given that it provides encryption, ESP is a required protocol for constructing an IPsec tunnel mode VPN.
A Common Partnership: It’s also important to note that IPsec is frequently paired with the Layer 2 Tunneling Protocol (L2TP) to form a complete VPN solution. In this partnership, L2TP is responsible for creating the tunnel, while IPsec handles the crucial encryption and security, making it a very robust combination.
Benefits of IPsec: Security without relying on applications
By far one of the strongest advantages of IPsec is that it provides a strong layer of security at Layer 3 and protects traffic before it reaches any applications.
- Transparent to application level: It is a security technology that operates at Layer 3 (the IP layer); applications and users do not need to be aware that security is being provided at that point. This allows for much easier deployment and management of security measures. Any device or service connected to the network that speaks IP can have security applied using IPsec without adding application-specific patches or changes.
- Built-in OS support: IPsec is built into almost every modern operating system (Windows, Mac, Linux, iOS, Android) and networking hardware. Because it exists in these operating systems, it can operate independently of third-party clients or modified drivers to support remote access and Ipsec site-to-site vpn explained deployment.
- Strong granular security: Due to IPsec providing separate protocols for authentication (AH) and encryption methods (ESP), it gives administrators a high degree of granular control over security policies at the Layer 3 level. You may choose to enforce integrity only, or confidentiality plus integrity, based on the needs of your network.
Drawbacks to using IPsec
Although IPsec does provide unmatched security, it is not without issues and challenges, specifically, in the area of complexity and compatibility with networks:
- Complexity: The IPsec protocol suite is complex. Configuration, deployment, and troubleshooting of IKE Phase 1 and Phase 2 negotiations is a very technical task, particularly with routers and hardware made from several vendors. Such operations can involve long deployment lead times and even cost overruns, given their complexity.
- Firewall and NAT issues: As noted above, IPsec encapsulates multiple protocols and ports (those ports being 50, 51, 500, 4500). For some time, firewalls and Network Address Translation (NAT) have struggled to understand these non-standard protocols. Connection failures used to occur frequently in IPsec on firewalls. NAT-Traversal (NAT-T) addresses some of these issues, but older or poorly configured networks still suffer from NAT issues.
- Performance overhead: Due to heavy encryption, authentication, and encapsulation (in IPsec tunnel mode), each packet adds more and more headers. As a result, unnecessary computing and bandwidth overhead significantly slow down the overall IPsec connection. Lighter and more efficient protocols like WireGuard, however, often avoid this problem. The choice between IPsec, WireGuard, and other protocols depends heavily on your specific needs for security, speed, and ease of use. For a detailed comparison to help you decide, see our guide to VPN protocols.
What is a VPN, and an IPsec VPN?
To better explain an IPsec VPN, here is some background information to clarify.
What is a VPN?
A Virtual Private Network (VPN) is a technology that offers a secure, encrypted connection referred to as a “tunnel” over a less secure network like the public Internet. A VPN creates a private network over a public network so you can share any data available on that private network, securely and privately, over the public network. VPN is a way of describing the service or tunnel.
What is an IPsec VPN?
An IPsec VPN is simply a VPN that utilizes the protocols and protocol suites of the Internet Protocol Security (IPsec) to establish and maintain a secure tunnel. It provides that cryptographic framework that consists of the rules, algorithms, and procedures to decide how data is encrypted, authenticated, and encapsulated into the tunnel.
IPsec is one of the most common robust protocols that support VPNs, especially in the enterprise and government sectors.
How users connect to an IPsec VPN
The connection method depends on the two options for VPN connections: remote-access VPN (single users), or site-to-site – used for office (network-to-network).
A remote-access IPsec VPN (client-to-gateway)
This is the scenario for an individual user connecting from a laptop or phone to a company.
- Client software: The user installs applications of client software (such as the VPN client, or in some cases, the OS built-in function using IKEv2/IPsec) to initiate.
- Authentication: During this stage, the client establishes its identity with the corporate VPN gateway by presenting credentials, a pre-shared key, or a digital certificate.
- Internet Key Exchange (IKE) Negotiation: IKE phase to set up both the client and gateway Security Association (SA).
- Tunnel establishment: Once the SA is established, it encrypts and encapsulates the user’s traffic in standard IPsec tunnel mode. The corporate gateway then receives this traffic, decrypts it, and forwards it into the private network.
IPsec site-to-site VPN (gateway-to-gateway)
This refers to a typical scenario you might find with an Ipsec site-to-site vpn explained in its simplest form. Two fixed corporate locations are connected:
- Dedicated gateways: Each office has dedicated routers or firewalls acting as the VPN endpoints (gateways).
- Persistent connection: The connection is typically always-on, unlike remote access, and automatically re-connects if it fails.
- Transparent operation: The gateways handle all the IPsec negotiations and traffic management, making the secure tunnel completely transparent to the users on both local area networks (LANs). Traffic leaving ‘Office A’ destined for ‘Office B’ is instantly encapsulated and secured by the local gateway.
What is the difference between IPsec tunnel mode and IPsec transport mode?
It can function in one of two modes that indicate the method of encapsulating the packet, and therefore, what is secured in transit:
IPsec tunnel mode
- Encapsulation: The system encrypts and authenticates the entire original IP packet—both the header and its data. It then wraps this secured package in a new IP header.
- Security scope: This mode secures all traffic by hiding the original source and destination. It does this by encapsulating the data with a new, outer IP header that only carries the addresses of the two VPN gateways.
- Use case: Tunnel mode is the mandatory choice for Ipsec site-to-site vpn explained and remote access VPNs, as it is designed to tunnel a private network’s traffic across the public internet securely. It’s essential to distinguish this from other anonymity-focused systems like the Tor network. While IPsec tunnel mode hides your original IP from the public internet, it only creates a single, encrypted tunnel to a known gateway (your company). For multi-layered, anonymous routing that bounces your traffic through several volunteer-run nodes, you would need a system like Onion Routing, which is the foundation of the Tor Browser.
IPsec transport mode
- Encapsulation: Only the data payload (and transport layer header) is encrypted and/or authenticated. It doesn’t touch the original IP header.
- Secured scope: This mode secures the data at the content level while retaining visibility of both source and destination IP addresses.
- Use case: Host-to-host communications mostly use Transport mode, where the endpoints are securing their own communication, such as the communication between a server and a client in an internal network. It is not a solution for tunneling private networks over the internet.
What is the effect of using IPsec on MSS and MTU?
Adding the protocol headers and trailers of the IPsec protocols adds to two important network parameters: Maximum Transmission Unit (MTU) and Maximum Segment Size (MSS).
- MTU (maximum transmission unit): The Maximum Transmission Unit (MTU) is the largest packet size (in bytes) that can be sent over a network interface without requiring fragmentation. The ordinary maximum on the internet is 1500 bytes.
- MSS (maximum segment size): The maximum amount of data a TCP segment can contain, which refers to MTU minus IP Header size and TCP Header size (typically 1460).
- The IPsec Effect: When using IPsec (especially IPsec tunnel mode), there are extra protocol headers for IKE, ESP, and the new external IP header. The added protocol headers add to the total packet size. If the original packet is close to 1500 bytes MTU, then adding the IPsec headers will push the overall packet size over the MTU and cause fragmentation.
Fragmentation can slow network performance and may even be blocked by firewalls. To prevent this, IPsec VPNs adjust the Maximum Segment Size (MSS) to a slightly lower value—a process known as MSS Clamping—to ensure smooth and efficient performance.
What does the future look like for IPsec?
Despite the introduction of newer and simpler, and faster alternative protocols, including WireGuard and TLS-based VPNs (OpenVPN), IPsec has not disappeared, and its strong integration into operating systems and networking hardware will further cement the protocol as relevant, especially for certain governmental or enterprise purposes.
The future of IPsec lies in specialization and refinement:
- Specialization in site-to-site: IPsec is likely to be the most suitable option for large and static, well-controlled environments—such as governmental networks, financial networks, or the linking of internal corporate campuses.
- Dominance of IKEv2: The new version of IPsec, IKEv2 (Internet Key Exchange Version 2), has better speeds, improved stability, and better support for mobile connections (which is why you often hear it promoted on smartphones). IKEv2 is now the face of modern IPsec.
- Algorithm agnosticism: The modular design of IPsec makes it highly adaptable, allowing it to evolve alongside emerging technologies. By supporting modern encryption and hashing algorithms such as AES-GCM and ChaCha20, IPsec ensures continued cryptographic strength and relevance. This flexibility allows IPsec to seamlessly integrate newer, more secure standards as older algorithms are phased out, ensuring robust protection for data in transit.
FAQs
A Security Association (SA) formally defines how two devices, known as peers, communicate securely. It outlines key details such as which encryption and authentication algorithms they use, the cryptographic keys for the session, and how long those keys remain valid. In simpler terms, the SA acts as a contract that both sides must follow to maintain a secure exchange of data. However, the devices cannot start exchanging data until they successfully establish this Security Association through the Internet Key Exchange (IKE) protocol.
The answer is no. IPsec is a suite of IPsec protocols. The Authentication Header (AH) protocol guarantees integrity and authentication, but does not guarantee encryption. The Encapsulating Security Payload (ESP) guarantees encryption (confidentiality), integrity, and authentication.
IPsec comes prebuilt into most commercial networking devices and operating systems, making it highly efficient for stable, predictable, and long-term connections. For instance, a corporate use is to create a secure line between two corporate offices (Ipsec site-to-site vpn explained). Most end-user / consumer type systems can reliably use an easier and simpler configuration and have an acceptable level of performance and a lower overhead than IPsec, with protocols like WireGuard or OpenVPN.
The major difference is where the two protocols operate. IPsec operates at the Network Layer (Layer 3) by providing security to the IP traffic itself. SSL/TLS operates at the Transport/Application layer (Layer 4/5) and secures data in transit between an application (i.e., browser) and server. While IPsec is a security approach that provides some level of security to any data if it is leaving the host, SSL/TLS would only provide security to the traffic that is leaving the application.
