How IPsec works, why we need it, and its biggest drawbacks

The IP Security protocol, which includes encryption and authentication technologies, is a common element of VPNs (Virtual Private Networks) running over the internet. This overview examines the pros and cons of IPsec.

ipsecurity protocols network security vpn3

What is IPsec?

IPsec (IP security) is a suite of protocols developed to ensure the integrity, confidentiality and authentication of data communications over an IP network. While the flexibility of the IPsec standards has drawn the interest of the commercial sector, this same flexibility has resulted in the identification of several problems with the protocols because of their complexity. As with other security systems, poor maintenance can easily lead to a critical system failure.

IPsec may be used in three different security domains: virtual private networks, application-level security and routing security. At this time, IPsec is predominately used in VPNs. When used in application-level security or routing security, IPsec is not a complete solution and must be coupled with other security measures to be effective, hindering its deployment in these domains [4].

IPsec operation

IPsec has two modes of operation, transport mode and tunnel mode. When operating in transport mode, the source and destination hosts must directly perform all cryptographic operations. Encrypted data is sent through a single tunnel that is created with L2TP (Layer 2 Tunneling Protocol). Data (ciphertext) is created by the source host and retrieved by the destination host. This mode of operation establishes end-to-end security.

When operating in tunnel mode, special gateways perform cryptographic processing in addition to the source and destination hosts. Here, many tunnels are created in series between gateways, establishing gateway-to-gateway security [6]. When using either of these modes, it's important to provide all gateways with the ability to verify that a packet is real and to authenticate the packet at both ends. Any invalid packets must be dropped [2].

Two types of data packet encodings (DPE) are required in IPsec. These are the authentication header (AH) and the encapsulating security payload (ESP) DPEs. These encodings offer network-level security for the data [4]. The AH provides authenticity and integrity of the packet. The authentication is made available through keyed hash functions, also known as MACs (message authentication codes). This header also prohibits illegal modification and has the option of providing antireplay security. The AH can establish security between multiple hosts, multiple gateways, or multiple hosts and gateways, all implementing AH [7]. The ESP header provides encryption, data encapsulation and data confidentiality. Data confidentiality is made available through symmetric key [3].

During its journey through the various tunnels and gateways, additional headers are added to the packet. On each pass through a gateway, a datagram is wrapped in a new header. Included in this header is the security parameter index (SPI). The SPI specifies the algorithms and keys that were used by the last system to view the packet. The payload is also protected in this system because any change or error in the data will be detected, causing the receiving party to drop the packet. The headers are applied at the beginning of each tunnel and then verified and removed at the end of each tunnel. This method prevents the buildup of unnecessary overhead [1].

An important part of IPsec is the security association (SA). The SA uses the SPI number that is carried in the AH and ESP to indicate which SA was used for the packet. An IP destination address is also included to indicate the endpoint: This may be a firewall, router or end user. A Security Association Database (SAD) is used to store all SAs that are used. A security policy is used by the SAD to indicate what the router should do with the packet. Three examples include dropping the packet altogether, dropping only the SA, or substituting a different SA. All of the security policies in use are stored in a security policy database [2].

Problems with IPsec

In some cases, direct end-to-end communication (i.e., transport mode) isn't possible. The following is a simple example in which H1 and H2 are two hosts on one direct tunnel, with H1 using a firewall referred to as FW1.

"In a large distributed system or inter-domain environment, the diversified regional security policy enforcement can create significant problems for end-to-end communication. In the above example, suppose FW1 needs to examine traffic content for the purpose of intrusion detection and a policy is set up at FW1 to deny all encrypted traffic to enforce its content examination requirement. Yet, H1 and H2 build a direct tunnel without awareness of existence of the firewall and its policy rules. Therefore, all the traffic will be dropped by FW1. The scenario shows that each policy satisfies its corresponding requirement while all policies together can cause conflicts" [5].

One of the biggest drawbacks of IPsec is its complexity. While IPsec's flexibility has contributed to its popularity, it also leads to confusion and has led security experts to state that "IPsec contains too many options and too much flexibility" [8]. Much of IPsec's flexibility and complexity may be attributed to the fact that IPsec was developed via a committee process. Due to the political nature of committees, additional features, options, and flexibility are often added to standards to satisfy various factions of the standardization body [8]. This process stands in stark contrast to the standardization process used in the development of the Advanced Encryption Standard (AES), the replacement for the Data Encryption Standard, which expired in 1998 [11]:

"It is instructive to compare this to the approach taken by NIST [the National Institute of Standards and Technology] for the development of AES. Instead of a committee, NIST organized a contest. Several small groups each created their own proposal, and the process is limited to picking one of them. At the time of writing there has been one stage of elimination, and any one of the five remaining candidates will make a much better standard than any committee could ever have made" [8].

Moreover, much of the documentation for IPsec is complex and confusing. No overview or introduction is provided, and nowhere are the goals of IPsec identified. The user must assemble the pieces and try to make sense of documentation that can be described as difficult to read at best. To illustrate the frustration a user must endure, consider the ISAKMP specifications. These specifications are missing many key explanations, contain numerous errors and contradict themselves in various locations [1, 8].

However, while IPsec may not be perfect, it is considered a significant improvement compared with previously available security protocols. As an example, consider the popular security system Secure Sockets Layer. While SSL is widely deployed in various applications, it is inherently limited in that it is used on the transport/application layer, requiring modifications to any application that wants to include the ability to use SSL. Because IPsec is used in Layer 3, it requires modification only to the operating system rather than to the applications that employ IPsec [3].

Is IPsec too complex and confusing?

IPsec incorporates all of the most commonly employed security services, including authentication, integrity, confidentiality, encryption and nonrepudiation. However, the major drawbacks to IPsec are its complexity and the confusing nature of its associated documentation. In spite of these various drawbacks, IPsec is believed by many to be one of the best security systems available. It is hoped that considerable improvement will be evidenced in future revisions of IPsec and that the problems identified with the architecture will be remedied.


FREE Download: Get the Spring 2019 digital issue of CSO magazine today!