More and more in recent years and never as in these days we speak of Remote Working.

One of the technologies necessary to allow this type of activity is undoubtedly the VPN which together with pfSense constitute an effective solution to the problem.

This guide aims to describe and investigate the differences between OpenVPN vs IPSec implemented through pfSense.

OpenVPN vs IPsec (Security and functionality)

Used Hardware:

This guide can be applied to all hardware certified by us of the firewall line that you can find here:

Software environment:

pfSense® 2.4.x

First some theory

A VPN (Virtual Priate Network) is a “virtual connection” that allows you to create a private network between two or more workstations, which are not located on the same LAN.

This virtual link is called  tunnel.

But what exactly is meant by security in a smart working environment?

It means the security of the communication channel, or of the tunnel.

The tunnel is in fact created through the public network (Internet) and therefore potential attackers could “sniff” our traffic, see our IP addresses of the internal networks, modify our data traffic, etc …

It is clear therefore that we cannot allow all this.

The solution is therefore to use secure VPNs, which allow you to create tunnels that guarantee:

  • Client authentication (remote user)
  • Confidentiality of communication
  • Integrity of the transmitted data: an attacker cannot modify the data without my being aware of it.
  • Replay and Filtering protection: the data must be received exactly in the order in which it is sent.

Let’s now see the most used techniques for “Tunneling” OpenVPN vs IPsec.

  • Tunnel safe with IPSec
  • Tunnel safe with SSL

pfSense and OPNSense implement both solutions.

Secure tunnel with IPSec

IPSec is an architecture that contains multiple protocols to ensure the security of IP OS transmission of the OSI model.
It allows in particular to:

  • create secure VPNs on untrusted networks (public networks)
  • make end-to-end security

IPSec we can define it as a tool with a more complex configuration than other tools to create secure VPNs.
This complexity derives from the fact that IPSec must be configured so that it manages two fundamental parts of communication:

  • I protocols which implement the exchange of symmetric keys in order to encrypt / decrypt the transmitted / received data.
  • The algorithms and modes which allow the actual encryption of the data.

Exchange of keys
Before defining how keys are exchanged, it is important to define the Security Association (SA) concept that underlies the functioning of IPsec.
We can define an SA as a “contract” between the two interlocutors, in which the mechanisms and keys to be used for communication with IPsec are established. The protocol for establishing SAs is called IKE (Internet Key Exchange).
IKE is made up of two phases:

First phase

creation of a first bidirectional SA (SA ISAKMP), which serves to protect subsequent SAs dedicated to actual IPsec communication.
This first phase can be done in two different ways :

  • main mode: heavier processing, but safer.
  • aggressive mode: less heavy, but less secure.

The image below (fig. 1) shows a part of the pfSense interface for configuring the first phase of IPSec. In fact, we note that it is possible to specify the field in which the mode in which this phase will be carried out (Negotiation mode) is described

 OpenVPN vs IPsec

In this first phase, the two interlocutors are authenticated.
The authentication can be done in two ways:

  • Pre shared Key (PSK): a symmetric key shared between the two parties is used. This modality could be rather insecure, as it is not possible to distinguish “who did what”, that is, I cannot distinguish the two counterparts. Furthermore, the fact of having a “shared secret” between the two counterparts could lead to security problems
  • Digital signature (RSA): The digital signature is based on asymmetric encryption, therefore without the use of a shared key, eliminating all the disadvantages of the “shared secret” as in the case of the PSK.
  • With the digital signature we can also perfectly distinguish the two counterparts

Also in the previous image we note the Authentication Method field of the pfSense interface, which specifies how authentication will be performed.

Symmetric data encryption within this first phase can be done with different algorithms; try to prefer “strong” algorithms, such as those of the AES family. Avoid DES or 3DES in a production environment as they are now obsolete and therefore vulnerable.
By defining encryption, you also need to define:

  • di Hash Algorithm for calculating data digest: MD5 not recommended
  • Diffie-Hellman (DH) group: DH is the protocol that allows the exchange of symmetric keys. The group is the number that identifies the “strength” of the key used in the key exchange process. They can be from 1 to 30, going from a minimum (1) to a maximum (30) safety level. The more we use a secure key, the more onerous the calculation that generates it and therefore computationally expensive. A good compromise between security and channel creation speed could be 14.

In the following image (fig. 2) we see how it is possible to specify these encryption characteristics in this first phase for pfSense.
The symmetric encryption algorithm is specified with the Encryption Algorithm field. In our case, AES256-GCM was chosen, where GCM defines the block algorithm and the Key length field is the length of the block of data that will be encrypted at each cycle. The Hash algorithm is specified with the Hash field, while the Diffie-Hellman group is specified by the DH Group field

pfSense: OpenVPN vs IPsec (Security and functionality)

Second phase:

In this phase the real IPsec SAs are negotiated.
These SAs are much faster than the previous ISAKMP SA, as they no longer have to worry about all the preliminary negotiation already carried out by the ISAKMP SA.

Algorithms and modalities
Let’s move on to defining how data is encrypted in transmission and how it is “wrapped” in the IP packet.
IPsec provides two protocols, namely AH and ESP, each of which guarantees different levels of security, particularly:

  • AH (Autentication Header), guarantees
    • Data integrity
    • Sender autentication
    • no Replay
  • ESP (Encapsulating Security Payload), guarantees
    • Data integrity
    • autentication
    • no Replay
    • Data confidentiality

Without going into too much detail, we see that AH does not guarantee data confidentiality.
This means that the data will not be encrypted and therefore a potential attacker could perform a “sniffing” of the data inside the tunnel.
There are two different ways to implement security in IPsec:

  • Transport Mode
  • Tunnel Mode

Transport Mode
This mode requires that IPsec is certified on the final hosts of the communication, that is end-to-end security.

  • benefits
    • computationally light
  • disadvantages
    • IP headers that specify source and destination are always in the clear
    • does not protect variable fields in packages
    • does not protect variable fields in packages
    • configuration of IPsec directly on the final hosts (PC, tablet etc …)

Tunnel Mode
Unlike Transport Mode, in this IPsec mode it attests to the gateways of the network.

  • benefits
    • protection also of variable fields in packages
    • a tunnel is established
  • disadvantages
    • computationally heavy

However, it makes little sense to talk about Tunnel Mode or Transport Mode without the AH or ESP protocol being associated.

To configure these pfSense settings, add the second phase of the Ipsec protocol by clicking on + Add P2.

In the following figure (fig. 3) we see that through the Mode field we can define the Ipsec mode, choosing between Transport Mode and Tunnel Mode.

IPsec e OpenVPN

Continuing in the configuration (fig. 4), we come to the choice between AH and ESP.
We note that the graphical interface changes based on the choice between AH and ESP: if you choose AH, you only have to define the Hash algorithm, through the Hash Algorithms entry.

pfSense: OpenVPN vs IPsec (Security and functionality)

If ESP is chosen, in addition to the hash algorithm, some encryption algorithm must be indicated to be used to guarantee the confidentiality of the data transmitted (fig. 5)

OpenVPN vs IPsec

Let’s now look at two example scenarios.
The first scenario will show an implementation of IPsec in Transport Mode highlighting what an attacker can or cannot see on the public network, based on the protocol used for security (AH or ESP).

The second scenario will be similar, but for an implementation of IPsec in Tunnel Mode.

IPsec e OpenVPN

We see in the image (fig. 6) that if the AH protocol is used, the traffic is not encrypted mail and therefore an attacker can snort the traffic at any point of the communication. Instead, integrity, authentication of the parts and no replays are guaranteed.

With ESP, on the other hand, only the IPs of the final hosts (PC1 and PC2) are shown in clear while all the rest of the sensitive data is encrypted. Like AH, integrity, parts authentication and no replay are guaranteed.

IPsec Tunnel Mode

Psec Tunnel Mode

In this scenario we see (fig. 7) that the complete encryption of the initial packet takes place within the TUNNEL IPsec, using the ESP protocol. In fact, using ESP, not only tunnel IPs are encrypted, while the IPs of the respective hosts (PC1 and PC2) are protected by encryption, together with sensitive data. An attacker who sniffinig on the TUNNEL would only see the Tunnel IPs in the clear.

If we use AH, the traffic would be all clear, as in the case of Transport Mode.

Note that the Tunnel Mode concentrates all the computational encryption load on the gataways and not on the final hosts.
This is a consideration that should not be underestimated in the business environment, as end hosts do not always have enough computing power to implement IPsec safely.

In smart working, the mode used is certainly Tunnel Mode, in which the tunnel is between the remote PC and the corporate gateway. The most common name for this type of VPN is road-warrior, but in academic terms it is called the Secure Gateway.
We see a representation in the image below (fig. 8).

IPsec Secure Gateway.

In recent years, the concept of BYOD, or “Bring your own device“, has spread widely within companies.
In fact, more and more companies allow their collaborators to access the corporate network remotely, through their personal devices (Laptop, Smartphone etc …).
The cost to pay is a non-trivial configuration of IPsec and the need to have sufficient computing power on the remote host and gateway.

However, it is possible to implement a VPN concentrator as a gateway. In fact, this device implements IPsec with hardware dedicated to encryption, thus avoiding slowing down communication.

Secure tunnel with SSL VPN

In the smart working area, it is essential that a remote user be able to access his company network in a short time and therefore be productive.
Without a doubt, the use of an SSL VPN that is easy to configure is right for us, allowing access to the office network even in a few minutes.

An SSL VPN does nothing but take advantage of a secure SSL connection by creating a real tunnel between the parties.

More specifically, what is done is a TUNNEL at the TCP / UDP level (Layer 4), unlike IPsec which creates a TUNNEL (in Tunnel Mode) at the IP level (Layer 3).

An SSL VPN guarantees:

  • Initial authentication of the parties
    • Server authentication
    • Client authentication (remote user)
  • Confidentiality of messages
  • Authentication and integrity of messages
  • Raplay protection

There are 4 different approaches for SSL VPN.

1. Clientless: the remote user authenticates for the VPN directly from a browser web page. Once the connection is established, the VPN is created on the SSL secure channel. Note that authentication of the remote user is a simple login, which could be potentially dangerous. The advantage is that it is totally independent of the remote user’s Operating System (fig. 9).

Clientless VPN

2. Browser plug-in: the VPN vendor provides a plug-in to be loaded on the browser. Authentication will no longer be performed on a web page but managed by the plugin. Also in this case it is totally independent of the Operating System (fig. 10).

SSL VPN Plugin

3. Stand alone executable: with this approach it is essential to install a tool made available by the vendor of the VPN service (for example OpenVPN). In this case, the application installed on the user’s workstation will authenticate.
However, this mode requires that you use a certificate (personal for each remote user) issued by the VPN manager. The certificate will then be used by the application to authenticate the user, all “strengthened” by the user login.
It is therefore clear that in this mode, a certificate must be provided by the company that makes the VPN available. This greatly improves the security levels for remote user authentication, to the detriment of the management and distribution of certificates for each user (fig. 11).

OpenVPN Standalone

4. Mobile app: there are smartphone apps that perform the same function as an SSL VPN application installed on the PC. OpenVPN provides this technology. in this article we show how to install it:

This also allows a smartphone to connect via SSL VPN. Again, a certificate is required for each remote user (fig. 12).

OpenVPN Certificate

Let’s compare these four modes by providing useful ideas for configuration.

In particular, the first two methods (Clientless and Plug-in Browser) do not require authentication of the remote user through a certificate. It is therefore only necessary to decide which encryption algorithms to set on the gateway that implements SSL VPN.
We avoid using weak encryption algorithms such as DES or 3DES and all associated block algorithms (CBC, CFB etc …).
To go into more detail of the encryption algorithms I leave you at the following article: which analyzes the connection speed based on the chosen algorithm.
A commonly used algorithm for encryption is BF-CBC or AES CBC which offer a good compromise between security and speed.

The last two types (Stand alone executable and Mobile app) may require the installation of a certificate on the device used by the remote user in addition to the usual login with username and password. This greatly increases VPN security.
Note that that certificate can be re-used on multiple devices by the same user; this is a big benefit for the scalability of the service (I don’t have to worry about creating a certificate for each device), at the expense of security (I could pass the certificate and access credentials to an unknown user, or a malicious user could take possession).

OpenVPN is one of the most used SSL VPN technologies, as it offers high performance and security, accompanied by ease of implementation (both for the remote user and for the network administrator).

Which one to choose?
One difference that could be determined on the choice of one VPN over another is the granularity with which I can manage access to the network.
Since the IPsec protocol operates at the IP level (Layer 3 OSI), an IPsec VPN would give full access to the corporate network regardless of the application that the remote user uses.
While with the use of SSL VPN that operates at the TCP/UPD level (Layer 4 OSI), it is possible to introduce limitations.

This article explains how to limit access:
In general, there is a tendency to prefer IPsec for site-to-site VPN, while for the access VPN (road warrior), SSL VPN is preferred for greater ease of implementation compared to IPsec.

In conclusion therefore, both SSL VPN solutions that IPsec only perform very well from the point of view of the transmission speed for the same hardware used.
It is up to the network administrator to decide which is the best compromise that the two technologies can offer based on the information and needs of their network infrastructure.

In the smart working scenario it is essential to have a VPN system that allows easy access to the corporate network, without wasting time in complicated user configurations, always guaranteeing security.
Statistically, OpenVPN is the solution that represents the best compromise.