Want to setup a Remote SPAN port but you don't have specialized hardware to do so? This article will walk you through how to do exactly that using pfSense and Security Onion 2.4 to analyze the network for intrusion detection and security monitoring.

Today we will continue our experiments with pfSense and SPAN options just like in the last blog (SPAN and Management on a Single Physical Port), but this time using OpenVPN and bridge interfaces to achieve real remote cross-network traffic mirroring. The motive for this experiment stemmed from a situation I came across in an environment where the underlying network infrastructure does not allow sending or receiving network data frames with manipulated source and/or destination addresses (Virtual VMware environment where Promiscuous mode and MAC changes are restricted). So I figured why not tunnel that traffic through a P2P layer 2 tunneling solution? GRE, L2TP, and OpenVPN were considered. Settled on OpenVPN tap mode as a final solution.

Components Used:

Virtualized pfSense firewall.

Virualized Securtiy Onion 2.4.

Objective:

• Achieve (E)RSPAN-like feature without any dedicated hardware.

Let's first take an overview at our setup:

We can achieve this setup using a tunneling solution that supports carrying Ethernet over TCP/UDP (OpenVPN in tap mode in our case), together with software SPAN on pfSense which is handled by ifconfig utility in the backend. We can do traffic mirroring and specify the tunnel interface as the destination. The other end of the tunnel will receive the mirrored traffic as if it is connected directly to a SPAN.

Let's take a look on the network diagram to understand the network flow;

All traffic hitting pfSense on the eth1 interface will be mirrored to the OpenVPN tunnel interface, and because we are using tap mode, the other end of the tunnel (at tap5) will be able to see any (RAW) traffic crossing eth1 on pfSense.

Now let's get into the details on how to do it step by step. The first thing would be to setup the tunnel and make sure that it is operable.

Let's follow the following steps on pfSense.

• Create a new OpenVPN server with the following configurations;

Server mode: Peer to Peer (SSL/TLS)

Device mode: tap - Layer 2 Tap Mode

IPv4 Tunnel Network: a single /32 address (e.g 192.168.224.31/32)

Custom options:

tun-mtu 1560;
mssfix 0;
reneg-sec 0;

Make sure to alter the tun-mtu value based on the parent interface max value minus the tunnel overhead.

These are the important settings to note. Complete the other settings as usually done when creating a server endpoint. You might also like to completely disable the tunnel's data channel encryption to reduce the unnecessary computation.

Now let's set up the client endpoint.

Obviously the first thing is to install OpenVPN package if it is not installed;

yum install openvpn
# apt install openvpn
# dnf install openvpn

Then create a client openvpn config file, below is a sample config for our scenario;

dev tap4
client
remote REMOTE_ADDRESS PORT
proto udp
tls-client
nobind
cipher none
tun-mtu 1560
auth none
auth-retry nointeract
persist-key
reneg-sec 0
persist-tun
script-security 2

# Script to run after the tunnel is up
up "/usr/share/scripts/ifenslave.sh"
ifconfig 192.168.224.32 255.255.255.255
route-nopull
fragment 0
mssfix 0

<ca>
-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----

-----END PRIVATE KEY-----
</key>
key-direction 1

In my use case, I am using Security Onion 2.4 as the client endpoint, and in order to tell the system that the monitor interface will be the tap4 interface, we use the command:

/sbin/ifenslave bond0 tap4

And to do that automatically whenever the openvpn tunnel is started, we specify the 'up' option in the openvpn config file and point to the script that will run the command;

> cat /usr/share/scripts/ifenslave.sh
#!/bin/bash

/sbin/ifenslave bond0 tap4

Once the tunnel is up and we verified that it is functioning correctly, we can then proceed and start mirroring packets to the tunnel interface using pfSense bridge interface (ifconfig), but first we need to define the tunnel interface on pfSense;

Interfaces  > Assignment > Interface Assignments

And add the the OpenVPN server instance as an interface, enable and save.

Then we add a new bridge interface in order to create a virtual SPAN port;

Interfaces  > Assignment > Bridges > Add

We specify only one interface on the member list (the interface to be mirrored), and specify the tunnel interface as the SPAN port.

After all is done; we should start receiving mirrored traffic on the other end of the tunnel and start analyzing packets.

> tcpdump -i tap4

Recap

We were able to achieve (E)RSPAN-like feature with only virtualized systems without any need for dedicated hardware.

This is Based on a research done by the Leviathan Security Group https://www.leviathansecurity.com/blog/tunnelvision

For an introduction and details on this security concern, please refer to the linked research above.

As outlined in the original research, most operating systems, including iOS, are vulnerable to the TunnelVision bug.

To exploit this flaw, a DHCP server capable of serving DHCP option 121 (Classless Static Route Option) is required.

Upon testing on an iPhone using a DHCP server that serves the routes 0.0.0.0/1 and 128.0.0.0/1, and subsequently connecting the iPhone to a privacy VPN, internet connectivity appears to be lost.

After numerous attempts, it was discovered that iOS detects any route that starts with 0.0.0.0 and treats it in a special way, resulting in a conflict when a VPN service attempts to add a default route to the iOS routing table.

Therefore, to render the VPN switch for iPhone users useless (and misleading), it is necessary to avoid using 0.0.0.0 and instead have a DHCP server serve the following routes:

• 1.0.0.0/8 VIA ROUTER
• 2.0.0.0/7 VIA ROUTER
• 4.0.0.0/6 VIA ROUTER
• 8.0.0.0/5 VIA ROUTER
• 16.0.0.0/4 VIA ROUTER
• 32.0.0.0/3 VIA ROUTER
• 64.0.0.0/2 VIA ROUTER
• 128.0.0.0/1 VIA ROUTER

In my case, I was using pfSense's DHCP server, and I needed to pass the routes as a HEX string. This can be achieved using the following HEX:

08:01:c0:a8:b4:01:07:02:c0:a8:b4:01:06:04:c0:a8:b4:01:05:08:c0:a8:b4:01:04:10:c0:a8:b4:01:03:20:c0:a8:b4:01:02:40:c0:a8:b4:01:01:80:c0:a8:b4:01

Mitigation

Switch to Android :)

There is really not much you can do about this flaw other than being cautious when connecting to untrusted WiFi networks and do the following when connecting to a VPN:

• Check routes using HE Tools app.
• Traceroute after connecting to a VPN to check packets path.
• If the goal of the VPN is to change the public IP address, then check IP before and after connceting to the VPN.

Ventoy is a great and handy bootloader that supports booting multiple file types including VHD(x).

However, I could not get it to work for Bitlocker encrypted VHDXs, if the disk is encrypted, the windows boot manager complains about not finding winload.exe.

So the workaround here is to do native VHDX boot from the Windows Boot Manager that is directly booted from UEFI without using Ventoy. In essence, when the UEFI Firmware starts, we have two entries, one to boot into Ventoy, and the other to boot the windows bootloader which can handle encrypted VHDX properly.

However, the original EFI partition that Ventoy creates is too small to fit the windows EFI files and we can not control its size.

What we can do instead is redesign the partition layout and create our own EFI system partition. When we first install Ventoy, we can instruct it to preserve some space at the end of the disk.

Then we need to create an EFI System partition which will hold the windows boot files. Right-click on the unallocated disk space > New Simple Volume. We need to format the partition as a FAT32 filesystem.

(Important) We now need to mount the VHDx file so we can later tell bcdboot where the windows directory is to copy the files to the EFI partition in the next step.

Now in a command prompt:

bcdboot D:\Windows /s E: /f UEFI

"D:\Windows" is the windows directory inside the VHDX file. Substitute for the correct drive letter.

"E:" is the newly created EFI partition that will hold the EFI files. Substitute for the correct drive letter.

Now all is done. We reboot to check the boot configurations. Because we now have two FAT32 partitions that both have valid boot files, by default, the UEFI firmware will try to boot from the first FAT32 partition, which is the Ventoy partition. We can add a second custom boot option using the second FAT32 partition from the UEFI boot configurations. Every firmware is different on how to do this, but most do have some way of adding a custom UEFI boot option.

Success! We can now boot bitlocker encrypted VHDXs and have ventoy installed on a single disk.

Notes:

• The FAT32 partition that will hold the windows boot files has to be a primary partition not a logical partition, otherwise booting windows will fail.
• This should only work on UEFI boot environment, legacy bios has not been tested.

Let's say you have a high resolution monitor and want to make use of it under VMware Workstation VMs. One way is to add multiple virtual monitors to a VM and divide the main screen to multiple parts with custom resolution.

For example, this is how the VM looks before:

and this is how it looks after:

All of this under a Virtual Machine.

So to do this, open a command prompt on the virtual machine and change directory to "%ProgramFiles%\VMware\VMware Tools"

cd %ProgramFiles%\VMware\VMware Tools

Then specify the number of virtual monitors and set custom resolution for each one with the following command:

VMwareResolutionSet.exe 0 3 , 0 0 1650 1931 , 1650 0 1677 966 , 1650 966 1677 965

The detailed syntax can be found on VMware website:

https://kb.vmware.com/s/article/2058577

After setting the resolution, you also need to tell vmware workstation to stop auto-fitting the vm window so you don't lose your resolution settings.

TL;DR

Use software port mirroring if you don't have an advanced network switch, one example is ifconfig utility in UNIX like systems. PfSense FTW.

Introduction

I was trying to setup and evaluate Security Onion v2, I had a spare laptop lying around to test with, but it only has one physical port and a WiFi adapter. Security Onion requires at least 2 Ethernet NICs, one for management and the other for traffic monitoring (RX mode only).

I tried playing with the Wifi adapter under a Windows VM running alongside SO under an ESXi host, but that looked ugly and too complicated. Instead, I considered the following approach:

To Achieve this, we need a layer 2 switch that supports port mirroring and can send mirrored traffic to a specific VLAN while still forwards and receives frames on the parent physical port. I believe this is only supported on very advanced enterprise gear like some high end Cisco switches (some flavor of RSPAN). Even then, additional features need to be supported by the switch like QinQ in order to mirror sub-interfaces.

I only had a Unifi switch that barely supports standard physical port mirroring, so the hardware option is not available.

What about a software port mirror?

It is possible to do port mirroring in software. In FreeBSD, ifconfig package provides an option to mirror a bridge port to another interface. I am going to walk through how to do it in Pfsense, since it uses ifconfig in the backend and it will be easier to setup and configure.

Prerequisite:

- a PfSense box with at least one unused Ethernet port.

First, let's get things clear by looking at our setup:

We need to connect our pfsense box directly to the analyzer device, not through a layer 2 switch.

First, we need to define our VLANs under the physical port eth3, in our case we need two:

Interface>Assignments>VLANs>Add

Then we assign an ip address to the management vlan and enable it . For the SPAN vlan, we only need to enable it, without assigning addresses.

To send mirrored traffic to the SPAN vlan, we first need to create a bridge interface, then add the LAN interface (or the one to be mirrored) to that bridge. The process is as follows:

Interface>Assignments>Bridge>Add, and choose the LAN interface as a member, then toggle 'Display Advanced', and under Span port option, choose the SPAN VLAN that we created earlier.

Last step is to assign and enable the bridge interface that we created, without assigning addresses to it.

If more than one interface needs to be mirrored, each mirrored interface has to be on a different bridge, while all can share the same SPAN VLAN.

Now anything enters or leaves the bridge interface is mirrored to the SPAN VLAN. The pfsense part is done, for the analyzer device end, we need to configure its interfaces.

For my specific setup, I use Security Onion as a VM under ESXi, so all I had to do was configure ESXi management interface and give SO VM two network connections, one management with VID 10, and the other SPAN with VID 20.

Caveat when using this approach

Sub-interfaces of the original interface to be mirrored do not get replicated to the SPAN VLAN. For example, in our setup, if the LAN interface had VLAN interfaces under it, they would not be mirrored to the SPAN VLAN.

To circumvent that, we could use QinQ (802.11ad) which is essentially a second VLAN layer. However, this will require some changes to the existing setup and interface reassignments (reassigning LAN to be under a QinQ interface). In most cases this will only be done in a lab, so if you want to mirror more than one interface, it might be easier to just add multiple bridge interfaces than deal with QinQ complexities.

Summary

The idea is to create virtual vlan tagged network interfaces in the host using Hyper-V virtual switch functionality (provided you don't have Intel or Realtek NIC; see the note at the end of this blog for details), then adding a bridge adapter to the virtual machine with the correspondent vlan interface.

Background

One would assume by convention that using virtual machines is sufficient in terms of isolating what's running inside the VM from the host and the rest of the network, but that's not always the case. VMware Workstation provides three types of network adapters; Host-only, NAT, and Bridged network. Host-only network adapters provides access to the host machine only, NAT adapters translate the VM source ip to the host ip address, while bridging provides an adapter that acts as if it is physically connected to the host NIC.

Now, let's follow what happens when the VM gets compromised in each adapter case. If a Host-only adapter is given to the VM, the host is exposed to the VM directly and nothing else, assuming no host-based protections are in effect, the host could be vulnerable.

In a case of NAT adapter, all traffic originating from the VM destined outside the VM net, is masqueraded under the host ip address, so in addition to having access to the host, the VM has access to whatever the host machine has in the local network.

Bridging the VM adapter directly to the host NIC can also be problematic, the VM will have direct access and discovery of the local devices in the network.

The Better Approach

Since VMware workstation does not provide some sort of vmnet firewall, it is better to offload that function to the physical firewall and have better visibility. This can be done by providing entirely isolated networks using VLANS. VMware Workstation does not support handling of VLANS since they are mostly managed by the host NIC drivers, nevertheless there are workarounds.

The way it is done is by creating virtual interfaces in the host, each corresponds to specific vlan tag. Then bridging that interface to the VM adapter, effectively passing-through that vlan to the VM.

Let's consider the following scenario:

Provided that the host NIC supports trunking, creating a virtual interface that is bound to a specific vlan tag in the host and bridging that interface to the VM, allows better control and isolation hence achieves our goal.

Create VLAN interfaces with Hyper-V VSwitch

First you need to enable Hyper-V Services and Hyper-V Module for Windows PowerShell from Windows features, click OK and restart.

Then open PowerShell with admin privileges and create a new virtual switch

New-VMSwitch -Name "External_network" -NetAdapterName "Ethernet"

Create virtual adapters for each vlan you want to assign.

Add-VMNetworkAdapter -ManagementOS -Name VLAN10 -SwitchName External_network

Then configure the interface you just created to listen for the vlan tag needed

Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName VLAN10 -Access -VlanID 10

Now you have a virtual interface that is configured for a specific vlan. To have access to that vlan inside a virtual machine, simply change the settings from VMware Workstation Virtual Network Editor and have a vmnet that is bridged with the Hyper-V virtual NIC.

If the host machine participation is not needed in a specific vlan interface (maybe for security reasons), you can disable all services and protocols from interface properties except for 'VMware Bridge protocol'.

Caveat when using Hyper-V

Once Hyper-V is enabled, your Windows host actually becomes a VM, so your VMware Workstation VMs becomes nested, and their performance greatly decreases.

Note if you have Intel or Realtek NIC

Intel and Realtek provide utilities for Windows to add virtual adapters with vlan tag support. You can use them instead of the Hyper-V Virtual Switch method.

I was trying to fix a windows machine not being able to activate using a valid KMS server. After setting the KMS server using the command

slmgr /skms <server-name>

and initiating the activation

slmgr /ato

The request times-out and gives the error code 0xC004F074. After digging through google for a solution, I found none.

I then decided to look deeper and fired Wireshark to look at the actual request. I found out that the machine tries to activate to a server with the ip 192.168.1.255 even though I had set it manually with slmgr /skms to a different one.

After spending some time searching through the Registry Editor, I found the rogue key. The SoftwareProtectionPlatform key has the configuration for the Windows activation.

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform

The KeyManagementServiceName string value is the one you set with /skms. However, there were other child keys under SoftwareProtectionPlatform that have these name patterns xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. They also contain a KeyManagementServiceName string value, so deleting them or changing their values to the proper server name solved the issue.

I believe this was caused by the installation of shady KMS activation crapware by whoever setup this machine before.

Background

Ntopng is a great tool for diagnosing and monitoring your network. It is available on pfsense firewall through the builtin package manager. Unfortunately, the pfsense port of  ntopng package which is installed through the gui package manger has been broken for a long time . In the latest pfsense 2.5 release, they updated ntopng to 4.2 which is great, but it contains a lot of bugs, sometimes ntopng keeps restarting on itself, other times it seems very slow, and I personally faced an issue that whenever an ntopng service restart occurs, all the package's config gets wiped out, so any modifications you make, like interface rename or adding alerts endpoints and recipients gets lost on next service restart. Also in the previous version of pfsense, which had ntopng 3.8, geolocation data was not being reported correctly. This all makes it useless to put in production environments.

Maybe the most optimal way to setup ntopng is to separate it from the firewall and use a dedicated box to record and analyze network traffic by using a port mirror. However, sometimes you are in a circumstance where it is not feasible to have a separate machine, or maybe the firewall box that you are using is powerful enough to add an active network monitoring function to it.

Tested On:

pfSense 2.5, ntopng 4.2

TL;DR

The better way to integrate ntopng with pfsense is by installing the ntopng package directly from the command line. These are the needed steps in short:

1- On pfsense 2.5, install ntopng and redis database using the shell

pkg install ntopng redis

2- Enable ntopng and redis services

sysrc redis_enable="YES"
sysrc ntopng_enable="YES"

3- Install Shellcmd package from pfsense gui package manager

4- Add the below entries to Shellcmd settings and save

service redis start
service ntopng start

5- Configure ntopng geolocation by downloading the relevant .mmdb files from MaxMind and place them in the folder "/usr/local/share/ntopng/httpdocs/geoip"

6- Edit the startup script of ntopng to add your custom command arguments located in "/usr/local/etc/rc.d/ntopng"

7- Start redis and ntopng services, or simply reboot pfsense

service start redis
service start ntopng

Details

First of all, you need to decide whether you are comfortable using the official but unstable ntopng development build from the original author's package repository (packages.ntop.org), or otherwise the official stable FreeBSD port maintained by the FreeBSD developer madpilot (pkg.freebsd.org), however, the one in FreeBSD repo sometimes gets too behind in development. Unfortunately the ntop.org does not provide a stable build, only a development snapshot (as the time of this writing).

If you decide to choose the stable build from (pkg.freebsd.org), then by default, pfsense uses (pkg.freebsd.org) so you do not need to add any additional sources. All you need to do is install ntopng and redis. To do that, connect to your pfsense using ssh or use the console and open the shell prompt.

pkg install ntopng redis

If you decide to install the latest but unstable build, then follow the instructions at (packages.ntop.org/FreeBSD), but use FreeBSD 12 if you are on pfsense 2.5 or later.

After installing ntopng and redis, you need to make them run automatically on boot. The way we do this by using a package called Shellcmd. It is available on pfsense package manager. Simply head over to system menu, then package manager and install Shellcmd. Once installed, go to services menu, then Shellcmd and add entries to start ntopng and redis on system boot.

Geolocation

Next, you need to setup geolocation databases if you want ntopng to report geolocation data. Refer to this Github Readme on how and why you need to do this. (https://github.com/ntop/ntopng/blob/dev/doc/README.geolocation.md).

Unfortunately, there is no pre-built geoipupdate package for FreeBSD, so you will have to download and update the databases manually, or you could lookup some scripts online that automate the process of downloading the databases from MaxMind for you. "/usr/local/share/ntopng/httpdocs/geoip" is where to put the downloaded database files.

Startup-Command Options

Now comes the part where you modify the startup script for ntopng to put whatever argument option you need. Edit the file located on "/usr/local/etc/rc.d/ntopng" using any text editor and go to the line that starts with "command_args=" and add the arguments that you need. You can refer to the official ntop docs here: (https://www.ntop.org/guides/ntopng/cli_options.html).

Ntopng is not designed to be used as an aggregator for network traffic data over long periods of time, instead, it is best used for live traffic monitoring. That is why if you want to keep network traffic for data retention, it is best to send flows to other databases that are more efficient for storing over longer periods.

For example, you could instruct ntopng to dump flows to an elasticsearch database using the argument:

--dump-flows 'es;flows;index_name%Y.%m.%d;http://ip_of_es:9200/_bulk;'

Another thing to note is, if you want to enable https for the ntopng access portal, after adding the command option --https-port "[port_number]", ntopng tries to find an ssl certificate but it doesn't find any. The easiest way correct this is to find and rename the provided dummy certificate to .pem file format. The dummy cert is located on "/usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem.dummy".

Finally, once everything is setup correctly, you can start ntopng with

service redis start
service ntopng start

or simply rebooting pfsense should bring those services up.

Now you can head over to http(s)//:ip-of-pfsense:3000 or whatever port you defined in the startup options.

I was trying to set up my Windows machine to create a system disk backup image to a VeraCrypt mounted drive. Windows seems not to like the way VeraCrypt mounts drives, it complains about not finding a proper backup device and giving error code 0x80070001.

A detailed behavior is explained more here: https://sourceforge.net/p/veracrypt/discussion/technical/thread/fb84dd7b/

This might be a driver compatibility issue with VeraCrypt drivers and Windows.

Fortunately, this issue seems to appear only when using the graphical interface of Windows Backup and Restore. When using the backend tool, which is 'wbadmin', the backup process goes smoothly without giving errors.

So the workaround here is to use the command prompt instead. The equivalent default command when creating a system drive image is:

wbadmin start backup -allcritical -backupTarget:X: -quiet

Where "x" is destination drive that will hold the backed-up image.

You can type 'wbadmin /?' to get more info and the syntax.

Similar situation can be found here:

How to back up to a VeraCrypt container using Windows 10 built-in tools?

Background: I am running Windows 10 Pro 1709. My C: drive is encrypted with BitLocker. I have an ASUS RT-AC55U with an 8TB HDD attached to its USB 3.0 port. My goal is to use part of that HDD as an

Super Useruser746633

I've tested this with both, a local encrypted drive, and a network attached storage using SMB that contains an encrypted file container, and it works fine in both cases.

Update:

At the end of the backup process, Windows reports that the volumes were successfully backed-up, but gives the error code "0x8004230f".

This might prevent the image restoration process since Windows considers the backup process a failure.

This is fine for me, since all I need is the .vhdx files that are created for each partition when creating a system image, so I won't be using the Restore Windows Image tool anyway.

I hope someone finds this helpful.

Update 2:

There is a much better way to do this if the only goal is to create a .vhdx disk image of a running system, use Sysinternal disk2vhd utility.

PfSense is a great firewall that can be implemented in different scenarios. With the addition of Virtual Private Cloud (VPC) feature in DigitalOcean, pfsense can be a great gateway and firewall to protect your droplets in a private isolated network.

Unfortunately, DigitalOcean does not provide (yet) a ready to use image for pfsense. There are ways to deploy pfsense on top of an existing FreeBSD installation, but this is not recommended by Netgate. Alternatively, You can import a custom image to use in DigitalOcean, but the import tool does not support importing iso files, it only support these extensions: gz, bz2, vmdk, vhdx, qcow, qcow2, vdi, raw, img, xz.

There is a simpler solution to this, but requires manual upload of the image from your local machine. You can simply download the pfsense iso image from pfsense.org, install it as a vm on a hypervisor (eg. VMware, VirtualBox, Hyper-V), once it finishes installing and reboots, switch it off, then upload the vm disk (vmdk, vdi, etc..) to DigitalOcean. Now you have a ready to set up pfsense image.

Before you create a disk image using a hypervisor, you should at least give it a 2GB disk for installation (no worries, disk size can be increased once deployed in DigitalOcean), you also don't want to give it more than 2GB because you don't want to waste your bandwidth later when you upload.

Now create a droplet from that image and enable VPC or choose an already created VPC instance. There is also some tweaking needed once you create a droplet. First, you need to access the console and follow pfsense setup, define the WAN network which is the first adapter, then make sure that you do not define a LAN network. This step is important because when you don't define a LAN, pfsense places an anti-lockout rule on the WAN interface, which is the interface you will be using to connect to pfsense GUI.

Once you finish the console setup wizard you should be able to access pfsense through the droplet's public ip address. Once you login make sure that the first step is changing the default admin password (I strongly recommend adding the droplet to a DO Cloud Firewall and denying all access to it except from your public ip address during the whole setup process).

Now it's time to define the LAN network, but before you actually do that, you need to add a firewall rule to the WAN interface to allow management, because once you define the LAN, the anti-lockout rule gets removed.

After you add the rule, you can now go to the interfaces tab and assign the new interface which is the VPC. You would want to configure it as a DHCP client because VPCs do not support running static ip addresses without some tweaking and advanced configurations as discussed in the previous blog here.

Another thing you need to watch for is disk space. Running gpart show in the command prompt, you will notice that pfsense uses only part of the disk space. You need to expand the partition, as well as the file system to reclaim all the unused free space. To do this, you need to follow this guide on FreeBSD.org here.

You can start by installing HAProxy, ACME and Snort packages to use pfsense as reverse proxy, load balancer and a layer 7 cloud firewall.

Security concerns with unencrypted traffic in VPCs

For droplets, VPCs are a transparent layer 2 medium. It is true that VPCs are private networks between droplets only you define, it is also true that traffic between nodes in a VPC never leave a data center, but it doesn't mean all your nodes live in the same rack space.

Let's say for example, for easier and simpler configurations, you want to run unencrypted http traffic between web servers and a proxy. This opens up the question of the possibility of someone sniffing traffic at the switching level. Of course I am not implying that someone is violating privacy terms, but it is a security best practices to not let any unencrypted traffic leave a physical link.

To the rescue, pfsense has a great and easy to use OpenVPN server. You could run it on pfsense and configure clients to send all unencrypted traffic through the VPN tunnel.

That being said, this goes into a different category, so I will leave it here, or maybe create a dedicated blog in the future.

On April 7th, Digital Ocean announced the availability of its new feature called Virtual Private Cloud (VPC). It is essentially a local network between your droplets, that let your resources communicate privately in a local network using a different interface without needing to hop to the public internet then back again. Put simply, it’s like a LAN between droplets.

You can enable VPC in any new droplet you create. If you have not defined a VPC before, the setup will create a default one for you when creating a new droplet if you choose to enable VPC.

This seems pretty good feature to use and simplifies deployments instead of manually designing custom tunneling solutions between droplets.

One use case for this feature I can image, is when you want to run all your services behind a proxy. You could easily assign a VPC to to the backend services with the proxy, while exposing only the proxy to the public internet.

However, there are some limitations, one is that VPCs cannot traverse across data centers. That means if you were to create a VPC on a data center, it would only be available for droplets residing on that particular data center. This is kinda obvious, because to be traversable across data centers it would require Layer 3 tunnels and other expensive complexities.

Another limitation is that, according to DO docs, VPCs do not support broadcast. That means you cannot run network protocols that rely on broadcast such as DHCP. However, this is almost true, because ARP does actually run fine on the network (or so I thought). To test that, I fired up two droplets, assigned them the same VPC, ran Wireshark and let them communicate.

I can clearly sea ARP requests and responses as in following screenshot.

Seeing the destination as broadcast in the first frame then seeing a reply to that frame felt weird for me. The documentation clearly states that broadcast and multicast are not supported. More on that later.

This seems all great, but is it actually that seamless?

Well, I encountered a problem connecting two droplets together in the same VPC and the same subnet. Before that, and as I usually do after I create a new droplet, I immediately lock it down to a single ip that is my public ip address for security and management purposes (I do this using the built-in firewall function in DO in the inbound rules section).

Back to the problem, the two droplets refused to establish any sort of connection. I could see the the ARP requests and responses and they look just fine, also the ARP tables of each host looks fine too. So it’s got to be a unicast issue.

After spending nearly an hour pulling my hair out, I found out that it is actually a cloud firewall limitation, in which public and private traffic are both affected by the rules defined in the cloud firewall, not just the public traffic. So if you want to use the firewall, you would have to make sure there are permit rules between the resources you want connect. For example, if your VPC subnet is 10.0.0.0/24, you would want to add a permit rule to the firewall with a destination of 10.0.0.0/24 to allow all resources in that firewall instance to communicate with each other.

After fixing the above issue, everything went smooth and I could connect droplets directly to each other. Then I further decided to test the reliability of the connection by measuring the bandwidth using iperf3.

It seems that this droplet is given a shared 10 GBit connection. I noticed that sometimes the bandwidth drops below 200 MBits/s, however this is certainly sufficient for most use cases.

Getting back to broadcast limitation again, as stated before, VPCs do not support broadcast. However, if that's the case, then how come ARP frames reach other nodes in the network? Well, it seems that some ARP frames are one exception to the rule. Notice that I say some, because the other some actually get dropped by whatever internal firewall DO are implementing. So let's generate some ARP traffic:

Here I use nmap to tell it to scan the the given range and effectively generate ARP request to every scanned ip address.

Here is how it looks on Wireshark on the machine that generated the requests:

And here is how it looks on other machines:

Apparently there is some sort of internal filtering at layer 2 that limits broadcast and multicast.

From experimenting with the situation, I find that:

1. Gratuitous ARP are filtered.

2. Only ARP requests that ask for an ip/mac combination that has already been assigned by the VPC's DHCP server are permitted.

3. Only ARP requests that are generated from an assigned address by the VPC's DHCP server are permitted.

To sum up point 2 and 3; only DHCP assigned addresses will be able to send and receive broadcast frames.

Beside not being able to run your own DHCP server, you cannot assign a static ip address to your machines and be able to connect to other machines on the VPC network, unless you add static ARP entries to each of your hosts.

I think there is still a lot to be explored in this topic. And I might have miss interpreted something during this analysis.

Resources:

https://www.youtube.com/watch?v=nbo5HrmZjXo/

https://www.digitalocean.com/docs/networking/vpc/

https://www.digitalocean.com/docs/networking/firewalls/