Tag: ubuntu

Software Defined Networking with Linux

Well well well, it’s been a while y’all.

Been busy developing and writing a few things, some more exciting stuff coming up in the pipeline.
A lot of the projects I’m working on have to kind of sort of “plug together” and to do a lot of this I use open-source solutions and a lot of automation.
Today I’d like to show you how to setup a Linux based router, complete with packet forwarding, DHCP, DNS, and dare I even say NTP!

Why and what now?

One of the projects I’m working on requires deployment into a disconnected environment, and it’s a lot of things coming together.  Half a dozen Red Hat products, some CloudBees, and even some GitLab in the mix.  Being disconnected, there needs to be some way to provide routing services.  Some would buy a router such as a Cisco ISR, I in many cases like to deploy a software-based router such as pfSense or Cumulus Linux.  In this environment, there’s a strict need to only deploy Red Hat Enterprise Linux, so that’s what I used and that’s what this guide is based around but it can be used with CentOS with little to no modification, and you can execute the same thing on Debian based system with some minor substitutions.

A router allows packets to be routed around and in and out of the network, DHCP allows other clients to obtain an IP automatically as you would at home, and DNS allows for resolution of URLs such as google.com into which can also be used to resolve hostnames internally.  NTP ensures that everyone is humming along at the same beat.  Your Asus or Nighthawk router and datacenters use Linux to route traffic every day and we’ll be using the same sort of technologies to deliver routing to our disconnected environment.

Today’s use case

Let’s imagine you start with this sort of environment, maybe something like this…

This slideshow requires JavaScript.

What we have here is a 7-node Raspberry Pi 3 B+ cluster!

3 nodes have 2x) 32gb USB drives in them to support a 3-node replica Gluster cluster (it’s fucking magic!).  Then 3 other nodes are a part of a Kubernetes cluster, and the last RPi is the brains of the operation!

In order to get all these nodes talking to each other, we could set static IPs on every node and tell everyone where everyone else is at and call it a day.  In reality, though, no one does that and it’s a pain if not daunting.  So the last Raspberry Pi will offer DHCP, DNS, and NTP to the rest of the Kubernetes and Gluster clusters while also offering service as a wifi bridge and bastion host to the other nodes!  I’ve already got this running on Raspbian and have some workloads operating so I’ve recreated this lab in VirtualBox with a Virtual Internal Network and Red Hat Enterprise Linux.

Step 1 – Configure Linux Router

Before we proceed, let’s go along with the following understandings of your Linux Router machine:

  • Running any modern Linux, RHEL, Cumulus Linux, Raspbian, etc
  • Has two network interface cards, we’ll call them eth0 and eth1:
    • WAN (eth0) – This is where you get the “internet” from.  In the RPi cluster, it’s the wlan0 wifi interface, in my RHEL VM it’s named enp0s3.
    • LAN (eth1) – This is where you connect the switch to that connects to the other nodes, or the virtual network that the VMs live in.  In my RHEL VM it’s named enp0s8.
  • We’ll be using the network on the LAN side (or netmask of for those who don’t speak CIDR), and setting our internal domain to kemo.priv-int

I’m starting with a fresh RHEL VM here, so the first thing I want to do is jump into root and set my hostname for my router, update packages, and install the ones we’ll need.

sudo -i
hostnamectl set-hostname router.kemo.priv-int
yum update -y
yum install firewalld dnsmasq bind-utils

Now that we’ve got everything set up, let’s jump right into configuring the network interface connections.  As I’m sure you all remember from your RHCSA exam prep, we’ll assign a connection to the eth1 interface to set up the static IP of the router on the LAN side and bring it up.  So assuming that your WAN on eth0 is already up (check with nmcli con show) and has a connection via DHCP, let’s make a connection for LAN/eth1 (my enp0s8)…

nmcli con add con-name lanSide-enp0s8 ifname enp0s8 type ethernet ip4 gw4
nmcli con modify lanSide-enp0s8 ipv4.dns

Before we bring up the connection, let’s set up dnsmasq.  dnsmasq will serve as both our DNS and DHCP servers which is really nice!  Go ahead and open /etc/dnsmasq.conf with your favorite editor…

vi /etc/dnsmasq.conf

And add the following lines:

# Bind dnsmasq to only serving on the LAN interface
# Listen on the LAN address assigned to this Linux router machine
# Upstream DNS, we're using Google here
# Never forward plain/short names
# Never forward addresses in the non-routed address space (bogon networks)
# Sets the DHCP range (keep some for static assignments), and the lifespan of the DHCP leases
# The domain to append short requests to, all clients in the subnet have FQDNs based on their hostname
# Add domain name automatically

Annnd go ahead and save that file.

Now, on a RHEL/CentOS 7 machine, we have firewalld enabled by default so let’s make sure to enable those services.

firewall-cmd --add-service=dns --permanent
firewall-cmd --add-service=dhcp --permanent
firewall-cmd --add-service=ntp --permanent
firewall-cmd --reload

Next, we’ll need to tell the Linux kernel to forward packets by modifying the /etc/sysctl.conf file and add the following line:


It might already be in the file but commented out, so simply remove the pound/hashtag in front and that’ll do.  Still, need to enable it though:

echo 1 > /proc/sys/net/ipv4/ip_forward

Yep, almost set so let’s just bring up the network interface connection for eth1, set some iptable NAT masquerading and save it, and enable dnsmasq…

iptables -t nat -A POSTROUTING -o enp0s3 -j MASQUERADE
iptables -A FORWARD -i enp0s3 -o enp0s8 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i enp0s8 -o enp0s3 -j ACCEPT
firewall-cmd --permanent --direct --passthrough ipv4 -t nat -I POSTROUTING -o enp0s3 -j MASQUERADE -s
iptables-save > /etc/iptables.ipv4.nat
nmcli con up lanSide-enp0s8
systemctl enable dnsmasq && systemctl start dnsmasq

Step 2 – Connect Clients & Test

So this part is pretty easy actually, you’ll just need to connect the clients/nodes to the same switch, or make a few other VMs in the same internal network.  Then you can check for DHCP leases with the following command:

tail -f /var/lib/dnsmasq/dnsmasq.leases

And you should see the lease time, MAC address, associated IP, and client hostname listed for each connected client on this routed network!  We should be able to ping all those hostnames now too…

This is great, and we have many of the core components needed by a routed and switched network.  Our use case needs some very special considerations for time synchronization so we’ll use this same Linux router to offer NTP services to the cluster as well!

Step 3 – Add NTP

Here most people would choose to use NTPd which is perfectly fine.  However, RHEL and CentOS (and many other modern Linux distros) come preconfigured with Chronyd which is sort of a newer, better, faster, stronger version of NTPd with some deeper integrations into systemd.  So today I’ll be using Chronyd to setup an NTP server on this Linux router.  Chronyd is also a bit better for disconnected environments, too.

Essentially, we just need to modify the /etc/chrony.conf and set the following lines:

stratumweight 0
local stratum 10

After that, enable NTP synchronization and restart with:

timedatectl set-ntp 1
systemctl restart chronyd

And give that a moment to sync and you should have a fully functional network core based on simple Linux and a few packages!

Next Steps

There are a few things that come to mind that you could do in this sort of environment…

  • Create an actual GPS-based NTP Server – Be your own source!
  • Set Static Host/IP Mappings – Make sure you have a section of IPs available that aren’t in the DHCP block to set the static IP reservations to.
  • Create site-to-site VPNs – Tack on a bit of OpenVPN and we could easily create a secure site-to-site VPN to join networks or access other resources!
  • Anything in your router’s web UI – Pretty much every router out there runs some sort of Linux embedded, and they’re all abstracting elements and functions that are primarily built-in and accessible to everyone.  Set up port-forwarding?  No problem. Add UPnP?  Not too hard either.
  • Add PiHole Ad-Blocker – Maybe you’re using a Raspberry Pi as a wireless bridge to connect some hard wired devices on a switch to a wifi network.  Wouldn’t it be nice to block ads for all those connected devices?  You can with PiHole!

Rolling up Let’s Encrypt on Ansible Tower’s UI

The other day someone asked me what I do for fun.

“Fun” really has a few different definitions for me, and I’d say for most people.  It could be entertainment, guttural satisfaction, leisurely adventuring about, or maybe for some slightly compulsive people like me, accomplishing a task.  Something I’m kind of overly compulsive about is proper SSL implementation and PKI.

So this morning I was having LOADS of fun.  My fast just started to kick in with some of the good energy and ‘umph’ so I was feeling great.  Bumping that new Childish summertime banger, really grooving.  I just finished spinning up a new installation of Ansible Tower and logged in.  That’s when the Emperor lost his groove.

I’ve seen the screen plenty of times in the Ansible Tower Workshops and simply, almost reflexively skip past the big warning sign you see when you first log into an Ansible Tower server’s UI.  The big warning sign isn’t too crucial in the large scheme of things, but it really stuck out to me this time.  Maybe because this server is part of a larger permanent infrastructure play, but it really got to me and I HAD to install some proper SSL certificates.

Self-signed SSL Warning

We all know what to do here, click Advanced and yadda-yadda…or shouldn’t we just fix the issue?


So let’s go over two different ways to fix this…


Ansible Tower uses Nginx (pronounced engine-x) as their HTTP server for the Web UI.  It’s not configured ‘normally’ like you’d see in most web hosting scenarios, there’s no site-available, mods-available, etc.  That’s good though because nothing else should really run on this server outside of Ansible Tower so the good guys at Ansible thought it’d be good to just stuff everything in the default nginx.conf file.

The certificate is self-signed and can be easily replaced.  Here are the lines from the nginx.conf file that matter for this scope, starting at line 42 as of today/this version:

# If you have a domain name, this is where to add it
server_name _;
keepalive_timeout 65;

ssl_certificate /etc/tower/tower.cert;
ssl_certificate_key /etc/tower/tower.key;

Method 1 – Let’s Encrypt

This is probably the more prevalent method nowadays.  It’s easy, free, no need to manage anything since ACME takes care of it.  If your Ansible Tower instance faces the publicly routable Internet, this is probably your go-to.  If it’s not able to reach the Let’s Encrypt ACME servers, you won’t be able to use Let’s Encrypt without some tunnel/proxy/cron tomfoolery, or their manual method which incurs extra steps.  Alternatively, skip to Method 2 which is how to install your own certificate from your own CA/PKI.

Remember a few lines up in the configuration snippet where it had a comment “# If you have a domain name, this is where to add it”?  Go ahead and do just that, edit the /etc/nginx/nginx.conf file and replace the underscore (“_”) with your FQDN.  Save, exit.

Go ahead and reload the nginx configuration

# systemctl reload nginx.service

Next, let’s enable the repos we need to install Let’s Encrypt.  Here are some one-liners, some parts will still be interactive (adding the PPA, accepting GPG keys in yum, etc).  Installing a PPA/EPEL and enabling repos where needed, updating, and installing the needed packages.  Slightly interactive prompts.


# add-apt-repository ppa:certbot/certbot && apt-get update && apt-get install python-certbot-nginx -y

Red Hat Enterprise Linux (RHEL)/CentOS in AWS

# rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm && yum -y install yum-utils && yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional && yum update -y && yum -y install python-certbot-nginx

Red Hat Enterprise Linux (RHEL)/CentOS (Normal?)

# rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm && yum -y install yum-utils && yum-config-manager --enable rhel-7-server-extras-rpms rhel-7-server-optional-rpms && yum update -y && yum -y install python-certbot-nginx

And boom, just like that we….are almost there.  One more command and we should be set:

# certbot --nginx

If the server_name variable in your nginx.conf was modified to point to your FQDN, nginx was reloaded, and all packages enabled properly, the Certbot/Let’s Encrypt command should give you the option of selecting “1: tower.example.com” and do so.  Important: Certbot will ask if you want to force all traffic to be HTTPS.  Ansible Tower already has this configuration in place, so just select “1” when asked about forcing HTTPS to skip that configuration change.

Navigate to your Ansible Tower Web UI, and you should have a “Secure” badged site.

Ansible Tower, Secured

Ansible Tower and Let’s Encrypt. That looks so good.


Method 2 – Manually replacing the SSL Certificate with your own

This is really easy to do actually.  All you have to do is place your certificate files on your Ansible Tower server (in /etc/ssh or /etc/certificates for example), and modify the nginx configuration to point to them.  You may recall the lines in the configuration from earlier…

ssl_certificate /etc/tower/tower.cert;
ssl_certificate_key /etc/tower/tower.key;

Yes, there.  All you have to do is replace those two files, or preferably deposit your own and change the configuration to point to the new files.

Now, this only works under one of the following considerations…

  1. Your certificate is from Comodo, VeriSign, etc.  A CA that’s generally in the root zone of most browser’s certificate store.
  2. Your certificate is from a CA that is installed in your browser’s or device’s root CA store.  Typical of enterprises who manage their own PKI and deploy to their endpoints.

Basically, as long as the Certificate Authority (or CA) that signed your replacement certificate is in your CA root zone you should be golden, otherwise, you’ll see the same SSL Warning message displayed since your browser doesn’t recognize the CA’s identity and therefore does not trust anything signed by them.  If there’s an Intermediary CA, make sure to include the full certificate chain to establish the full line of trust.


Gee, that was fun, right?!

Well, I had fun at least.  If not fun, maybe someone needs to secure their Ansible Tower installation(s) and finds this useful and it brings along a sense of accomplishment or relief.

I have half a mind to make this into an Ansible Role…EDIT: Holy crap, I did make this into an Ansible Role. Has a couple neat tricks it does yeah.