Introduction
My website opens3.net is built using Kotlin (Ktor), NGINX running on a FreeBSD droplet.
The micro services now consist of the following (all written in Kotlin and Perl):
- Content Management System
- Weather warning SMS bot (sends weather alerts it parses from BOM)
- Weather warning back-end
- Log file processor and IP to location converter for my Ad Blocking Android app
Most of these projects are Open Source.
The configuration files for this write up are available here.
Overview
My FreeBSD droplet runs on Digital Ocean and has a floating IP address. I have configured two jails. One to run the micro services and another to run the database. At a later date I can export the complete jail and run it on a new server or run it at home without having to mess around with importing and exporting databases.
Configuring Jails
The first step is to configure the jails. I am using ezjail. Reference documentation is here.
# pkg install ezjail
First we need to configure our rc.conf to enable our cloned interfaces. Add this to rc.conf:
cloned_interfaces="lo1 lo2 lo3 lo4"
These interfaces will be brought up at boot. To bring up without a reboot:
# service netif cloneup
The following commands will configure the jails to match the previous diagram.
# ezjail-admin install
# ezjail-admin create jvm 'lo1|10.200.1.1'
# ezjail-admin create database 'lo2|10.200.2.1'
# ezjail-admin start jvm
# ezjail-admin start database
In order to have internet inside the jails you will need to add a name server to /etc/resolv.conf inside of each jail.
# ezjail-admin console jvm
# echo "nameserver 8.8.8.8" > /etc/resolv.conf
Then we need to configure NAT in pf by editing /etc/pf.conf:
# Interfaces
wan="vtnet0"
local="vtnet1"
jail1="lo1"
jail2="lo2"
jail3="lo3"
# Options
set block-policy return
set skip on lo0
set skip on $local
scrub in all
# NAT
nat on $wan from ($jail1:network) to any -> ($wan:0)
nat on $wan from ($jail2:network) to any -> ($wan:0)
nat on $wan from ($jail3:network) to any -> ($wan:0)
# default block
block in all
# out is ok
pass in log quick on { $wan } proto { udp tcp } from any to any port { 22 53 80 443 2211 1194 5000 5030}
# Our microservice ports
pass in log quick on { $jail1 } proto { udp tcp } from any to any port { 3000 8080 7890 }
# MySQL and Redis ports
pass in log quick on { $jail2 } proto { udp tcp } from any to any port { 3306 6379}
pass in log quick on { $jail3 } proto { udp tcp } from any to any port { 3306 8080 }
pass out log quick all keep state
# icmp all good
pass out log inet proto icmp from any to any keep state
pass in log quick inet proto icmp from any to any keep state
In order for NAT to work we need to add the following to our rc.conf:
# Enable Packet Filter
pf_enable="YES"
# Enable Packet Forwarding
gateway_enable="YES"
Start packet forwarding (reboot or enter the following commands):
# sysctl net.inet.ip.forwarding=1
# service start pf
Internet should now work inside both of the jails.
Configuring NGINX
My NGINX configuration is rather complicated as I have the following micro services running:
- Content Management System
- WeatherBot Administration Backend
- GoAccess NGINX web log analyzer
Edit the /usr/local/etc/nginx/nginx.conf file.
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
# Log Format for GoAccess
log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_response_time"';
# Store log file in JVM jail
access_log /usr/jails/jvm/var/log/http-access.log main;
sendfile on;
keepalive_timeout 65;
gzip on;
# OpenS3.net main website
upstream myapp {
server 10.200.1.1:8080;
}
# WeatherBot admin backend
upstream noti {
server 10.200.1.1:3000;
}
# HTTP website. Redirects to HTTPS
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://myapp;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
return 301 https://$host$request_uri;
}
# HTTPS website. Certificate provided by lets encrypt certbot.
server {
listen 443 ssl http2;
server_name localhost;
ssl on;
ssl_certificate /usr/local/etc/letsencrypt/live/opens3.net/fullchain.pem;
ssl_certificate_key /usr/local/etc/letsencrypt/live/opens3.net/privkey.pem;
location / {
proxy_pass http://myapp;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Catch the /ws for GoAccess and route to GoAccess daemon.
location /ws {
proxy_pass http://10.200.1.1:7890;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
# WeatherBot Admin Backend
server {
listen 5000 ssl http2;
server_name localhost;
ssl on;
ssl_certificate /usr/local/etc/letsencrypt/live/opens3.net/fullchain.pem;
ssl_certificate_key /usr/local/etc/letsencrypt/live/opens3.net/privkey.pem;
location / {
proxy_pass http://noti;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
}
Running the GoAccess Web Log Analyzer
To run the GoAccess web log analyzer add the following to the /home/nobody/.goaccessrc file:
time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^"
Run GoAccess using the following command to run it as the nobody user.
# su -m nobody -c 'goaccess -p /home/nobody/.goaccessrc /var/log/http-access.log -o /home/nobody/www/report.html --real-time-html --ws-url=wss://opens3.net:443/ws --daemonize'
Final thoughts
Running processes in a jail gives us security, control and flexibilty. If you have multiple CPUs you can dedicate a CPU to a jail. You can also limit memory usage.
You can migrate jails to another machine, or backup the jail to save its state.
Using top you can view the processes running is a specific jail:
GoAccess is a great little tool, to visit my websites state you can view it here (only works in Google Chrome).
[wpvideo KtUs8yTq]
