Mike Slinn
Mike Slinn

Using Nginx as a reverse proxy with SSL

Published 2022-07-08. Last modified 2022-08-05.
Time to read: 4 minutes.

This site is categorized under Internet, Security, Ubuntu.

Recently I migrated ScalaCourses.com from AWS EC2/S3/CloudFront to a server in my apartment, which has fiber optic internet service. The server’s motherboard is an ASUS Sabertooth x79 with an Intel i7 4820, 32 GB DDR3 RAM, and a 4TB SATA SSD. At the time of this writing, the server runs Ubuntu 22.04.

The backup server is currently being set up. I am repurposing an old Hackintosh as a fallback to the Ubuntu server. That motherboard is a Gigabyte GA-X58A-UD3R (Rev 2). It also has 32 GB DDR3 RAM and a 2 TB SATA drive.

Why Am I Doing This?

Once again I control my hardware, my software, my network, and all ancillary services. After several years of using PaaS vendor servers, I am now reverting to running ScalaCourses.com on my own hardware and software, using my own network.

No longer will I subject myself to the unlimited financial liability that current PaaS vendors expose their customers to.

Dealing With Details

The old Pound v2.8-2 reverse proxy that was the front end for the old Play Framework app that runs ScalaCourses.com is no longer viable, and the new version 3 of Pound is incomplete. Depending on configuration, reverse proxies can provide extra security from external attacks on a website, decrypt https requests to http, and act as stream editors for the content.

This site still uses AWS CloudFront, for the moment. I am researching alternatives, with the goal of closing my AWS account... unless they introduce a way to cap financial liability before I fully migrate off AWS.

Apache httpd vs. Nginx

Two popular reverse proxies are Apache mod_proxy_http2 and nginx. Anything Pound can do, both nginx and Apache httpd/2 can do. While they both work well, Apache httpd has an older code base, in fact, many of my websites ran on Apache httpd at the turn of the century.

Nginx is relatively newer than Apache httpd. It is performant, well supported, well documented, and widely available. I decided to use nginx as the reverse proxy because I had found that nginx worked well on previous projects.

Installing nginx and the Letsencrypt software is easy on Debian distros such as Ubuntu:

Shell
$ sudo apt install nginx certbot python3-certbot-nginx

Verifying the Nginx Build

When acting as a proxy server, nginx requires the http_sub_module to translate HTTP content. This allows the website content to be stream edited, so various links and paths that are not translated properly can be fixed up.

The nginx package provided by Ubuntu includes the ‑‑with‑http_sub_module option. You can verify that your instance of nginx was built with the ‑‑with‑http_sub_module option as follows:

Shell
$ nginx -V
nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-9P0wNJ/nginx-1.18.0=.
-flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat
-Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions
-flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx
--conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat
--with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module
--with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module
--with-threads --add-dynamic-module=/build/nginx-9P0wNJ/nginx-1.18.0/debian/modules/http-geoip2
--with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module
--with-http_sub_module 

Nginx SSL Configuration

Two files are needed for Letsencrypt to be able to create SSL certificates for nginx:

  1. The contents of /etc/letsencrypt/options-ssl-nginx.conf need to be stored into /etc/letsencrypt/options-ssl-nginx.conf. I later discovered this file was also available locally at /usr/lib/python3/dist-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf.
  2. The contents of /etc/letsencrypt/ssl-dhparams.pem need to be stored into /etc/letsencrypt/ssl-dhparams.pem. I later discovered that this file was also available locally at /usr/lib/python3/dist-packages/certbot/ssl-dhparams.pem.

Making a Wildcard SSL Certificate

I knew an SSL wildcard certificate would be needed, so I made one using Letsencrypt. Please read Creating and Renewing Letsencrypt Wildcard SSL Certificates for details.

Defining the Nginx Website Reverse Proxy

I saved the following configuration in a new file called /etc/nginx/sites-available/scalacourses.com. Note that a single server block answers on ports 80 and 443, using IPv4 and IPv6, for SSL and non-SSL requests, for the apex domain (scalacourses.com) and the www.scalacourses.com subdomain. No redirects are used.

Shell
server {
  listen 80;
  listen [::]:80;
  listen 443 ssl;
  listen [::]:443 ssl;

  server_name scalacourses.com www.scalacourses.com;

  ssl_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem;
  ssl_certificate_key /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  root /var/www/html;
  index index.html;  # This gets served if the proxied website is down

  location / {
    proxy_pass http://localhost:9000/;
    proxy_set_header Accept-Encoding "";  # sub_filter requires this

    sub_filter 'https://localhost:9000/authenticate/' 'https://www.scalacourses.com/authenticate/' ;
    sub_filter_once off;
    sub_filter_types text/html;
  }
}

I disabled the default site:

Shell
$ sudo rm /etc/nginx/sites-enabled/default

I enabled the new scalacourses.com site:

Shell
$ sudo ln /etc/nginx/sites-{available,enabled}/scalacourses.com

The nginx configuration was tested for syntax:

Shell
$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful 

The nginx configuration was reloaded:

Shell
$ sudo systemctl reload nginx

Flush DNS Cache

Ensure that DNS requests receive up-to-date values by flushing the DNS cache. The following works on Ubuntu, but not when running on WSL/WSL2.

Shell
$ sudo resolvectl flush-caches

The following works on Windows 10:

Command and PowerShell consoles
C:\Users\Mike Slinn> ipconfig /flushdns

Verifying nginx Works

I verified that nginx was listening on ports 80 and 443. I highlighted the nginx process number – you might need to scroll the output below to the right to see it.

Shell
$ sudo netstat -tulpn | grep ':\(443\|80\)'
tcp    0    0 0.0.0.0:80       0.0.0.0:*       LISTEN   -
tcp6   0    0 :::80            :::*            LISTEN   -
tcp    0    0 0.0.0.0:443      0.0.0.0:*       LISTEN   87487/nginx: master 

The executable for process 87487 can be found by:

Shell
$ ls -l /proc/87487/exe
lrwxrwxrwx 1 mslinn mslinn 0 Jul  7 09:13 /proc/5166/exe -> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java* 

If more detail is desired, get it from the process list. Enclosing a character of the process id within square brackets is an old trick for only showing the desired process.

Shell
$ ps aux | grep [8]7487
mslinn      87487  2.4  1.3 6596476 439456 ?      Sl   09:13   0:20 java -Xms1024m -Xmx1024m -Dhttp.port=9000 /path/to/jar 

If there is any problem getting things to work, it is often helpful to monitor the logs as you click on the web pages:

Shell
$ sudo tail -f /var/log/nginx/*.log

Finishing Up

Make nginx start each time the system starts as follows.

Shell
$ sudo systemctl enable nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx 

$ sudo update-rc.d nginx defaults

Performance Test

KeyCDN offfers free website performance statistics. The result column labeled TTFB means “time to first byte”, which is the length of time required for the website content to begin being received by a user's web browser.

I am in Montreal, Canada. Response time for TTFB reported by KeyCDN varies, from a minimum of 68ms in New York, to a maximum of 815ms in Bangalore. This range of response times is typical for websites that have a centralized process. The speed of light is finite, after all.

Free Availability Monitoring

HetrixTools offers free availability monitoring for up to 10 websites. It was quick and easy to set up.

We Are Live!

ScalaCourses.com now serves Scala students from its newly refurbished server! 😁