Published 2022-07-08.
Last modified 2023-03-02.
Time to read: 4 minutes.
posts collection, categorized under Internet, SSL, Security, Ubuntu, nginx.
Recently I migrated ScalaCourses.com from AWS EC2/S3/CloudFront to a server in my apartment, which has fiber optic internet service. The server’s motherboard is an ASUS Sabertooth x79 with an Intel i7 4820, 32 GB DDR3 RAM, and a 4TB SATA SSD. This motherboard is unable to boot from NVMe drives, so SATA is necessary. At the time of this writing, the server runs Ubuntu 22.04.
The backup server is currently being set up. I am repurposing an old Hackintosh as a fallback to the Ubuntu server. That motherboard is another ASUS Sabertooth x79, with 32 GB DDR3 RAM and two 2 TB SATA drives.
Why Am I Doing This?
Once again, I control my hardware, my software, my network, and all ancillary services. After several years of using PaaS vendor servers, I am now reverting to running ScalaCourses.com on my own hardware and software, using my own network.
Definitions
A proxy is a person or process serving as an authorized agent or substitute for another.
In computer science, a more specific term is forward proxy.
A proxy server is a server process that acts as an intermediary between a client requesting a resource,
and the process that provides the resource.
A reverse proxy is a process that sits in front of other processes, and forwards client requests to them.
The term forwarding process is similar to reverse proxy.
The definitions for proxy, forward proxy and reverse proxy all sound identical.
The key difference between a reverse proxy and a forward proxy
is that a forward proxy enables computers isolated on a private network to connect to the public internet,
while a reverse proxy enables computers on the internet to access a private subnet.
Dealing With Details
The old Pound v2.8-2 reverse proxy that was the front end for the old Play Framework app that runs ScalaCourses.com is no longer viable, and the new version 3 of Pound is incomplete. Depending on configuration, reverse proxies can provide extra security from external attacks on a website, decrypt https requests to http, and act as stream editors for the content.
This site still uses AWS CloudFront, for the moment. I am researching alternatives, with the goal of closing my AWS account... unless they introduce a way to cap financial liability before I fully migrate off AWS.
Apache httpd vs. Nginx
Two popular reverse proxies are
Apache mod_proxy_http2 and
nginx.
Anything Pound can do, both nginx and Apache httpd/2 can do.
While they both work well, Apache httpd has an older code base,
in fact, many of my websites ran on Apache httpd at the turn of the century.
Nginx is relatively newer than Apache httpd. It is performant, well supported, well documented, and widely available. I decided to use nginx as the reverse proxy because I had found that nginx worked well on previous projects.
Installing nginx and the Letsencrypt software is easy on Debian distros such as Ubuntu:
$ yes | sudo apt install nginx certbot python3-certbot-nginx
Verifying the Nginx Build
When acting as a proxy server, nginx requires the http_sub_module to translate HTTP content.
This allows the website content to be stream edited,
so various links and paths that are not translated properly can be fixed up.
The nginx package provided by Ubuntu includes the
‑‑with‑http_sub_module option.
You can verify that your instance of nginx was built with the
‑‑with‑http_sub_module option as follows:
$ nginx -V nginx version: nginx/1.18.0 (Ubuntu) built with OpenSSL 3.0.2 15 Mar 2022 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-9P0wNJ/nginx-1.18.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --add-dynamic-module=/build/nginx-9P0wNJ/nginx-1.18.0/debian/modules/http-geoip2 --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module
Nginx SSL Configuration
Two files are needed for Letsencrypt to be able to create SSL certificates for nginx:
-
The contents of
/etc/letsencrypt/options-ssl-nginx.confneed to be stored into/etc/letsencrypt/options-ssl-nginx.conf. I later discovered this file was also available locally at/usr/lib/python3/dist-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf. -
The contents of
/etc/letsencrypt/ssl-dhparams.pemneed to be stored into/etc/letsencrypt/ssl-dhparams.pem. I later discovered that this file was also available locally at/usr/lib/python3/dist-packages/certbot/ssl-dhparams.pem.
Making a Wildcard SSL Certificate
I knew an SSL wildcard certificate would be needed, so I made one using Letsencrypt. Please read Creating and Renewing Letsencrypt Wildcard SSL Certificates for details.
Update – 8 months after writing this article, I wrote about a better way to generate wildcard SSL certificates: Wildcard SSL Certificates for Let's Encrypt with Lego.
Defining the Nginx Website Reverse Proxy
Please see my article entitled Cross-Origin Resource Sharing (CORS) for a discussion of how to configure servers whose content needs to be proxied.
I saved the following configuration in a new file called
/etc/nginx/sites-available/scalacourses.com.
Note that a single server block answers on ports 80 and 443,
using IPv4 and IPv6, for SSL and non-SSL requests,
for the apex domain (scalacourses.com) and the www.scalacourses.com subdomain.
No redirects are used.
server {
listen 80;
listen [::]:80;
listen 443 ssl;
listen [::]:443 ssl;
server_name scalacourses.com www.scalacourses.com;
ssl_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem;
ssl_certificate_key /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;
ssl_trusted_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/chain.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
root /var/www/html;
index index.html; # This gets served if the proxied website is down
location / {
proxy_pass http://localhost:9000/;
proxy_set_header Accept-Encoding ""; # sub_filter requires this
sub_filter 'https://localhost:9000/authenticate/' 'https://www.scalacourses.com/authenticate/';
sub_filter_once off;
}
}
I disabled the default site:
$ sudo rm /etc/nginx/sites-enabled/default
I enabled the new scalacourses.com site:
$ sudo ln /etc/nginx/sites-{available,enabled}/scalacourses.com
The nginx configuration was tested for syntax:
$ sudo nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
The nginx configuration was reloaded:
$ sudo systemctl reload nginx
Flush DNS Cache
Ensure that DNS requests receive up-to-date values by flushing the DNS cache. The following works on Ubuntu, but not when running on WSL/WSL2.
$ sudo resolvectl flush-caches
The following works on Windows 10:
C:\Users\Mike Slinn> ipconfig /flushdns
Verifying nginx Works
I verified that nginx was listening on ports 80 and 443.
I highlighted the nginx process number –
you might need to scroll the output below to the right to see it.
$ sudo netstat -tulpn | grep ':\(443\|80\)' tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - tcp6 0 0 :::80 :::* LISTEN - tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 87487/nginx: master
The executable for process 87487 can be found by:
$ ls -l /proc/87487/exe lrwxrwxrwx 1 mslinn mslinn 0 Jul 7 09:13 /proc/5166/exe -> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java*
If more detail is desired, get it from the process list. Enclosing a character of the process id within square brackets is an old trick for only showing the desired process.
$ ps aux | grep [8]7487 mslinn 87487 2.4 1.3 6596476 439456 ? Sl 09:13 0:20 java -Xms1024m -Xmx1024m -Dhttp.port=9000 /path/to/jar
If there is any problem getting things to work, it is often helpful to monitor the logs as you click on the web pages:
$ sudo tail -f /var/log/nginx/*.log
Finishing Up
Make nginx start each time the system starts as follows.
$ sudo systemctl enable nginx Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable nginx $ sudo update-rc.d nginx defaults
Performance Test
KeyCDN offfers free website performance statistics. The result column labeled TTFB means “time to first byte”, which is the length of time required for the website content to begin being received by a user's web browser.
I am in Montreal, Canada. Response time for TTFB reported by KeyCDN varies, from a minimum of 68ms in New York, to a maximum of 815ms in Bangalore. This range of response times is typical for websites that have a centralized process. The speed of light is finite, after all.
Free Availability Monitoring
HetrixTools offers free availability monitoring for up to 10 websites. It was quick and easy to set up.
We Are Live!
ScalaCourses.com
now serves Scala students from its newly refurbished server!
😁
