May 2018

Setting up remotes.club on an actual VPS

I set up the original remotes.club in an LXC container back in 2014. The full description of that is here https://rasmus.remotes.club/remotes.html. And while interesting to read, it is quite dated now.

The goals (same as before):

Unlike 4 years ago, you can now get a full VPS for $5/month from a number of different providers. I tested some of them here https://toys.lerdorf.com/low-cost-vps-testing. For this project I am using an UpCloud VPS. You can get your own and follow along and set up your own team server like this one by clicking on this link to get a $25 UpCloud credit: https://www.upcloud.com/register/?promo=7QZ77S

Initial Setup

Deploying a new VPS on UpCloud is easy and only takes a couple of minutes. Make sure you upload your public key and use that on the initial deploy. My preferred OS is Debian, so the following instructions will work verbatim on a Debian 9.x VPS. Once it is ready you will see your ip in the UpCloud UI. Just ssh root@<ip> to get started.

Create your personal account, copy key and enable sudo

adduser rasmus
cp --parents .ssh/authorized_keys ~rasmus
chown -R rasmus.rasmus ~rasmus/.ssh
usermod -a -G sudo rasmus

Vim and EtcKeeper

apt-get update
apt-get install vim git git-core
update-alternatives --set editor /usr/bin/vim.basic
git config --global user.name "Rasmus Lerdorf"
git config --global user.email rasmus@...
apt-get install etckeeper

Misc extra stuff

Add new users to the www-data group automatically by adding

EXTRA_GROUPS="www-data"

to /etc/adduser.conf

apt-get install screen tmux zsh tcsh gcc autoconf make ack htop

PHP 7.2 from Ondrej’s Debian repo

I usually build my own, of course, but Ondrej has been doing a great job keeping his PHP 7 repo current so this is an easier way to do it for most people:

apt-get install apt-transport-https lsb-release ca-certificates
wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | tee /etc/apt/sources.list.d/php.list
apt-get update
apt-get install php7.2-fpm php7.2-cli php7.2-dev php7.2-curl php7.2-mysql php7.2-mbstring php7.2-zip \
                php7.2-xml composer

A quick ini file to set limits and common options along with enabling per-user .user.ini so people can change their own settings.

/etc/php/7.2/fpm/php.ini:

[PHP]
memory_limit = 128M
post_max_size = 8M
upload_max_filesize = 8M
max_file_uploads = 20
max_execution_time = 30
max_input_time = 60
default_socket_timeout = 60
file_uploads = On
expose_php = Off
cgi.fix_pathinfo = Off
zend.enable_gc = Off
error_reporting = -1
display_errors = Off
display_startup_errors = Off
log_errors = On
html_errors = Off
auto_globals_jit = On
zend.assertions = -1
user_ini.filename = .user.ini
user_ini.cache_ttl = 300

/etc/php/7.2/fpm/php-fpm.conf:

pid = /run/php/php7.2-fpm.pid
error_log = /var/log/php-fpm.log
events.mechanism = epoll
include = /etc/php/7.2/fpm/pool.d/*.conf

/etc/php/7.2/fpm/pool.d/www.conf:

[www]
user = www-data
group = www-data
listen = /run/php/php7.2-fpm.sock
listen.owner = www-data
listen.group = www-data
pm = static
pm.max_children = 25

The VPS only has 1G of ram to work with, so the buffers and opcache segments are kept relatively small.

MySQL

apt-get install mariadb-server mariadb-client
mysql_secure_installation

nginx

apt-get install nginx/stretch-backports nginx-doc/stretch-backports
gunzip -c /usr/share/doc/nginx-doc/examples/nginx_modsite.gz > /usr/local/bin/nginx_modsite
chmod +x /usr/local/bin/nginx_modsite

A basic top-level /etc/nginx/nginx.conf file with the PHP 7.2 FPM Upstream configured:

user  www-data;
worker_processes  1;
worker_rlimit_nofile 65535;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  8192;
    multi_accept on;
    use epoll;
}


http {
    upstream php {
        server unix:/run/php/php7.2-fpm.sock;
    }
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" "$host"';

    rewrite_log on;
    access_log  /var/log/nginx/access.log  main;

    sendfile       on;
    tcp_nopush     on;

    keepalive_timeout  65;

    proxy_buffers         16 1024k;  # Buffer pool = 16 1M response buffers
    proxy_buffer_size     64k;    # 64k header buffer
    client_body_buffer_size 10k;
    client_header_buffer_size 1k;
    client_max_body_size 200m;
    large_client_header_buffers 2 1k;
    reset_timedout_connection on;

    open_file_cache off;

    gzip  on;
    gzip_proxied any;
    gzip_types text/plain text/xml text/css application/x-javascript;
    gzip_vary on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    gzip_min_length 8192;
    gzip_comp_level 6;

    types_hash_max_size 2048;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

And /etc/nginx/sites-available/remotes with Slackin and certs added after the fact by Certbot (see below)

upstream slackin_server {
    server localhost:3000;
}

ssl_certificate /etc/letsencrypt/live/remotes.club/fullchain.pem; # managed by Certbot
ssl_trusted_certificate /etc/letsencrypt/live/remotes.club/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/remotes.club/privkey.pem; # managed by Certbot

ssl_protocols   TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers    "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:
                ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:
                DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:
                ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:
                ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:
                ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:
                DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:
                AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:
                DES-CBC3-SHA:!DSS";
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
ssl_ecdh_curve secp384r1;
ssl_prefer_server_ciphers on;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains;";
ssl_stapling on;
ssl_stapling_verify on;
ssl_session_cache   shared:SSL:10m;
ssl_session_timeout 10m;

server {
    listen       80;
    listen       443 ssl;
    listen       [::]:80;
    listen       [::]:443 ssl;
    server_name  remotes.club;

    return 301 https://www.remotes.club$request_uri;
}

server {
    listen 80;
    listen [::]:80;
    server_name  server_name  ~^(?<user>.+)\.remotes\.club$;

    location / {
        return 301   https://$host$request_uri;
    }
}

server {
    listen       80;
    listen       443 ssl;
    listen       [::]:80;
    listen       [::]:443 ssl;
    server_name  slack.remotes.club;

    location / {
        proxy_pass http://slackin_server;
    }
}

server {
    listen       443 ssl;
    listen       [::]:443 ssl;
    server_name  server_name  ~^(?<user>.+)\.remotes\.club$;
    root         /home/$user/web;
    access_log   /home/$user/logs/access.log main; 

    location / {
        index     index.html index.htm index.php;
        autoindex on;
    }

    location ~ \.php$ {
        try_files                $uri =404;
        include                  fastcgi_params;
        include fastcgi.conf;
        fastcgi_intercept_errors on;
        fastcgi_pass php;
    }
}

So each user gets has https://.remotes.club pointing to their ~/web directory and separate logs in ~/logs. To rotate those I added this to /etc/logrotate.d/nginx:

/home/*/logs/access.log {
        daily
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 0640 nginx adm
        sharedscripts
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi \
    endscript
        postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript
}

Letsencrypt Wildcard Cert

Install the latest certbot along with the Cloudflare dns challenge plugin:

sudo apt-get install build-essential libssl-dev libffi-dev python-dev python-setuptools libyaml-dev
git clone https://github.com/certbot/certbot
cd certbot
python setup.py install
cd certbot-nginx
python setup.py install
cd ../certbot-dns-cloudflare
python setup.py install

Grab your Cloudflare API key and make a file like this:

dns_cloudflare_email = rasmus@...
dns_cloudflare_api_key = <your_api_key>

Then fetch and install your cert:

certbot --installer nginx -d *.remotes.club -d remotes.club --agree-tos --manual-public-ip-logging-ok \
  --dns-cloudflare --dns-cloudflare-credentials .secret \
  --server https://acme-v02.api.letsencrypt.org/directory

Email forwarding using Postfix

apt-get install postfix postfix-policyd-spf-python

And add a SPF TXT record to our DNS for remotes.club:

@  IN TXT "v=spf1 a -all"

Next we need DKIM. See https://wiki.debian.org/opendkim Most of that worked ok, but I used a unix domain socket instead so /etc/opendkim.conf has:

Socket    local:/var/spool/postfix/var/run/opendkim/opendkim.sock

and /etc/default/opendkim has:

SOCKET=local:/var/spool/postfix/var/run/opendkim/opendkim.sock

and point postfix to it in the /etc/postfix/main.cf file:

# We are running chroot'ed so this is relative to /var/spool/postfix
smtpd_milters = unix:var/run/opendkim/opendkim.sock
non_smtpd_milters = unix:var/run/opendkim/opendkim.sock

I also added an /etc/postfix/dkim/TrustedHosts file as well containing:

127.0.0.1
::1
localhost
209.50.56.120
2605:7380:1000:1310:202e:aff:fe0a:3ccb
remotes.club

And the bottom of the /etc/opendkim.conf file is:

KeyTable                refile:/etc/postfix/dkim/keytable
SigningTable            refile:/etc/postfix/dkim/signingtable
ExternalIgnoreList      refile:/etc/postfix/dkim/TrustedHosts
InternalHosts           refile:/etc/postfix/dkim/TrustedHosts

We can also make use of our wildcard cert to turn on TLS for our SMPTD. In /etc/postfix/main.cf:

smtpd_tls_cert_file=/etc/letsencrypt/live/remotes.club/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/remotes.club/privkey.pem
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

And also, turn on a few restrictions to get rid of some junk right at the SMTP stage:

smtpd_helo_restrictions = reject_unknown_helo_hostname
smtpd_sender_restrictions = reject_unknown_sender_domain
smtpd_relay_restrictions = permit_sasl_authenticated, permit_mynetworks,
                           reject_unauth_destination,
                           check_policy_service unix:private/policy-spf,
                           reject_rbl_client zen.spamhaus.org,
                           reject_rhsbl_reverse_client dbl.spamhaus.org,
                           reject_rhsbl_helo dbl.spamhaus.org,
                           reject_rhsbl_sender dbl.spamhaus.org
smtpd_data_restrictions = reject_unauth_pipelining
policy-spf_time_limit = 3600s

And make sure the bottom of /etc/postfix/master.cf has:

policy-spf  unix  -       n       n       -       -       spawn
     user=nobody argv=/usr/bin/policyd-spf

And finally, to disable local delivery until users set up their forwarding address, I added /etc/skel/.forward containing:

|"echo 'No forwarding address'; exit 67"

Postfix will reply with:

<rasmus@remotes.club>: user unknown. Command output: No forwarding address

If people try to email a user who hasn’t configured a forwarding address.

Done

This doc was created using:

pandoc -s -S -c css/pandoc.css --highlight-style pygments remotes2.md -o remotes2.html

You can see the source markdown here: https://rasmus.remotes.club/remotes2.md