How to exit the matrix

Posted on 19 Sep 2014 - filed under: server, hosting

Update #1 2017-06-09 : Sovereign has been updated quite a bit, most of the information here is now obsolete. A nice addition is LetsEncrypt support which provides free HTTPS certificates. Some information might still be of use.

A long time ago I stumbled upon this (edit 2016-06-09: this website is now down, I’m hosting a mirror of it here) online manual/book/article. Here’s its abstract:

Privacy and anonymity have been eroded to the point of non-existence in recent years. Our personal, private information is stockpiled and sold to the highest bidder like so much inventory at a warehouse. National Security Letters are written to make countless requests for records from our search engines, libraries, and book stores with no court oversight. Email and especially searchable data is practically unprotected from anyone who might ask to have it. All our electronic communications are tapped. Massive governmental data mining schemes are being built to record everything we publish on the web.

It’s a very interesting read, specially the “philosophical” discussion that is the first chapter. The rest is a HOWTO on how to be anonymous online.

This was in a pre-Snowden, people with this kind a concerns were thought to be extremely paranoid, nobody really cared…. And eventually I stopped caring.

So back in 2009 something after reading the afromentioned book, I rented a decicated server and proceeded to installing my own mail server, website, file server, music streaming, the whole deal.

Everything was pip and dandy but the hassle of maintaining it eventually got the best of me… Installing updates, fixing issues configuring iptables, prevent brute force attempts; these were all too much for the little gain I perceived.

But this was before…

Sovereign

As it’s often the case with The Internet, one link leads to another. And this time serendipity lead me to this comment on HN.

The tool mentionned was sovereign, a set of Ansible playbooks that automates the server setup.

From Ansible’s [wikipedia page](https://en.wikipedia.org/wiki/Ansible_(software)):

Ansible is an open-source software platform for configuring and managing computers. It combines multi-node software deployment, ad hoc task execution, and configuration management.[1] It manages nodes over SSH and does not require any additional remote software (except Python 2.4 or later)[2] to be installed on them. Modules work over JSON and standard output and can be written in any programming language. The system uses YAML to express reusable descriptions of systems.[3]

For example the following snippet is extracted from the playbook that sets up UFW:

{% highlight ruby %} - name: Install ufw apt: pkg=ufw state=present - name: Deny everything and enable UFW ufw: state=enabled policy=deny ignore_errors: yes {% endhighlight %}

What you will get

Sovereign lets you customize what roles you want your server to assume, but comes with a truckload of stuff out of the box, check out the list here.

The Setup

I rented a VPS Classic from OVH. Specs are:

  • Debian 7
  • 1 core
  • 2 euros/month
  • 10 GB of disk space
  • 1 GB of RAM
  • 100 Mbps internet
  • No restriction
  • A web based KVM, and overall a pretty good “control panel”

It’s been ~3 weeks now, and I must say it’s a pretty good bang for the buck.

Also get a domain from your favorite registrar, I personally have a good experience with gandi.

Automated Awesomeness

Here’s my experience running Sovereign on my setup, I found that sovereign’s was a bit light on some details, so I’ll do my best to detail each step.

Once you have your VPS installed:

Change the root password

As root, run passwd.

Create the deploy user

That’s the account Ansible will use to setup stuff.

{% highlight sh %} $> useradd deploy $> passwd deploy $> mkdir -p /home/deploy/.ssh $> chmod 700 /home/deploy/.ssh $> nano /home/deploy/.ssh/authorized_keys #paste your public SSH key here $> chmod 400 /home/deploy/.ssh/authorized_keys $> chown deploy:deploy /home/deploy -R {% endhighlight %}

Install sudo if not already installed, then edit your sudoer file using visudo, and add:

{% highlight ruby %} deploy ALL=(ALL) NOPASSWD: ALL {% endhighlight %}

Configure what you want

Edit site.yml to comment out (or not) some services (roles) you don’t need.

And edit vars/user.yml to configure users and password for the services you chose to install, the comments are pretty straightforward.

Following sovereign’s README to find out how to generate hashes for your passwords when needed. Pay attention to the comments in users.yml and the instruction in the README.

Install Ansible

Instructions can be found here

Note that you have to install Ansible on your computer (not the server). You’ll be running Ansible from your computer, which will connect as the deploy user on your server to install your shit.

SSL Certificate

I didn’t feel like paying anything, so I used a self-signed certificate. You can always get a free cert from StartSSL, but it’ll only be for 1 domain. It was not worth the trouble to me.

I followed Sovereign’s how to on generating a self signed certificate with no issues.

Disable IPv6

Call me stupid, but I don’t get IPv6. It just makes shit harder, and this convinced me not to care for now.

For the OVH VPS, here’s a link (french) explaining how to achieve that.

Edit (Un)complicated FireWall Config

Edit roles/common/tasks/ufw.yml to add ignore_errors: yes to the Deny everything and enable UFW task. (cf. patch below)

UFW is in fact running but the command returns a cryptic error, causing Ansible to fail on the task thus exiting before allowing SSH access…

An issue is opened on Github here regarding this.

This is an apparently known bug with UFW on Debian, IIRC it’s related to the iptables version used in Debian 7 (but I can’t find the link anymore).

Not 100% sure the second chunk of the diff is necessary, but I had some issues without these additional allows.

Forgetting to do this might lock you out of your server. You’ve been warned.

{% highlight diff linenos %} diff –git a/roles/common/tasks/ufw.yml b/roles/common/tasks/ufw.yml index 9b50043..c114b9e 100644 — a/roles/common/tasks/ufw.yml +++ b/roles/common/tasks/ufw.yml @@ -7,6 +7,7 @@

  • name: Deny everything and enable UFW ufw: state=enabled policy=deny

  • ignore_errors: yes

  • name: Set firewall rule for DNS ufw: rule=allow port=domain @@ -14,6 +15,12 @@

  • name: Set firewall rule for mosh ufw: rule=allow port=60000:61000 proto=udp

+- name: Set firewall rule for pgsql, and various daemon (mail related) + ufw: rule=allow port=5432 proto=tcp + ufw: rule=allow port=5432 proto=udp + ufw: rule=allow port=10023 proto=tcp #postgrey + ufw: rule=allow port=8891 proto=tcp #opendkim + - name: Set firewall rules for web traffic and SSH ufw: rule=allow port={{ item }} proto=tcp with_items: {% endhighlight %}

Run Ansible

Edit the hosts file and put your domain name in it. This file references the servers on with Ansible will login into to execute the playbooks.

Run: {% highlight sh %} ansible-playbook -i ./hosts site.yml –tags=prosody {% endhighlight %}

Take a nice cup of coffee.. you deserved it (:

If you just want to install one role, you can run:

{% highlight sh %} ## Only install the git server ansible-playbook -i ./hosts site.yml –tags=gitolite {% endhighlight %}

Tags can be found if you peek at the roles/X/main.yml. Or just the the command with an invalid tag, Ansible will print out available tags.

Iptables tweak

I guess there’s something I did wrong, but after enabling UFW (Which should be called Fucking Complicated FireWall instead of its current name), all outgoing connection were failing.

After some googling I had to run:

{% highlight sh %} $> iptables -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT {% endhighlight %}

And everything was fixed. Even after a reboot.. iptables works in mysterious ways..

PostgreSQL

I ran into an issue with pgsql which just wouldn’t start, /etc/init.d/pgsql start had no effect.

To fix it I had to install a specific verison explicitly via APT. Weird thing is, the Ansible playbook explicitly installs the 9.1 version.

But anyway, if you encounter the same run: {% highlight sh %} $> apt-get remove –purge postgresql-9.1 $> apt-get install postgresql-9.1 {% endhighlight %}

That did the trick for me.

Configure your DNS

Zone file

Your zone file should look something like the following snippet. I recommend using only one A record that points to your server’s IP and making the other subdomains point to that record via CNAMEs as it makes changing the server’s IP (if you need to one day) much easier. Of course, edit this so that it reflects the services you actually want to set up

{% highlight ruby linenos %} @ 10800 IN A autoconfig 10800 IN CNAME 25.wf. git 10800 IN CNAME 25.wf. mail 10800 IN CNAME 25.wf. @ 10800 IN MX 10 mail.25.wf. @ 10800 IN TXT “v=spf1 mx -all” default._domainkey.25.wf 10800 IN TXT “v=DKIM1; k=rsa; p=” {% endhighlight %}

Setup SPF & DKIM

Setting up SPF and DKIM minimizes the chances that the email you send get tagged as spam.

The SPF record is pretty straightforward.

For DKIM, on your server cat the contents of /etc/opendkim/keys/<YOUR_DOMAIN>/default.txt, and put this in your zone file, it should look like line 7 of my zone file snippet (above).

You can send an email to check-auth@verifier.port25.com to get a report on your server.

Issues

Postgrey

Postgrey is a neat little tool that rejects incomming emails on their first attempt to be received. So that the sending server needs to sending again later after a pre-defined timeout. Spammers usually don’t try that hard (:

Unfortunately on Debian 7 it fails with Insecure dependency. To fix the issue apply this patch and edit /etc/defaults/postgrey and make sure your POSTGREY_OPTS looks like this:

POSTGREY_OPTS="--inet=127.0.0.1:10023"

to ensure IPv4 is used.

Reboot

Make sure everyting’s OK, it should, don’t worry (:

Congratulations

Enjoy your (relative) privacy, and feel free to comment (: