Feep! » Blog » Post

Getting a public HTTPS certificate for a private service

I run a personal GitLab instance on a local machine. Since there’s no real reason to expose it to the internet, and I don’t trust myself to stay on top of security patches, it’s only accessible inside my network. However, I still want it to have an HTTPS certificate, mostly because more and more things (correctly) insist on using an encrypted connection. If the instance were publicly accessible, this would be easy: GitLab has built-in support for Let’s Encrypt, which would handle everything automatically. But since my server is behind a firewall and not publicly reachable, Let’s Encrypt can’t just talk to the server directly when it’s verifying I control the domain, so I had to do something rather more complicated.

(I do use GitLab to host the source code and CI for the search engine, among other things, but really this post is only tangentially about Feep!. It seemed like it would be relevant to my audience, though, so I’m writing about it here anyway.)

The Problem

To get an HTTPS certificate, I needed to make /.well-known/acme-challenge/ accessible to Let’s Encrypt—and, therefore, the rest of the Internet—without exposing the rest of my GitLab instance. Fortunately, I have a server in the cloud that hosts my personal websites; I can set up a tunnel there and configure NGINX to route only the ACME challenge requests to my GitLab instance, while keeping everything else private.

However, my initial setup was a mess. It worked, but only because of quirks in my home network rather than by design.

I decided to start from scratch with a cleaner approach.

The New Setup

I recently set up Tailscale on my personal devices, for unrelated reasons; but I decided to use it here, too. (I have mixed feelings about Tailscale: it’s certainly easier to set up than raw WireGuard, but it goes the other way and has a ton of magic which I’m not a fan of. I think I’d prefer something that had the NAT hole-punching but didn’t default to turning on an SSH server.)

To expose GitLab to my devices and the public server, I run a Tailscale instance in a sidecar container alongside the GitLab container. My public server has an allocated IPv6 range, and I assigned a subnet in that range to the GitLab Tailscale instance, giving it a public IPv6 address. I then added this address to the public DNS so that both private and public traffic will use it. On my devices, Tailscale handles subnet routing, ensuring that my devices connect to the container. For everyone else, the same address routes to the public server, allowing Let’s Encrypt validation requests to be proxied through Nginx. (The only exception is that my public server knows it owns the IPv6 range, so it routes the address locally instead of through Tailscale. To work around this, I added an override in /etc/hosts to force it to use the Tailscale-internal IPv4 address instead. This still involves some split-horizon weirdness, but at least it’s contained in one place.)

A half-baked tutorial

I’m going to try to explain how I set this up, but there are a lot of moving pieces so while I’ve tried not to leave anything out I can’t promise that I haven’t missed out a crucial step somewhere. I’ll start with the Tailscale sidecar container configuration, using nginx as an example application:

services:
  web:
    image: nginx
    network_mode: service:tailscale

  tailscale:
    image: 'tailscale/tailscale'
    environment:
      #- TS_AUTHKEY=tskey-auth-... # Only needed once during bringup, can be removed after
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_USERSPACE=false
      - TS_EXTRA_ARGS=--advertise-routes=2600:3c01:e000:9e1:400::11/128
    cap_add:
      - net_admin
    networks:
      gitlab-ts:
        ipv6_address: 2600:3c01:e000:9e1:400::11 # Arbitrarily selected single IP in range
    volumes:
      - /srv/gitlab/wolf-tailscale-state:/var/lib/tailscale
      - /dev/net/tun:/dev/net/tun
    
networks:
  gitlab-ts:
    enable_ipv6: true
    ipam:
      config:
        - subnet: 2600:3c01:e000:9e1:400::/70 # public range, here being routed privately

GitLab only listens on IPv4 by default, so I had to tell it to do IPv6 as well:

    environment:
      GITLAB_OMNIBUS_CONFIG: |
        # ...
        nginx['listen_addresses'] = ["0.0.0.0", "[::]"]
        registry_nginx['listen_addresses'] = ['*', '[::]']
        mattermost_nginx['listen_addresses'] = ['*', '[::]']
        pages_nginx['listen_addresses'] = ['*', '[::]']

On the client machines, I had to accept the advertised route with:

sudo tailscale up --accept-routes

The public server needs to listen on the same IP address, so I added that to /etc/systemd/network/05-eth0.network:

[Network]
# ...
Gateway=fe80::1
Address=2600:3c01:e000:9e1:400::11/64

The hostname gets an AAAA address in DNS, but I had to override that on this server to avoid a routing loop, so I added an entry to /etc/hosts:

100.86.231.74 gitlab.wolfgangfaust.com

Finally, gitlab-acme.nginx.conf:

server {
	server_name gitlab.wolfgangfaust.com;
	listen 80;
	listen [::]:80;

	location / {
		return 404;
	}
	location /.well-known/acme-challenge/ {
		proxy_pass http://gitlab.wolfgangfaust.com;
	}
}

Conclusion

With this setup, Let’s Encrypt can validate my domain and issue an HTTPS certificate, while GitLab remains securely hidden from the public internet. Using a public IPv6 address removes the need for split-horizon DNS, SSH tunnels, and host file overrides; instead, the split is in the routing tables, where it can be managed by Tailscale. This solution was a headache to set up (I haven’t worked with either Tailscale routing or IPv6 before and there were a lot of red herrings I left out of this post), but it seems to be more robust and avoids depending on the weird network quirks that plagued my previous attempt.