Gitea SSL with Apache as reverse proxy

To use container registry in k8s we need it over ssl

Page content

We want to host a nice and secure container registry - to push docker images there and our kubernetes cluster would pull them from this registry. So came the idea to use the gitea over ssl.

  1. Gitea already has container registry
  2. Apache as TLS terminating proxy will add HTTPS to our Gitea.
  3. That how it’s started… Rot CA, self-signed certificates…

Every registry has it’s own secrets

When we need the registry in k8s

Unfortunately registry in kubernetes cluster created by kubespray doesn’t work for me.

  • to push into it we need to create temporary tunnel via kube-proxy
  • after pushing into it mu current version of new cluster could not pull images from this internal registry

After spending some awesome two evenings trying to fix it, I have decided just to use Gitea’s internal container registry, just need to configure HTTPS access to it.

Will make it public anyway, so docker and k8s doesn’t have to do any docker logins. Maybe. Let’s see how we go.

Testing if it’s working

To test if container registry suits us we want to be able

  • to push image there and
  • to create deployment in k8s from that image
sudo docker pull alpine:3.12.0
sudo docker images
sudo docker tag a24bb4013296 localhost:5000/rg/alpine:version.3.12.0
sudo docker push localhost:5000/rg/alpine:version.3.12.0

Now xed alp1.yaml or nano alp1.yaml depending on what mood you are in and

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alp-registry-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alp-registry-test
  template:
    metadata:
      labels:
        app: alp-registry-test
    spec:
      containers:
        - name: alpine-test
          image: localhost:5000/rg/lpine:version.3.12.0
      imagePullSecrets:
      - name: registry-secret

Here is this file available for download Then creating this deployment

kubectl create -f alp1.yaml
kubectl get pods

kubectl describe po alp-registry-test-5f5cb94b97-njsp2
# or whatever pod was created or wasn't created

Yes I know about this bit

      imagePullSecrets:
      - name: registry-secret

registry-secret is the name of the secret kubespray created.

Cleanup

kubectl delete -f alp1.yaml

How

A bunch of documentation is available on Gitea site: https-setup

And on that page: https://docs.gitea.com/administration/reverse-proxies

Step 1 - Install Apache and create simple test site

Install Apache

sudo apt install apache2

Check what we have in the firewall

sudo ufw status

If firewall is active think which port you want to expose via https and allow it. Standart apache configs are

sudo ufw app list

We might see something like

Available applications:
  Apache
  Apache Full
  Apache Secure
  OpenSSH

and to enable only port 443 we execute

sudo ufw allow 'Apache Secure'

ok. now check the service status

sudo systemctl status apache2

Next - create simple virtual server to test the Apache

sudo mkdir /var/www/reg.homelab
sudo chown -R $USER:$USER /var/www/reg.homelab
sudo chmod -R 755 /var/www/reg.homelab
sudo nano /var/www/reg.homelab/index.html

Put there

<html>
    <head>
        <title>Welcome to reg.homelab!</title>
    </head>
    <body>
        <h1>Success!  The reg.homelab virtual host is working!</h1>
    </body>
</html>

then

sudo nano /etc/apache2/sites-available/reg.homelab.conf

and put there

<VirtualHost *:3080>
    ServerAdmin webmaster@localhost
    ServerName reg.homelab
    ServerAlias www.reg.homelab
    DocumentRoot /var/www/reg.homelab
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

then disable default site, enable this one and check how we are doing

sudo a2ensite reg.homelab.conf
sudo a2dissite 000-default.conf
sudo apache2ctl configtest

see something like this?

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message

then

sudo nano /etc/apache2/apache2.conf

add to the end:

ServerName reg.homelab

And this is still not the end! now need to remove port 80 binding attempt

sudo nano /etc/apache2/ports.conf

put there

Listen 3030
...
Listen 443

And now

sudo systemctl restart apache2
sudo systemctl status apache2
journalctl -xeu apache2.service
curl localhost:3080

OK! Now navigate to :3080

Step 2 - Converting this site to insecure reverse proxy :) to Gitea

sudo nano /etc/apache2/sites-available/reg.homelab.conf

putting there

<VirtualHost *:443>
    ServerAdmin webmaster@localhost
    ServerName reg.homelab
    ServerAlias www.reg.homelab
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
    ProxyPreserveHost On
    ProxyRequests off
    AllowEncodedSlashes NoDecode
    ProxyPass / http://localhost:3000/ nocanon
</VirtualHost>

Do some config test

sudo apache2ctl configtest

Add some apache modules and restart Apache

sudo a2enmod proxy proxy_http ssl
sudo systemctl restart apache2
sudo systemctl status apache2

OK, now navigate to or curl

# Yes, it is still a http, but on the 443 port
curl http://localhost:443
http://<Server_IP_Address>:443/

Step 3 Self - signed root CA and site cert

SweetHome-RootCA.

CANAME=MostImportant-RootCA

# optional, create a directory
mkdir $CANAME
cd $CANAME

# generate aes encrypted private key
openssl genrsa -aes256 -out $CANAME.key 4096

# create certificate, 1826 days = 5 years
openssl req -x509 -new -nodes -key $CANAME.key -sha256 -days 1826 -out $CANAME.crt -subj '/CN=My Root CA/C=AT/ST=Vienna/L=Vienna/O=MyOrganisation'

# create certificate for service
MYCERT=reg.homelab
openssl req -new -nodes -out $MYCERT.csr -newkey rsa:4096 -keyout $MYCERT.key -subj '/CN=My Firewall/C=AT/ST=Vienna/L=Vienna/O=MyOrganisation'

# create a v3 ext file for SAN properties
cat > $MYCERT.v3.ext << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = reg.homelablab
DNS.2 = gitea.homelablab
IP.1 = 192.168.0.10
IP.2 = 192.168.0.11
EOF

openssl x509 -req -in $MYCERT.csr -CA $CANAME.crt -CAkey $CANAME.key -CAcreateserial -out $MYCERT.crt -days 730 -sha256 -extfile $MYCERT.v3.ext
On machines connecting to gitea / registry

Register root certificate on linux:

sudo cp MostImportant-RootCA.crt /usr/local/share/ca-certificates
sudo update-ca-certificates

Registre root certificate on windows:

  • Doubleclick file MostImportant-RootCA.crt
  • Import to local user
  • Select Truster Root CA.
  • When prompted about importing untrusted certificate - click yes

When git pull on windows says about

Unable to resolve "unable to get local issuer certificate...

you can tell git to use windows networking layer by

git config --global http.sslbackend schannel

Step 4 - Securing proxy with self-signed certificate

https://httpd.apache.org/docs/2.4/ssl/ssl_howto.html

Create a self-signed certificate IF you didn’t in step 3

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -subj "/CN=reg.homelab" \
  -addext "subjectAltName = DNS:reg.homelab" \
  -keyout /etc/ssl/private/apache-selfsigned-reg.homelab.key \
  -out /etc/ssl/certs/apache-selfsigned-reg.homelab.crt

Or just take it from previous step

sudo cp reg.homelab.crt /etc/ssl/certs/apache-selfsigned-reg.homelab.crt
sudo cp reg.homelab.key /etc/ssl/private/apache-selfsigned-reg.homelab.key

Again, open virtual host config

sudo nano /etc/apache2/sites-available/reg.homelab.conf

Add there to the bottom the SSL section with certs

<VirtualHost *:443>
   ServerAdmin webmaster@localhost
   ServerName reg.homelab
   ServerAlias www.reg.homelab
   ErrorLog ${APACHE_LOG_DIR}/error.log
   CustomLog ${APACHE_LOG_DIR}/access.log combined
   ProxyPreserveHost On
   ProxyRequests off
   AllowEncodedSlashes NoDecode
   ProxyPass / http://localhost:3000/ nocanon
    
   SSLEngine on
   SSLCertificateFile /etc/ssl/certs/apache-selfsigned-reg.homelab.crt
   SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned-reg.homelab.key
</VirtualHost>

Check config, restart server, check status and navigate to our Gitea over SSL

sudo apache2ctl configtest
sudo systemctl restart apache2
sudo systemctl status apache2

# goto: http://<Server_IP_Address>:443/
# or
curl -k -v https://localhost

the browser will warn about the self-signed certificate

Your connection is not private
Attackers might be trying to steal your information from reg.homelab (for example, passwords, messages or credit cards). Learn more
NET::ERR_CERT_AUTHORITY_INVALID

But we will ignore it, for now, till we use Let’s Encrypt.

Now k8s test

Do some DNSing…

On each k8s node:

sudo nano /etc/hosts

and add there

192.168.18.200 gitea.homelab
192.168.18.200 reg.homelab

Root CA

On each k8s node:

sudo cp SweetHome-RootCA.crt /usr/local/share/ca-certificates
sudo update-ca-certificates

And restart now

Create secret with registry credentials

https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

sudo docker login reg.homelab
kubectl create secret generic regcred --from-file=.dockerconfigjson=/home/rg/.docker/config.json --type=kubernetes.io/dockerconfigjson

or

kubectl create secret docker-registry regcred --docker-server=your-registry-server --docker-username=your-name --docker-password=your-pword --docker-email=your-email

New docker image and k8s deployment

sudo docker pull alpine:3.12.0
sudo docker images
sudo docker tag a24bb4013296 reg.homelab/rg/alpine:version.3.12.0
sudo docker push reg.homelab/rg/alpine:version.3.12.0

Now nano alp2.yaml, file is [avaliable]avaliable

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alp-registry-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alp-registry-test
  template:
    metadata:
      labels:
        app: alp-registry-test
    spec:
      containers:
        - name: alpine-test
          image: reg.homelab/rg/alpine:version.3.12.0
      imagePullSecrets:
      - name: regcred

Here is this file available for download: Then creating this deployment

kubectl create -f alp2.yaml
kubectl get pods
kubectl describe po alp...

Cleanup

kubectl delete -f alp2.yaml

https://docs.gitea.com/administration/reverse-proxies

https://httpd.apache.org/docs/2.4/ssl/ssl_howto.html

https://docs.gitea.com/packages/packages/container