mirror of
https://codeberg.org/forgejo/docs.git
synced 2025-05-10 12:46:15 +03:00
developer: infrastructure at hetzner
* document the current setup at hetzner01 and maintenance procedures
* document the new hardware setup at hetzner{02,03} for hosting code.forgejo.org
This commit is contained in:
Earl Warren
parent
a1ead9463a
commit
69a63cea19
2 changed files with 171 additions and 14 deletions
|
|
@ -15,3 +15,27 @@ To make these repositories easier to find, the following push mirrors are in pla
|
||||||
|
|
||||||
- https://code.forgejo.org/forgejo/runner => https://codeberg.org/forgejo/runner
|
- https://code.forgejo.org/forgejo/runner => https://codeberg.org/forgejo/runner
|
||||||
- https://code.forgejo.org/forgejo/act => https://codeberg.org/forgejo/act
|
- https://code.forgejo.org/forgejo/act => https://codeberg.org/forgejo/act
|
||||||
|
|
||||||
|
## Hardware
|
||||||
|
|
||||||
|
https://code.forgejo.org runs on the `code` LXC container hosted on the [hetzner{02,03}.forgejo.org LXC hypervisor](../infrastructure).
|
||||||
|
|
||||||
|
It is deployed and upgraded using the following [Enough command line](https://enough-community.readthedocs.io):
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ mkdir -p ~/.enough
|
||||||
|
$ git clone https://forgejo.octopuce.forgejo.org/forgejo/enough-code ~/.enough/code.forgejo.org
|
||||||
|
$ enough --domain code.forgejo.org service create --host bind-host forgejo
|
||||||
|
```
|
||||||
|
|
||||||
|
Upgrading only Forgejo:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
$ enough --domain code.forgejo.org playbook -- --limit bind-host,localhost --private-key ~/.enough/code.forgejo.org/infrastructure_key venv/share/enough/playbooks/forgejo/forgejo-playbook.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Login in the machine hosting the Forgejo instance for debugging purposes:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
enough --domain code.forgejo.org ssh bind-host
|
||||||
|
```
|
||||||
|
|
|
||||||
|
|
@ -21,32 +21,165 @@ $ enough --domain octopuce.forgejo.org service create openvpn
|
||||||
|
|
||||||
## Hetzner
|
## Hetzner
|
||||||
|
|
||||||
|
All hardware is running Debian GNU/linux bookworm.
|
||||||
|
|
||||||
|
### hetzner01
|
||||||
|
|
||||||
https://hetzner01.forgejo.org runs on an [EX101](https://www.hetzner.com/dedicated-rootserver/ex101) Hetzner hardware.
|
https://hetzner01.forgejo.org runs on an [EX101](https://www.hetzner.com/dedicated-rootserver/ex101) Hetzner hardware.
|
||||||
|
|
||||||
## OVH
|
There is no backup, no redundancy and is dedicated to Forgejo runner instances.
|
||||||
|
If the hardware reboots, the runners do not restart automatically, they have to be restarted manually.
|
||||||
|
|
||||||
https://code.forgejo.org runs on an OVH virtual machine using the same
|
It hosts LXC containers setup with [lxc-helpers](https://code.forgejo.org/forgejo/lxc-helpers/):
|
||||||
OVH account used for the forgejo.org domain name and mails.
|
|
||||||
|
|
||||||
It is deployed and upgraded using the following [Enough command line](https://enough-community.readthedocs.io):
|
- `forgejo-runners`
|
||||||
|
|
||||||
```shell
|
Dedicated to Forgejo runners for the https://codeberg.org/forgejo organization.
|
||||||
$ mkdir -p ~/.enough
|
|
||||||
$ git clone https://forgejo.octopuce.forgejo.org/forgejo/enough-code ~/.enough/code.forgejo.org
|
```sh
|
||||||
$ enough --domain code.forgejo.org service create --host bind-host forgejo
|
lxc-helpers.sh lxc_container_run forgejo-runners -- sudo --user debian bash
|
||||||
|
cd codeberg.org/forgejo/
|
||||||
|
forgejo-runner-3.2.0 --config config.yml daemon >& runner.log &
|
||||||
|
```
|
||||||
|
|
||||||
|
- `runner01-lxc`
|
||||||
|
|
||||||
|
Dedicated to Forgejo runners for the https://code.forgejo.org
|
||||||
|
organization with two labels: **docker** and **self-hosted**.
|
||||||
|
|
||||||
|
- https://code.forgejo.org/forgejo
|
||||||
|
- https://code.forgejo.org/actions
|
||||||
|
- https://code.forgejo.org/forgejo-integration
|
||||||
|
- https://code.forgejo.org/forgejo-contrib
|
||||||
|
|
||||||
|
```sh
|
||||||
|
lxc-helpers.sh lxc_container_run runner01-lxc -- sudo --user debian bash
|
||||||
|
cd code.forgejo.org
|
||||||
|
for runner in forgejo-contrib forgejo forgejo-integration actions ; do ( cd $runner ; HOME=/srv/$runner forgejo-runner-3.2.0 --config config.yml daemon >&runner.log & ) ; done
|
||||||
|
```
|
||||||
|
|
||||||
|
The runners are installed with something like:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo wget -O /usr/local/bin/forgejo-runner-3.2.0 https://code.forgejo.org/forgejo/runner/releases/download/v3.2.0/forgejo-runner-3.2.0-linux-amd64
|
||||||
|
sudo chmod +x /usr/local/bin/forgejo-runner-3.2.0
|
||||||
```
|
```
|
||||||
|
|
||||||
Upgrading only Forgejo:
|
### hetzner{02,03}
|
||||||
|
|
||||||
```shell
|
https://hetzner02.forgejo.org & https://hetzner03.forgejo.org run on [EX44](https://www.hetzner.com/dedicated-rootserver/ex44) Hetzner hardware.
|
||||||
$ enough --domain code.forgejo.org playbook -- --limit bind-host,localhost --private-key ~/.enough/code.forgejo.org/infrastructure_key venv/share/enough/playbooks/forgejo/forgejo-playbook.yml
|
|
||||||
|
A vSwitch is assigned via the Robot console on both servers
|
||||||
|
and [configured](https://docs.hetzner.com/robot/dedicated-server/network/vswitch#example-debian-configuration)
|
||||||
|
in /etc/network/interfaces for each of them with something like:
|
||||||
|
|
||||||
|
```
|
||||||
|
auto enp5s0.4000
|
||||||
|
iface enp5s0.4000 inet static
|
||||||
|
address 10.53.100.2
|
||||||
|
netmask 255.255.255.0
|
||||||
|
vlan-raw-device enp5s0
|
||||||
|
mtu 1400
|
||||||
```
|
```
|
||||||
|
|
||||||
Login in the machine hosting the Forgejo instance for debugging purposes:
|
#### DRBD
|
||||||
|
|
||||||
|
DRBD is configured with hetzner02 as the primary and hetzner03 as the secondary:
|
||||||
|
|
||||||
```shell
|
|
||||||
enough --domain code.forgejo.org ssh bind-host
|
|
||||||
```
|
```
|
||||||
|
resource r0 {
|
||||||
|
net {
|
||||||
|
# A : write completion is determined when data is written to the local disk and the local TCP transmission buffer
|
||||||
|
# B : write completion is determined when data is written to the local disk and remote buffer cache
|
||||||
|
# C : write completion is determined when data is written to both the local disk and the remote disk
|
||||||
|
protocol C;
|
||||||
|
cram-hmac-alg sha1;
|
||||||
|
# any secret key for authentication among nodes
|
||||||
|
shared-secret "***";
|
||||||
|
}
|
||||||
|
disk {
|
||||||
|
resync-rate 1000M;
|
||||||
|
}
|
||||||
|
on hetzner02 {
|
||||||
|
address 10.53.100.2:7788;
|
||||||
|
volume 0 {
|
||||||
|
# device name
|
||||||
|
device /dev/drbd0;
|
||||||
|
# specify disk to be used for devide above
|
||||||
|
disk /dev/nvme0n1p5;
|
||||||
|
# where to create metadata
|
||||||
|
# specify the block device name when using a different disk
|
||||||
|
meta-disk internal;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
on hetzner03 {
|
||||||
|
address 10.53.100.3:7788;
|
||||||
|
volume 0 {
|
||||||
|
device /dev/drbd0;
|
||||||
|
disk /dev/nvme1n1p5;
|
||||||
|
meta-disk internal;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The DRBD device is mounted on `/var/lib/lxc`.
|
||||||
|
|
||||||
|
In `/etc/fstab` there is a noauto line:
|
||||||
|
|
||||||
|
```
|
||||||
|
/dev/drbd0 /var/lib/lxc ext4 noauto,defaults 0 0
|
||||||
|
```
|
||||||
|
|
||||||
|
To prevent split brain situations a manual step is required at boot
|
||||||
|
time, on the machine that is going to be the primary, which is
|
||||||
|
hetzner02 in a normal situation.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo drbdsetup status
|
||||||
|
sudo drbdadm primary r0
|
||||||
|
sudo mount /var/lib/lxc
|
||||||
|
sudo lxc-autostart start
|
||||||
|
sudo lxc-ls -f
|
||||||
|
sudo drbdsetup status
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Fast storage on /srv
|
||||||
|
|
||||||
|
The second disk on each node is mounted on /srv and can be used when
|
||||||
|
fast storage is needed and there is no need for backups, such as Forgejo runners.
|
||||||
|
|
||||||
|
#### LXC
|
||||||
|
|
||||||
|
LXC is setup with [lxc-helpers](https://code.forgejo.org/forgejo/lxc-helpers/).
|
||||||
|
|
||||||
|
The `/etc/default/lxc-net` file is the same on both machines:
|
||||||
|
|
||||||
|
```
|
||||||
|
USE_LXC_BRIDGE="true"
|
||||||
|
LXC_ADDR="10.6.83.1"
|
||||||
|
LXC_NETMASK="255.255.255.0"
|
||||||
|
LXC_NETWORK="10.6.83.0/24"
|
||||||
|
LXC_DHCP_RANGE="10.6.83.2,10.6.83.254"
|
||||||
|
LXC_DHCP_MAX="253"
|
||||||
|
LXC_IPV6_ADDR="fc16::216:3eff:fe00:1"
|
||||||
|
LXC_IPV6_MASK="64"
|
||||||
|
LXC_IPV6_NETWORK="fc16::/64"
|
||||||
|
LXC_IPV6_NAT="true"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Public IP addresses
|
||||||
|
|
||||||
|
The public IP addresses attached to the hosts are not failover IPs that can be moved from one host to the next.
|
||||||
|
The DNS entry needs to be updated if the primary hosts changes.
|
||||||
|
|
||||||
|
#### Containers
|
||||||
|
|
||||||
|
It hosts LXC containers setup with [lxc-helpers](https://code.forgejo.org/forgejo/lxc-helpers/).
|
||||||
|
|
||||||
|
- `code`
|
||||||
|
|
||||||
|
Dedicated to [code.forgejo.org](code-forgejo-org) and has its own [failover IP](https://docs.hetzner.com/robot/dedicated-server/ip/failover).
|
||||||
|
|
||||||
## Uberspace
|
## Uberspace
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue