Initial guide

README.md

@@ -1,2 +1,41 @@
-# systems-guide
-Guide for systems / infra maintanance and creation
+# Guide for systems / infra maintanance and creation
+
+This guide is assuming you are setting up or maintaining an AlmaLinux (9, 10) server.
+It guides you from setting up a fresh machine, to installing build dependencies, configuring the firewall, serving and monitoring your apps and lastly protecting them from DDOS.
+
+## [Setting up a new machine](setup.md)
+
+Prepare a new machine for use with an `admin` user. Lock out everything but to it. 
+
+## [Installing aplications and build tools](applications.md)
+
+Download packages to build future applications such as NodeJS / Rust.
+Set up your apps in `/srv/` partition.
+
+## [Firewall (opening/checking ports)](firewall.md)
+
+How to open firewall to let outside traffic to your apps.
+
+## [Running services and logging with systemd](systemd.md)
+
+How to set up a new systemd service and read its logs.
+
+## [Nginx & Certbot](nginx-certbot.md)
+
+How to set up Nginx and Certbot to serve your apps with an SSL cert for your desired domain.
+
+## [Monitoring servers (Beszel)](beszel.md)
+
+How to monitor servers with Beszel tool which gives you system load stats and systemd service stats.
+
+## [Monitoring websites and apis and serving a Status Page (Uptime Kuma)](uptime-kuma.md)
+
+How to monitor your services and websites with Open Kuma. 
+
+## [Protecting your services and websites with BunnyCDN](bunny-cdn.md)
+
+How to protect and hide your services and websites from DDOS attacks. 
+
+## [WTF is AlmaLinux](alma-linux.md)
+
+Could be the first link, but really its not that important.

alma-linux.md

@@ -0,0 +1,13 @@
+# Why and why .. Alma Linux?
+
+AlmaLinux is a continuation of open source work for enterprise grade Red Hat Enterprise Linux (RHEL).
+Its a spiritual successor to CentOS and bug-for-bug compatible with RHEL same as Rocky Linux but without the toxic community.
+
+### What about Debian / Ubuntu ???
+
+Tbh, at the time of setting up new servers the `Debian` install was not working on my host, thats why i picked `AlmaLinux` for all installations to keep them uniform.
+Even though its a move from our usual package manager (`apt`) its the same shit. It uses `dnf` and `yum` and everything works the same.
+
+It has a better optimized network stack comapred to `Debians` of the world as its geared strictly towards servers and not desktop systems.
+
+Another selling point is `SELinux` which makes it mucho more secure but needs to be configured carefully.

applications.md

@@ -0,0 +1,41 @@
+# Installing applications and their dependencies
+
+## Put your apps in `/srv/`
+
+Do not just install stuff in `/home/` dirs.
+
+Rather `git clone` the repos to your home and then move it to `/srv/` partition.
+
+```sh
+sudo mv your_repo /srv/your_repo
+```
+
+This way other users of the system can also access and use your app without letting them into your $HOME.
+
+## Installing NodeJS
+
+Install NVM
+
+```sh
+curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
+```
+
+Restart the shell session (or exit / reconnect to ssh) to use `nvm`.
+
+Install and set the latest LTS (long term support) NodeJS.
+
+```sh
+nvm install --lts
+```
+
+## Installing Rust
+
+```sh
+# Install required packages
+# openssl-devel is optional, but you will need it for web stuff such as Axum server
+sudo dnf install curl epel-release cmake gcc make openssl-devel -y
+# Install Rustup
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+```
+
+Restart the shell session (or exit / reconnect to ssh) to use Rust.

beszel.md

@@ -0,0 +1,47 @@
+# Monitoring your machines with Beszel
+
+It provides a cool look into all your machines load stats (cpu, temps and such) plus `systemd` service.
+
+![Beszel overview](public/beszel_overview.png "Beszel")
+
+## Installing
+
+On the Baszel server machine follow instructions [from official docs](https://beszel.dev/guide/hub-installation#binary).
+
+Its a one-liner like:
+
+```sh
+curl -sL https://get.beszel.dev/hub -o /tmp/install-hub.sh && chmod +x /tmp/install-hub.sh && /tmp/install-hub.sh
+```
+
+This will install the beszel binary. The beszel itself will update itself if you accept it when prompted.
+
+## Open ports for Beszel to ssh into machine you want to monitor
+
+Even though beszel operates on Websockets primarily it will SSH into the machine in case Websocket connection is lost.
+Make sure port `45876` on target machine to SSH into is open.
+
+If you havent done so before, check [Firewall](firewall.md) section for extra tips, but here is a quick oneliner:
+
+```sh
+sudo firewall-cmd --permanent --zone=public --add-port=45876/tcp &&
+sudo firewall-cmd --reload &&
+sudo systemctl restart beszel-agent
+```
+
+also make sure that the default zone on the machine is `public`.
+If its `drop` then run:
+
+```sh
+sudo firewall-cmd --set-default-zone=public && sudo firewall-cmd --reload
+```
+
+## Add machines on Beszel
+
+Click **Add New System**
+
+Input relevant info and click **Copy linux command**.
+
+![How to add system to beszel](public/beszel_add_system.png "Beszel add system")
+
+Use this command on the target machine to easily connect Baszel.

bunny-cdn.md

@@ -0,0 +1,45 @@
+# Protecting your services and websites with BunnyCDN
+
+BunnyCDN is your friend in 3 ways at least:
+
+- it will protect you against DDOS
+- it will obscure the real origin of your service
+- it will cache your content that u want cached
+
+It operates with "zones". Think 1 zone = 1 service / website / url you run.
+
+## No caching please
+
+By default if server doesnt include a `Cache-control` header, bunny will cache it.
+
+To make it never cache, include header `Cache-control` with value `no-cache` in your responses.
+
+## Purging cache
+
+To purge the cache on BunnyCDN, there is a button on the top right of the UI.
+
+## How to add a service / website to BunnyCDN
+
+Consult the screenshots below on how to set BunnyCDN in your browser.
+
+#### 1. Create a new pull zone
+
+![New zone 1](public/bunny1.png "Step 1")
+
+#### 2. Add your domain
+
+![New zone 2](public/bunny2.png "Step 2")
+
+#### 3. Add a DNS record and SSL cert on BunnyCDN
+
+Get details for DNS from the popup.
+
+![New zone 3](public/bunny3.png "Step 3")
+
+Enter it into your DNS and Verify SSL back on BunnyCDN.
+
+⚠️ MAKE SURE TO ADD A DOT AT THE END. Bunny doesnt put it in the modal shown to you. If you do not put a dot at the end of CNAME record then it will append your domain at the end of CNAME-d url.
+
+![New zone 4](public/bunny4.png "Step 4")
+
+Voila, this is it

firewall.md

@@ -0,0 +1,78 @@
+# Firewall
+
+We use `firewalld` to configure firewalls. It uses so called "zones" to define the rules.
+
+In the [Setup](setup.md) guide we have installed and set up `firewalld`, left `ssh` open and closing all other traffic by setting the default zone to "drop".
+
+Check [Offish guide](https://www.redhat.com/en/blog/firewalld-rules-and-scenarios) for more details.
+
+### Zones
+
+Drop: Connections are dropped without any notifications. Outgoing connections are possible.
+Public: This zone is used for devices on the untrusted public network.
+
+We are gonna be using Public zone to serve content over the internet.
+
+### Services and ports
+
+With firewalld
+
+## Checking status
+
+To see all info for active zones
+
+```sh
+sudo firewall-cmd --list-all
+```
+
+If you wanna check a specific zone (aka public), just add `--zone=public` for example
+Most often you will be interested in the public zone.
+
+To see all open ports in a zone such as "public"
+
+```sh
+sudo firewall-cmd --zone=public --list-ports
+
+```
+
+## Preparing to serve over http/https or any other port
+
+### Changing default zone
+
+Change the default zone to `public` with
+
+```sh
+sudo firewall-cmd --permanent --zone=public
+```
+
+Then open the relevant ports and reload the firewall
+  
+```sh
+sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
+sudo firewall-cmd --permanent --zone=public --add-port=443/tcp
+sudo firewall-cmd --reload
+```
+
+OR
+
+You can also open "services", these are just aliases for port/protocol pairing (aka service=http is equal to port 80/tcp)
+
+```sh
+sudo firewall-cmd --permanent --zone=public --add-service=http
+sudo firewall-cmd --permanent --zone=public --add-service=https
+sudo firewall-cmd --reload
+```
+
+Note: use `--permanent` flag to add stuff to firewall config permanently. Otherwise it will reset when you do `--reload`.
+
+If you need to open just a specific port, you can do it as above example:
+
+```sh
+sudo firewall-cmd --permanent --zone=public --add-port=$PORT_NUMBER/$PROTOCOL
+# where $PROTOCOL is one of `tcp , udp , sctp or dccp`
+# For example
+# sudo firewall-cmd --permanent --zone=public --add-port=1234/tcp
+sudo firewall-cmd --reload
+```
+
+## TODO: How to only allow specific IPs to access your service

nginx-certbot.md

@@ -0,0 +1,87 @@
+# Serve with Nginx and certify with Certbot SSL
+
+Note: `snapd` isnt stable on Almalinux 10 at the time of writing, you might wanna install it via `pip` or some other way.
+
+## Installing packages
+
+```sh
+# Install packages
+sudo dnf install snapd nginx && systemctl enable nginx && systemctl start nginx
+# Snapd for certbot (https certs)
+sudo systemctl enable --now snapd.socket &&
+sudo ln -s /var/lib/snapd/snap /snap &&
+sudo ln -s /snap/bin/certbot /usr/bin/certbot &&
+sudo snap install --classic certbot &&
+sudo ln -s /snap/bin/certbot /usr/bin/certbot
+```
+
+This installed our nginx and certbot.
+
+## Serving a website via nginx
+
+Create an empty config file 
+
+```sh
+nvim /etc/nginx/conf.d/$YOUR_SERVICE.conf
+```
+
+and then add
+
+```nginx
+server {
+	server_name sub.yourdomain.com
+
+	location / {
+		root html;
+		index index.html;
+	}
+}
+```
+
+then
+
+```sh
+sudo certbot --nginx -d sub.yourdomain.com
+sudo certbot renew --dry-run # test renewal
+```
+
+This is all you need to have a self-renewing certificate for a service or website.
+
+## Passing the traffic thru to your APIs with `upstream`
+
+For an API service, do the same as above point, but go into your config and change the `/your_api` block to include:
+
+```nginx
+upstream your_api {
+    server 127.0.0.1:3498;
+}
+
+# ...
+
+server {
+
+  # ...
+
+  location / {
+        proxy_pass         http://your_api;
+        proxy_http_version 1.1;
+        proxy_set_header   Host              $host;
+        proxy_set_header   X-Real-IP         $remote_addr;
+        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header   X-Forwarded-Proto $scheme;
+
+        # WebSocket support (if needed)
+        proxy_set_header   Upgrade           $http_upgrade;
+        proxy_set_header   Connection        "Upgrade";
+
+        # Timeouts
+        proxy_connect_timeout 5s;
+        proxy_read_timeout    60s;
+        proxy_send_timeout    60s;
+    }
+
+  # ...
+
+}
+
+```

setup.md

@@ -0,0 +1,113 @@
+# Setting up a new machine
+
+This guides assumes you are on a fresh machine running AlmaLinux 9 or 10, ssh-ed as `root`.
+It will show how to install required packages, configure firewalls, lock the server to just SSH and create an `admin` user.
+
+## Installing initial packages
+
+First Update system
+```sh
+dnf update -y && dnf upgrade -y
+```
+
+Then install the neccessary packages
+
+```sh
+dnf install -y epel-release firewalld bind-utils git fail2ban neovim
+```
+
+`epel-release` is neccessary to get fail2ban and some later dependencies
+It stands for "extra packages for enterprise linux".
+
+## Closing the Firewall to anything but SSH
+
+We are gonna close the firewall to anything but SSH for now.
+We dont have anything running on the server yet anyway.
+
+```sh
+# Enable firewalld
+systemctl enable --now firewalld
+# This will set the default firewall zone to DROP, which means traffic will get dropped by default.
+# Later on you will want to change this to zone=public so you can serve traffic over http/https
+firewall-cmd --set-default-zone=drop &&
+firewall-cmd --add-service=ssh --permanent &&
+firewall-cmd --reload
+```
+
+## Unattended upgrades
+
+```sh
+dnf install -y dnf-automatic
+systemctl enable --now dnf-automatic.timer
+```
+
+## Enabling fail2ban
+
+This will keep the masses of bots of trying to SSH / log into our server by banning their ips if they are annoying us.
+
+Create a new jail file at `/etc/fail2ban/jail.local`
+
+```toml
+[sshd]
+enabled = true
+```
+
+Start it
+
+```sh
+systemctl enable --now fail2ban
+```
+
+## Creating the admin user
+
+To avoid using `root` user we are gonna create a new user with `sudo` privileges and prevent logging into root via `ssh`.
+
+The password for `admin` user is in our Bitwarden.
+
+```sh
+adduser admin
+passwd admin # Will prompt you for a new password
+usermod -aG wheel admin # Give elevated (sudo) privileges to the user
+```
+
+### Add the SSH key to `admin` user's authorized keys so you can SSH into it
+
+Switch to `admin` account
+
+```sh
+su -i admin
+```
+
+Create files and paste your public key
+
+```sh
+cd ~ # in case you arent in $HOME dir
+mkdir .ssh
+nvim .ssh/authorized_keys # paste relevant SSH public keys in here
+```
+
+Try opening a new terminal and ssh-ing into `admin` user on the server, it should work.
+Be sure this is the case before you lock `root` account out.
+
+### Locking the `root` account
+
+Go back to `root` account now, otherwise you will need to `sudo` the commands below.
+The following commands will lock out the root by configuring `/etc/ssh/sshd_config` file.
+
+```sh
+sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
+sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
+systemctl restart sshd
+````
+
+If done correctly `root` is no longer available.
+
+### Accessing the `root` account locally thru another account
+
+Even though we went thru the trouble of setting up a dedicated admin user, you can still switch into `root` account by simply executing:
+
+```sh
+sudo -i
+```
+
+It will prompt you for `admin` password and then log into `root`.

systemd.md

@@ -0,0 +1,98 @@
+# Configuring, running and logging with `systemd`
+
+## Why you want this
+
+`systemd` provides an out-of-the box linux native way of running and logging for your services.
+
+You could as well use things like `tmux`, `screen`, `pm2` or just a background task, but they dont cut all the cases.
+`systemd` provides a way to declaratively define your app startup, re-startup, service dependencies and pipe its outputs in a structured manner to `journalctl` where they are stored in the most efficient manner.
+
+
+## How to create a new service that runs persistently even if server restarts
+
+First create your new service file
+
+```sh
+sudo nvim /etc/systemd/system/$SERVICE_NAME.service
+```
+
+then paste and edit this template:
+
+```toml
+[Unit]
+Description=Service name
+After=network.target
+Wants=network.target
+
+[Service]
+# Set the working directory where your binary resides
+WorkingDirectory=/srv/YOUR_PATH
+# Absolute path to your binary
+ExecStart=/srv/YOUR_PATH/target/release/BINARY_NAME
+
+# Ensure the service restarts on failure
+Restart=always
+RestartSec=5
+
+# Optional: prevent rapid restart loops
+StartLimitIntervalSec=60
+StartLimitBurst=3
+
+# Direct stdout and stderr to the system journal
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=YOUR_SERVICE
+
+# Optional: increase file descriptor limits if needed
+LimitNOFILE=4096
+
+[Install]
+WantedBy=multi-user.target
+```
+
+then enable and start your service
+
+```sh
+sudo systemctl enable $YOUR_SERVICE
+sudo systemctl start $YOUR_SERVICE
+```
+
+it will now keep running even after restarts.
+
+## Logging your server
+
+A better guide: [click here](https://www.loggly.com/ultimate-guide/using-journalctl/)
+
+#### To get all logs simply do
+
+```sh
+journalctl -u your_service
+```
+
+#### To follow logs append `-f` option.
+
+```sh
+journalctl -u your_service -f
+```
+
+#### To query logs by time use `--since` and `--until`
+
+```sh
+# it uses a human readable way. You can operate with hours, seconds etc.
+journalctl -u your_service --since "2 hours ago"
+# no timestamps, you need the below format to do precise times
+journalctl -u your_service --since "2015-06-26 23:15:00" --until "2015-06-26 23:20:00"
+```
+
+#### To get precise number of logs, use `-n` option.
+
+```sh
+journalctl -u your_service -n 100
+```
+
+If you want to get latest N messages, use also the `-r` option to reverse the log order to get latest messages.
+
+```sh
+# Get last 100 logs
+journalctl -u your_service -r -n 100
+```

uptime-kuma.md

@@ -0,0 +1,11 @@
+# Monitoring your websites and APIs with Uptime Kuma
+
+Uptime Kuma is very straightforward. The UI is very self explanatory for adding monitoring targets and status pages.
+
+## Installing and updating Uptime Kuma
+
+Follow instructions: [https://github.com/louislam/uptime-kuma/wiki/%F0%9F%94%A7-How-to-Install](https://github.com/louislam/uptime-kuma/wiki/%F0%9F%94%A7-How-to-Install)
+
+Its a couple of commands and `pm2` to keep the service running.
+
+The folloup installation occurs when you first visit your uptime kuma instance.