diff --git a/README.md b/README.md index 1303114..a4e94b5 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,41 @@ -# systems-guide -Guide for systems / infra maintanance and creation +# Guide for systems / infra maintanance and creation + +This guide is assuming you are setting up or maintaining an AlmaLinux (9, 10) server. +It guides you from setting up a fresh machine, to installing build dependencies, configuring the firewall, serving and monitoring your apps and lastly protecting them from DDOS. + +## [Setting up a new machine](setup.md) + +Prepare a new machine for use with an `admin` user. Lock out everything but to it. + +## [Installing aplications and build tools](applications.md) + +Download packages to build future applications such as NodeJS / Rust. +Set up your apps in `/srv/` partition. + +## [Firewall (opening/checking ports)](firewall.md) + +How to open firewall to let outside traffic to your apps. + +## [Running services and logging with systemd](systemd.md) + +How to set up a new systemd service and read its logs. + +## [Nginx & Certbot](nginx-certbot.md) + +How to set up Nginx and Certbot to serve your apps with an SSL cert for your desired domain. + +## [Monitoring servers (Beszel)](beszel.md) + +How to monitor servers with Beszel tool which gives you system load stats and systemd service stats. + +## [Monitoring websites and apis and serving a Status Page (Uptime Kuma)](uptime-kuma.md) + +How to monitor your services and websites with Open Kuma. + +## [Protecting your services and websites with BunnyCDN](bunny-cdn.md) + +How to protect and hide your services and websites from DDOS attacks. + +## [WTF is AlmaLinux](alma-linux.md) + +Could be the first link, but really its not that important. diff --git a/alma-linux.md b/alma-linux.md new file mode 100644 index 0000000..182d14e --- /dev/null +++ b/alma-linux.md @@ -0,0 +1,13 @@ +# Why and why .. Alma Linux? + +AlmaLinux is a continuation of open source work for enterprise grade Red Hat Enterprise Linux (RHEL). +Its a spiritual successor to CentOS and bug-for-bug compatible with RHEL same as Rocky Linux but without the toxic community. + +### What about Debian / Ubuntu ??? + +Tbh, at the time of setting up new servers the `Debian` install was not working on my host, thats why i picked `AlmaLinux` for all installations to keep them uniform. +Even though its a move from our usual package manager (`apt`) its the same shit. It uses `dnf` and `yum` and everything works the same. + +It has a better optimized network stack comapred to `Debians` of the world as its geared strictly towards servers and not desktop systems. + +Another selling point is `SELinux` which makes it mucho more secure but needs to be configured carefully. diff --git a/applications.md b/applications.md new file mode 100644 index 0000000..e85c0aa --- /dev/null +++ b/applications.md @@ -0,0 +1,41 @@ +# Installing applications and their dependencies + +## Put your apps in `/srv/` + +Do not just install stuff in `/home/` dirs. + +Rather `git clone` the repos to your home and then move it to `/srv/` partition. + +```sh +sudo mv your_repo /srv/your_repo +``` + +This way other users of the system can also access and use your app without letting them into your $HOME. + +## Installing NodeJS + +Install NVM + +```sh +curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash +``` + +Restart the shell session (or exit / reconnect to ssh) to use `nvm`. + +Install and set the latest LTS (long term support) NodeJS. + +```sh +nvm install --lts +``` + +## Installing Rust + +```sh +# Install required packages +# openssl-devel is optional, but you will need it for web stuff such as Axum server +sudo dnf install curl epel-release cmake gcc make openssl-devel -y +# Install Rustup +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +``` + +Restart the shell session (or exit / reconnect to ssh) to use Rust. diff --git a/beszel.md b/beszel.md new file mode 100644 index 0000000..2d0bcc7 --- /dev/null +++ b/beszel.md @@ -0,0 +1,47 @@ +# Monitoring your machines with Beszel + +It provides a cool look into all your machines load stats (cpu, temps and such) plus `systemd` service. + +![Beszel overview](public/beszel_overview.png "Beszel") + +## Installing + +On the Baszel server machine follow instructions [from official docs](https://beszel.dev/guide/hub-installation#binary). + +Its a one-liner like: + +```sh +curl -sL https://get.beszel.dev/hub -o /tmp/install-hub.sh && chmod +x /tmp/install-hub.sh && /tmp/install-hub.sh +``` + +This will install the beszel binary. The beszel itself will update itself if you accept it when prompted. + +## Open ports for Beszel to ssh into machine you want to monitor + +Even though beszel operates on Websockets primarily it will SSH into the machine in case Websocket connection is lost. +Make sure port `45876` on target machine to SSH into is open. + +If you havent done so before, check [Firewall](firewall.md) section for extra tips, but here is a quick oneliner: + +```sh +sudo firewall-cmd --permanent --zone=public --add-port=45876/tcp && +sudo firewall-cmd --reload && +sudo systemctl restart beszel-agent +``` + +also make sure that the default zone on the machine is `public`. +If its `drop` then run: + +```sh +sudo firewall-cmd --set-default-zone=public && sudo firewall-cmd --reload +``` + +## Add machines on Beszel + +Click **Add New System** + +Input relevant info and click **Copy linux command**. + +![How to add system to beszel](public/beszel_add_system.png "Beszel add system") + +Use this command on the target machine to easily connect Baszel. diff --git a/bunny-cdn.md b/bunny-cdn.md new file mode 100644 index 0000000..0db6b85 --- /dev/null +++ b/bunny-cdn.md @@ -0,0 +1,45 @@ +# Protecting your services and websites with BunnyCDN + +BunnyCDN is your friend in 3 ways at least: + +- it will protect you against DDOS +- it will obscure the real origin of your service +- it will cache your content that u want cached + +It operates with "zones". Think 1 zone = 1 service / website / url you run. + +## No caching please + +By default if server doesnt include a `Cache-control` header, bunny will cache it. + +To make it never cache, include header `Cache-control` with value `no-cache` in your responses. + +## Purging cache + +To purge the cache on BunnyCDN, there is a button on the top right of the UI. + +## How to add a service / website to BunnyCDN + +Consult the screenshots below on how to set BunnyCDN in your browser. + +#### 1. Create a new pull zone + +![New zone 1](public/bunny1.png "Step 1") + +#### 2. Add your domain + +![New zone 2](public/bunny2.png "Step 2") + +#### 3. Add a DNS record and SSL cert on BunnyCDN + +Get details for DNS from the popup. + +![New zone 3](public/bunny3.png "Step 3") + +Enter it into your DNS and Verify SSL back on BunnyCDN. + +⚠️ MAKE SURE TO ADD A DOT AT THE END. Bunny doesnt put it in the modal shown to you. If you do not put a dot at the end of CNAME record then it will append your domain at the end of CNAME-d url. + +![New zone 4](public/bunny4.png "Step 4") + +Voila, this is it diff --git a/firewall.md b/firewall.md new file mode 100644 index 0000000..203e210 --- /dev/null +++ b/firewall.md @@ -0,0 +1,78 @@ +# Firewall + +We use `firewalld` to configure firewalls. It uses so called "zones" to define the rules. + +In the [Setup](setup.md) guide we have installed and set up `firewalld`, left `ssh` open and closing all other traffic by setting the default zone to "drop". + +Check [Offish guide](https://www.redhat.com/en/blog/firewalld-rules-and-scenarios) for more details. + +### Zones + +Drop: Connections are dropped without any notifications. Outgoing connections are possible. +Public: This zone is used for devices on the untrusted public network. + +We are gonna be using Public zone to serve content over the internet. + +### Services and ports + +With firewalld + +## Checking status + +To see all info for active zones + +```sh +sudo firewall-cmd --list-all +``` + +If you wanna check a specific zone (aka public), just add `--zone=public` for example +Most often you will be interested in the public zone. + +To see all open ports in a zone such as "public" + +```sh +sudo firewall-cmd --zone=public --list-ports + +``` + +## Preparing to serve over http/https or any other port + +### Changing default zone + +Change the default zone to `public` with + +```sh +sudo firewall-cmd --permanent --zone=public +``` + +Then open the relevant ports and reload the firewall + +```sh +sudo firewall-cmd --permanent --zone=public --add-port=80/tcp +sudo firewall-cmd --permanent --zone=public --add-port=443/tcp +sudo firewall-cmd --reload +``` + +OR + +You can also open "services", these are just aliases for port/protocol pairing (aka service=http is equal to port 80/tcp) + +```sh +sudo firewall-cmd --permanent --zone=public --add-service=http +sudo firewall-cmd --permanent --zone=public --add-service=https +sudo firewall-cmd --reload +``` + +Note: use `--permanent` flag to add stuff to firewall config permanently. Otherwise it will reset when you do `--reload`. + +If you need to open just a specific port, you can do it as above example: + +```sh +sudo firewall-cmd --permanent --zone=public --add-port=$PORT_NUMBER/$PROTOCOL +# where $PROTOCOL is one of `tcp , udp , sctp or dccp` +# For example +# sudo firewall-cmd --permanent --zone=public --add-port=1234/tcp +sudo firewall-cmd --reload +``` + +## TODO: How to only allow specific IPs to access your service diff --git a/nginx-certbot.md b/nginx-certbot.md new file mode 100644 index 0000000..b46966c --- /dev/null +++ b/nginx-certbot.md @@ -0,0 +1,87 @@ +# Serve with Nginx and certify with Certbot SSL + +Note: `snapd` isnt stable on Almalinux 10 at the time of writing, you might wanna install it via `pip` or some other way. + +## Installing packages + +```sh +# Install packages +sudo dnf install snapd nginx && systemctl enable nginx && systemctl start nginx +# Snapd for certbot (https certs) +sudo systemctl enable --now snapd.socket && +sudo ln -s /var/lib/snapd/snap /snap && +sudo ln -s /snap/bin/certbot /usr/bin/certbot && +sudo snap install --classic certbot && +sudo ln -s /snap/bin/certbot /usr/bin/certbot +``` + +This installed our nginx and certbot. + +## Serving a website via nginx + +Create an empty config file + +```sh +nvim /etc/nginx/conf.d/$YOUR_SERVICE.conf +``` + +and then add + +```nginx +server { + server_name sub.yourdomain.com + + location / { + root html; + index index.html; + } +} +``` + +then + +```sh +sudo certbot --nginx -d sub.yourdomain.com +sudo certbot renew --dry-run # test renewal +``` + +This is all you need to have a self-renewing certificate for a service or website. + +## Passing the traffic thru to your APIs with `upstream` + +For an API service, do the same as above point, but go into your config and change the `/your_api` block to include: + +```nginx +upstream your_api { + server 127.0.0.1:3498; +} + +# ... + +server { + + # ... + + location / { + proxy_pass http://your_api; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # WebSocket support (if needed) + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "Upgrade"; + + # Timeouts + proxy_connect_timeout 5s; + proxy_read_timeout 60s; + proxy_send_timeout 60s; + } + + # ... + +} + +``` diff --git a/public/beszel_add_system.png b/public/beszel_add_system.png new file mode 100644 index 0000000..e358cf0 Binary files /dev/null and b/public/beszel_add_system.png differ diff --git a/public/beszel_overview.png b/public/beszel_overview.png new file mode 100644 index 0000000..ce27db0 Binary files /dev/null and b/public/beszel_overview.png differ diff --git a/public/bunny1.png b/public/bunny1.png new file mode 100644 index 0000000..138a898 Binary files /dev/null and b/public/bunny1.png differ diff --git a/public/bunny2.png b/public/bunny2.png new file mode 100644 index 0000000..240dcd0 Binary files /dev/null and b/public/bunny2.png differ diff --git a/public/bunny3.png b/public/bunny3.png new file mode 100644 index 0000000..e62b79c Binary files /dev/null and b/public/bunny3.png differ diff --git a/public/bunny4.png b/public/bunny4.png new file mode 100644 index 0000000..d262fa9 Binary files /dev/null and b/public/bunny4.png differ diff --git a/setup.md b/setup.md new file mode 100644 index 0000000..6fe65ca --- /dev/null +++ b/setup.md @@ -0,0 +1,113 @@ +# Setting up a new machine + +This guides assumes you are on a fresh machine running AlmaLinux 9 or 10, ssh-ed as `root`. +It will show how to install required packages, configure firewalls, lock the server to just SSH and create an `admin` user. + +## Installing initial packages + +First Update system +```sh +dnf update -y && dnf upgrade -y +``` + +Then install the neccessary packages + +```sh +dnf install -y epel-release firewalld bind-utils git fail2ban neovim +``` + +`epel-release` is neccessary to get fail2ban and some later dependencies +It stands for "extra packages for enterprise linux". + +## Closing the Firewall to anything but SSH + +We are gonna close the firewall to anything but SSH for now. +We dont have anything running on the server yet anyway. + +```sh +# Enable firewalld +systemctl enable --now firewalld +# This will set the default firewall zone to DROP, which means traffic will get dropped by default. +# Later on you will want to change this to zone=public so you can serve traffic over http/https +firewall-cmd --set-default-zone=drop && +firewall-cmd --add-service=ssh --permanent && +firewall-cmd --reload +``` + +## Unattended upgrades + +```sh +dnf install -y dnf-automatic +systemctl enable --now dnf-automatic.timer +``` + +## Enabling fail2ban + +This will keep the masses of bots of trying to SSH / log into our server by banning their ips if they are annoying us. + +Create a new jail file at `/etc/fail2ban/jail.local` + +```toml +[sshd] +enabled = true +``` + +Start it + +```sh +systemctl enable --now fail2ban +``` + +## Creating the admin user + +To avoid using `root` user we are gonna create a new user with `sudo` privileges and prevent logging into root via `ssh`. + +The password for `admin` user is in our Bitwarden. + +```sh +adduser admin +passwd admin # Will prompt you for a new password +usermod -aG wheel admin # Give elevated (sudo) privileges to the user +``` + +### Add the SSH key to `admin` user's authorized keys so you can SSH into it + +Switch to `admin` account + +```sh +su -i admin +``` + +Create files and paste your public key + +```sh +cd ~ # in case you arent in $HOME dir +mkdir .ssh +nvim .ssh/authorized_keys # paste relevant SSH public keys in here +``` + +Try opening a new terminal and ssh-ing into `admin` user on the server, it should work. +Be sure this is the case before you lock `root` account out. + +### Locking the `root` account + +Go back to `root` account now, otherwise you will need to `sudo` the commands below. +The following commands will lock out the root by configuring `/etc/ssh/sshd_config` file. + +```sh +sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config +sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config +systemctl restart sshd +```` + +If done correctly `root` is no longer available. + +### Accessing the `root` account locally thru another account + +Even though we went thru the trouble of setting up a dedicated admin user, you can still switch into `root` account by simply executing: + +```sh +sudo -i +``` + +It will prompt you for `admin` password and then log into `root`. diff --git a/systemd.md b/systemd.md new file mode 100644 index 0000000..01b6f65 --- /dev/null +++ b/systemd.md @@ -0,0 +1,98 @@ +# Configuring, running and logging with `systemd` + +## Why you want this + +`systemd` provides an out-of-the box linux native way of running and logging for your services. + +You could as well use things like `tmux`, `screen`, `pm2` or just a background task, but they dont cut all the cases. +`systemd` provides a way to declaratively define your app startup, re-startup, service dependencies and pipe its outputs in a structured manner to `journalctl` where they are stored in the most efficient manner. + + +## How to create a new service that runs persistently even if server restarts + +First create your new service file + +```sh +sudo nvim /etc/systemd/system/$SERVICE_NAME.service +``` + +then paste and edit this template: + +```toml +[Unit] +Description=Service name +After=network.target +Wants=network.target + +[Service] +# Set the working directory where your binary resides +WorkingDirectory=/srv/YOUR_PATH +# Absolute path to your binary +ExecStart=/srv/YOUR_PATH/target/release/BINARY_NAME + +# Ensure the service restarts on failure +Restart=always +RestartSec=5 + +# Optional: prevent rapid restart loops +StartLimitIntervalSec=60 +StartLimitBurst=3 + +# Direct stdout and stderr to the system journal +StandardOutput=journal +StandardError=journal +SyslogIdentifier=YOUR_SERVICE + +# Optional: increase file descriptor limits if needed +LimitNOFILE=4096 + +[Install] +WantedBy=multi-user.target +``` + +then enable and start your service + +```sh +sudo systemctl enable $YOUR_SERVICE +sudo systemctl start $YOUR_SERVICE +``` + +it will now keep running even after restarts. + +## Logging your server + +A better guide: [click here](https://www.loggly.com/ultimate-guide/using-journalctl/) + +#### To get all logs simply do + +```sh +journalctl -u your_service +``` + +#### To follow logs append `-f` option. + +```sh +journalctl -u your_service -f +``` + +#### To query logs by time use `--since` and `--until` + +```sh +# it uses a human readable way. You can operate with hours, seconds etc. +journalctl -u your_service --since "2 hours ago" +# no timestamps, you need the below format to do precise times +journalctl -u your_service --since "2015-06-26 23:15:00" --until "2015-06-26 23:20:00" +``` + +#### To get precise number of logs, use `-n` option. + +```sh +journalctl -u your_service -n 100 +``` + +If you want to get latest N messages, use also the `-r` option to reverse the log order to get latest messages. + +```sh +# Get last 100 logs +journalctl -u your_service -r -n 100 +``` diff --git a/uptime-kuma.md b/uptime-kuma.md new file mode 100644 index 0000000..375e06f --- /dev/null +++ b/uptime-kuma.md @@ -0,0 +1,11 @@ +# Monitoring your websites and APIs with Uptime Kuma + +Uptime Kuma is very straightforward. The UI is very self explanatory for adding monitoring targets and status pages. + +## Installing and updating Uptime Kuma + +Follow instructions: [https://github.com/louislam/uptime-kuma/wiki/%F0%9F%94%A7-How-to-Install](https://github.com/louislam/uptime-kuma/wiki/%F0%9F%94%A7-How-to-Install) + +Its a couple of commands and `pm2` to keep the service running. + +The folloup installation occurs when you first visit your uptime kuma instance.