When a server starts to feel sluggish, most folks jump straight to CPU or disk I/O. Memory is the silent saboteur that slips under the radar until it’s too late. On modern Linux, every service lives in a cgroup, and systemd‑cgtop gives you a live, per‑service view of how that cgroup is eating RAM. Unlike top or ps, it aggregates the numbers for you, so you can spot the offender in a glance.
Kernel updates are a blessing and a curse. They bring new features, hardware support, and security patches, but they also can break proprietary modules that depend on a specific ABI. The NVIDIA driver is a prime example: a new kernel may require a new driver build, and if you’re running a GPU‑heavy workload or a GPU‑accelerated service, an unexpected reboot or a “module not found” error can be costly.
[Read More]The first time I noticed the problem, a Docker‑based microservice was silently timing out on every outbound request. The logs were full of “no servers could be reached” errors, but the host machine could resolve the same hostnames without issue. After a quick check of the container’s /etc/resolv.conf, I saw an odd line: search mydomain.com. The host’s /etc/resolv.conf had no such entry. That single missing search domain was the root of the DNS failure inside containers.
BorgBackup is a popular deduplicating backup tool that works well with remote repositories. In many homelab or small‑business setups the backup repository lives on an NFS share because it’s easy to share across multiple hosts and integrates nicely with existing storage infrastructure. When a single file goes missing or gets corrupted, you want a fast, reliable way to pull it back without pulling the whole archive.
[Read More]When a new kernel lands, the kernel‑specific modules and initramfs changes can
break the ExecStartPre or ExecStart scripts that a timer relies on.
The result? A timer that never fires, or that fires with the wrong environment.
Below is a step‑by‑step guide to diagnose and restore a broken timer, with security‑aware notes and trade‑offs you’ll run into in a production or homelab setup.
[Read More]A quick way to see who’s repeatedly failing to log in via SSH is to pull the relevant lines from /var/log/auth.log and turn them into a CSV. The CSV can then be fed into a spreadsheet, a Grafana dashboard, or a simple shell script that alerts you when a user crosses a threshold. Below is a step‑by‑step recipe that uses only grep and awk, two tools that are guaranteed to be present on any modern Linux distribution.
/etc/hosts suddenly stops being respected after a kernel upgradeWhen a kernel upgrade lands, you expect only low‑level changes. In practice, the upgrade can ripple through the entire userland, especially when systemd‑resolved is involved. A common symptom is that hostname lookups that used to resolve via /etc/hosts now fall back to DNS or fail entirely. The culprit is often a subtle change in how systemd‑resolved loads its configuration or how the Name Service Switch (NSS) library consults /etc/hosts.
If a box has more than one NIC—like a wired LAN that talks to a corporate DNS and a Wi‑Fi that talks to a public ISP—mixing the two can bite you. A query for server will be sent to the DNS server of the interface that sent the request, but the search list that the resolver appends is taken from the global configuration unless you tell it otherwise. That means you can end up asking your public DNS for an internal host or, even worse, leaking internal names to the Internet. Per‑interface search domains keep the resolver tidy and stop accidental leakage.
Kernel upgrades are usually painless, but on a few systems they can trigger a cascade of “carrier lost” messages that make the interface drop and come back up repeatedly. The messages are buried deep in the system journal, so you may not notice them until the network flaps start causing outages. This post shows how to locate those hidden messages with journalctl, correlate them with real‑world symptoms, and apply a fix that keeps the interface stable.
Rebuilding the initramfs is the most reliable way to escape an emergency shell after a kernel upgrade that leaves the system stuck in emergency mode.
The problem usually stems from a mismatch between the running kernel and the modules or hooks that the initramfs contains.
Below is a step‑by‑step guide that covers the common causes, the exact commands you’ll run, and the security‑aware trade‑offs you should keep in mind.
When you install a new kernel on Ubuntu 24.04, GRUB points to the new image, but the initramfs that ships with it must contain:
[Read More]