Tag: self-hosted

  • The Privacy-Friendly Mail Parser You’ve Been Waiting For

    The Privacy-Friendly Mail Parser You’ve Been Waiting For

    As you may or may not know (but now totally do), I have another beloved website, Exploit.to. It’s where I let my inner coder run wild and build all sorts of web-only tools. I’ll save those goodies for another project post, but today, we’re talking about my Mail Parser—a little labor of love born from frustration and an overdose of caffeine.

    See, as a Security Analyst and incident responder, emails are my bread and butter. Or maybe my curse. Parsing email headers manually? It’s a one-way ticket to losing your sanity. And if you’ve ever dealt with email headers, you know they’re basically the Wild West—nobody follows the rules, everyone’s just slapping on whatever they feel like, and chaos reigns supreme.

    The real kicker? Every single EML parser out there at the time was server-side. Let me paint you a picture: you, in good faith, upload that super-sensitive email from your mom (the one where she tells you your laundry’s done and ready for pick-up) to some rando’s sketchy server. Who knows what they’re doing with your mom’s loving words? Selling them? Training an AI to perfect the art of passive-aggressive reminders? The horror!

    So, I thought, “Hey, wouldn’t it be nice if we had a front-end-only EML parser? One that doesn’t send your personal business to anyone else’s server?” Easy peasy, right? Wrong. Oh, how wrong I was. But I did it anyway.

    You can find the Mail Parser here and finally parse those rogue headers in peace. You’re welcome.

    Technologies

    • React: Handles the user interface and dynamic interactions.
    • Astro.js: Used to generate the static website efficiently. (technically not needed for this project)
    • TailwindCSS: For modern and responsive design.
    • ProtonMail’s jsmimeparser: The core library for parsing email headers.

    When I first approached this project, I tried handling email header parsing manually with regular expressions. It didn’t take long to realize how complex email headers have become, with an almost infinite variety of formats, edge cases, and inconsistencies. Regex simply wasn’t cutting it.

    That’s when I discovered ProtonMail’s jsmimeparser, a library purpose-built for handling email parsing. It saved me from drowning in parsing logic and ensured the project met its functional goals.

    Sharing the output of this tool without accidentally spilling personal info all over the place is kinda tricky. But hey, I gave it a shot with a simple empty email I sent to myself:

    The Code

    As tradition dictates, the code isn’t on GitHub but shared right here in a blog post 😁.

    Kidding (sort of). The repo is private, but no gatekeeping here—here’s the code:

    mailparse.tsx
    import React, { useState } from "react";
    import { parseMail } from "@protontech/jsmimeparser";
    
    type Headers = {
      [key: string]: string[];
    };
    
    const MailParse: React.FC = () => {
      const [headerData, setHeaderData] = useState<Headers>({});
      const [ioc, setIoc] = useState<any>({});
    
      function extractEntitiesFromEml(emlContent: string) {
        const ipRegex =
          /\b(?:\d{1,3}\.){3}\d{1,3}\b|\b(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}\b/g;
        const emailRegex = /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/g;
        const urlRegex = /(?:https?|ftp):\/\/[^\s/$.?#].[^\s]*\b/g;
        const htmlTagsRegex = /<[^>]*>/g; // Regex to match HTML tags
    
        // Match IPs, emails, and URLs
        const ips = Array.from(new Set(emlContent.match(ipRegex) || []));
        const emails = Array.from(new Set(emlContent.match(emailRegex) || []));
        const urls = Array.from(new Set(emlContent.match(urlRegex) || []));
    
        // Remove HTML tags from emails and URLs
        const cleanEmails = emails.map((email) => email.replace(htmlTagsRegex, ""));
        const cleanUrls = urls.map((url) => url.replace(htmlTagsRegex, ""));
    
        return {
          ips,
          emails: cleanEmails,
          urls: cleanUrls,
        };
      }
    
      function parseDKIMSignature(signature: string): Record<string, string> {
        const signatureParts = signature.split(";").map((part) => part.trim());
        const parsedSignature: Record<string, string> = {};
    
        for (const part of signatureParts) {
          const [key, value] = part.split("=");
          parsedSignature[key.trim()] = value.trim();
        }
    
        return parsedSignature;
      }
    
      const handleFileChange = async (
        event: React.ChangeEvent<HTMLInputElement>
      ) => {
        const file = event.target.files?.[0];
        if (!file) return;
    
        const reader = new FileReader();
        reader.onload = async (e) => {
          const buffer = e.target?.result as ArrayBuffer;
    
          // Convert the buffer to a string
          const bufferArray = Array.from(new Uint8Array(buffer)); // Convert Uint8Array to number[]
          const bufferString = String.fromCharCode.apply(null, bufferArray);
    
          const { attachments, body, subject, from, to, date, headers, ...rest } =
            parseMail(bufferString);
    
          setIoc(extractEntitiesFromEml(bufferString));
          setHeaderData(headers);
        };
    
        reader.readAsArrayBuffer(file);
      };
    
      return (
        <>
          <div className="p-4">
            <h1>Front End Only Mailparser</h1>
            <p className="my-6">
              Have you ever felt uneasy about uploading your emails to a server you
              don't fully trust? I sure did. It's like handing over your private
              correspondence to a stranger. That's why I decided to take matters
              into my own hands.
            </p>
            <p className="mb-8">
              With this frontend-only mail parser, there's no need to worry about
              your privacy. Thanks to{" "}
              <a
                href="https://proton.me/"
                className="text-pink-500 underline dark:visited:text-gray-400 visited:text-gray-500 hover:font-bold after:content-['_↗']"
              >
                ProtonMail's
              </a>{" "}
              <a
                className="text-pink-500 underline dark:visited:text-gray-400 visited:text-gray-500 hover:font-bold after:content-['_↗']"
                href="https://github.com/ProtonMail/jsmimeparser"
              >
                jsmimeparser
              </a>
              , you can enjoy the same email parsing experience right in your
              browser. No more sending your sensitive data to external servers.
              Everything stays safe and secure, right on your own system.
            </p>
    
            <input
              type="file"
              onChange={handleFileChange}
              className="block w-full text-sm text-slate-500
          file:mr-4 file:py-2 file:px-4
          file:rounded-full file:border-0
          file:text-sm file:font-semibold
          file:bg-violet-50 file:text-violet-700
          hover:file:bg-violet-100
        "
            />
    
            {Object.keys(headerData).length !== 0 && (
              <table className="mt-8">
                <thead>
                  <tr className="border dark:border-white border-black">
                    <th>Header</th>
                    <th>Value</th>
                  </tr>
                </thead>
                <tbody>
                  {Object.entries(headerData).map(([key, value]) => (
                    <tr key={key} className="border dark:border-white border-black">
                      <td>{key}</td>
                      <td>{value}</td>
                    </tr>
                  ))}
                </tbody>
              </table>
            )}
          </div>
    
          {Object.keys(ioc).length > 0 && (
            <div className="mt-8">
              <h2>IPs:</h2>
              <ul>
                {ioc.ips && ioc.ips.map((ip, index) => <li key={index}>{ip}</li>)}
              </ul>
              <h2>Emails:</h2>
              <ul>
                {ioc.emails &&
                  ioc.emails.map((email, index) => <li key={index}>{email}</li>)}
              </ul>
              <h2>URLs:</h2>
              <ul>
                {ioc.urls &&
                  ioc.urls.map((url, index) => <li key={index}>{url}</li>)}
              </ul>
            </div>
          )}
        </>
      );
    };
    
    export default MailParse;

    Yeah, I know, it looks kinda ugly as-is—but hey, slap it into VSCode and let the prettifier work its magic.

    Most of the heavy lifting here is courtesy of the library I used. The rest is just some plain ol’ regex doing its thing—filtering for indicators in the email header and body to make life easier for further investigation.

    Conclusion

    Short and sweet—that’s the vibe here. Sometimes, less is more, right? Feel free to use this tool wherever you like—internally, on the internet, or even on a spaceship. You can also try it out anytime directly on my website.

    Don’t trust me? Totally fair. Open the website, yank out your internet connection, and voilà—it still works offline. No sneaky data sent to my servers, pinky promise.

    As for my Astro.js setup, I include the “mailparse.tsx” like this:

    ---
    import BaseLayout from "../../layouts/BaseLayout.astro";
    import Mailparse from "../../components/mailparse";
    ---
    
    <BaseLayout>
      <Mailparse client:only="react" />
    </BaseLayout>

    See you on the next one. Love you, byeeeee ✌️😘

  • KarlGPT – My Push to Freedom

    KarlGPT – My Push to Freedom

    KarlGPT represents my pursuit of true freedom, through AI. I’ve realized that my ultimate life goal is to do absolutely nothing. Unfortunately, my strong work ethic prevents me from simply slacking off or quietly quitting.

    This led me to the conclusion that I need to maintain, or even surpass, my current level of productivity while still achieving my dream of doing nothing. Given the advancements in artificial intelligence, this seemed like a solvable problem.

    I began by developing APIs to gather all the necessary data from my work accounts and tools. Then, I started working on a local AI model and server to ensure a secure environment for my data.

    Now, I just need to fine-tune the entire system, and soon, I’ll be able to automate my work life entirely, allowing me to finally live my dream: doing absolutely nothing.

    This is gonna be a highly censored post as it involves certain details about my work I can not legally disclose

    Technologies

    Django and Django REST Framework (DRF)

    Django served as the backbone for the server-side logic, offering a robust, scalable, and secure foundation for building web applications. The Django REST Framework (DRF) made it simple to expose APIs with fine-grained control over permissions, serialization, and views. DRF’s ability to handle both function-based and class-based views allowed for a clean, modular design, ensuring the APIs could scale as the project evolved.

    Celery Task Queue

    To handle asynchronous tasks such as sending emails, performing background computations, and integrating external services (AI APIs), I implemented Celery. Celery provided a reliable and efficient way to manage long-running tasks without blocking the main application. This was critical for tasks like scheduling periodic jobs and processing user-intensive data without interrupting the API’s responsiveness.

    React with TypeScript and TailwindCSS

    For the frontend, I utilized React with TypeScript for type safety and scalability. TypeScript ensured the codebase remained maintainable as the project grew. Meanwhile, TailwindCSS enabled rapid UI development with its utility-first approach, significantly reducing the need for writing custom CSS. Tailwind’s integration with React made it seamless to create responsive and accessible components.

    This is my usual front end stack, usually also paired with Astrojs. I use regular React, no extra framework.

    Vanilla Python

    Due to restrictions that prohibited the use of external libraries in local API wrappers, I had to rely on pure Python to implement APIs and related tools. This presented unique challenges, such as managing HTTP requests, data serialization, and error handling manually. Below is an example of a minimal API written without external dependencies:

    import re
    import json
    from http.server import BaseHTTPRequestHandler, HTTPServer
    
    
    items = {"test": "mewo"}
    
    
    class ControlKarlGPT(BaseHTTPRequestHandler):
        def do_GET(self):
            if re.search("/api/helloworld", self.path):
                self.send_response(200)
                self.send_header("Content-type", "application/json")
                self.end_headers()
                response = json.dumps(items).encode()
                self.wfile.write(response)
            else:
                self.send_response(404)
                self.end_headers()
                
    def run(server_class=HTTPServer, handler_class=ControlKarlGPT, port=8000):
        server_address = ("", port)
        httpd = server_class(server_address, handler_class)
        print(f"Starting server on port http://127.0.0.1:{port}")
        httpd.serve_forever()
    
    
    if __name__ == "__main__":
        run()

    By weaving these technologies together, I was able to build a robust, scalable system that adhered to the project’s constraints while still delivering a polished user experience. Each tool played a crucial role in overcoming specific challenges, from frontend performance to backend scalability and compliance with restrictions.

    File based Cache

    To minimize system load, I developed a lightweight caching framework based on a simple JSON file-based cache. Essentially, this required creating a “mini-framework” akin to Flask but with built-in caching capabilities tailored to the project’s needs. While a pull-based architecture—where workers continuously poll the server for new tasks—was an option, it wasn’t suitable here. The local APIs were designed as standalone programs, independent of a central server.

    This approach was crucial because some of the tools we integrate lack native APIs or straightforward automation options. By building these custom APIs, I not only solved the immediate challenges of this project (e.g., powering KarlGPT) but also created reusable components for other tasks. These standalone APIs provide a solid foundation for automation and flexibility beyond the scope of this specific system

    How it works

    The first step was to identify what tasks I perform in the daily and the tools I use for each of them. To automate anything effectively, I needed to abstract these tasks into programmable actions. For example:

    • Read Emails
    • Respond to Invitations
    • Check Tickets

    Next, I broke these actions down further to understand the decision-making process behind each. For instance, when do I respond to certain emails, and how do I determine which ones fall under my responsibilities? This analysis led to a detailed matrix that mapped out every task, decision point, and tool I use.

    The result? A comprehensive, structured overview of my workflow. Not only did this help me build the automation framework, but it also provided a handy reference for explaining my role. If my boss ever asks, “What exactly do you do here?” I can present the matrix and confidently say, “This is everything.”

    As you can see, automating work can be a lot of work upfront—an investment in reducing effort in the future. Ironically, not working requires quite a bit of work to set up! 😂

    The payoff is a system where tasks are handled automatically, and I have a dashboard to monitor, test, and intervene as needed. It provides a clear overview of all ongoing processes and ensures everything runs smoothly:

    AI Magic: Behind the Scenes

    The AI processing happens locally using Llama 3, which plays a critical role in removing all personally identifiable information (PII) from emails and text. This is achieved using a carefully crafted system prompt fine-tuned for my specific job and company needs. Ensuring sensitive information stays private is paramount, and by keeping AI processing local, we maintain control over data security.

    In most cases, the local AI is fully capable of handling the workload. However, for edge cases where additional computational power or advanced language understanding is required, Claude or ChatGPT serve as backup systems. When using cloud-based AI, it is absolutely mandatory to ensure that no sensitive company information is disclosed. For this reason, the system does not operate in full-auto mode. Every prompt is reviewed and can be edited before being sent to the cloud, adding an essential layer of human oversight.

    To manage memory and task tracking, I use mem0 in conjunction with a PostgreSQL database, which acts as the system’s primary “brain” 🧠. This database, structured using Django REST Framework, handles everything from polling for new tasks to storing results. This robust architecture ensures that all tasks are processed efficiently while maintaining data integrity and security.

    Conclusion

    Unfortunately, I had to skip over many of the intricate details and creative solutions that went into making this system work. One of the biggest challenges was building APIs around legacy tools that lack native automation capabilities. Bringing these tools into the AI age required innovative thinking and a lot of trial and error.

    The preparation phase was equally demanding. Breaking down my daily work into a finely detailed matrix took time and effort. If you have a demanding role, such as being a CEO, it’s crucial to take a step back and ask yourself: What exactly do I do? A vague answer like “represent the company” won’t cut it. To truly understand and automate your role, you need to break it down into detailed, actionable components.

    Crafting advanced prompts tailored to specific tasks and scenarios was another key part of the process. To structure these workflows, I relied heavily on frameworks like CO-START and AUTOMAT (stay tuned for an upcoming blog post about these).

    I even created AI personas for the people I interact with regularly and designed test loops to ensure the responses generated by the AI were accurate and contextually appropriate. While I drew inspiration from CrewAI, I ultimately chose LangChain for most of the complex workflows because its extensive documentation made development easier. For simpler tasks, I used lightweight local AI calls via Ollama.

    This project has been an incredible journey of challenges, learning, and innovation. It is currently in an early alpha stage, requiring significant manual intervention. Full automation will only proceed once I receive explicit legal approval from my employer to ensure compliance with all applicable laws, company policies, and data protection regulations.

    Legal Disclaimer: The implementation of any automation or AI-based system in a workplace must comply with applicable laws, organizational policies, and industry standards. Before deploying such systems, consult with legal counsel, relevant regulatory bodies, and your employer to confirm that all requirements are met. Unauthorized use of automation or AI may result in legal consequences or breach of employment contracts. Always prioritize transparency, data security, and ethical considerations when working with sensitive information.

  • Locking Down the Web: How I Secure My WordPress and Other Self-Hosted Public Sites

    Locking Down the Web: How I Secure My WordPress and Other Self-Hosted Public Sites

    Securing a WordPress hosting setup requires more than just the basics—it’s about creating a layered defense to protect your server and adapt to emerging threats. Today I am going to show you what I do to keep Karlcom hosted systems secure from outside attackers.

    Firewall Restriction

    To minimize exposure, my server only accepts traffic from Cloudflare’s IP ranges and only on port 443. This ensures that attackers cannot directly access my server’s IP address, significantly reducing the attack surface.

    On my Firewall it looks like this:

    • One rule to allow Cloudflare
    • One to allow my server to come back in from the internet
    • One block all rule for anything else

    This works pretty well so far.

    Cloudflare’s Web Application Firewall (WAF)

    I leverage Cloudflare’s free WAF to filter out malicious traffic before it reaches my server. It’s an effective first line of defense that helps block known attack patterns and suspicious behavior.

    Here you can find some more Information about it.

    I felt kind of weird sharing my WAF rules here, since you know people reading this can use them to build scans that get around but I figured, I am up for the challenge so lets go:

    (starts_with(http.request.full_uri, "http://10.107.0.150//xmlrpc.php")) or (starts_with(http.request.full_uri, "http://10.107.0.150/xmlrpc.php")) or (ends_with(http.request.uri, "xmlrpc.php")) or (http.request.full_uri contains "setup-config.php") or (http.request.full_uri contains "wp-admin/install.php") or (http.request.uri.path wildcard r"//*")

    This is pretty WordPress specific, I know you can set these on your reverse proxy as well as your wordpress server as well, but I figured letting Cloudflare handle it with their admittedly much more powerful server and taking some steam off of mine would be a good thing to do.

    EDIT:

    While writing this post attacks changed a little and I got some really annoying scans from some IP ranges that all came from Russia, so I ended up Rick Rolling all Russian IPs trying to get through to my home network. Nothing personal.

    Continuous Monitoring with Grafana Labs Loki

    Despite these measures, some scanners and attackers still manage to slip through. To address this, I use Grafana Labs Loki to analyze server logs. By identifying suspicious activity or unusual access paths, I can create new Cloudflare WAF rules to block emerging threats proactively.

    Here you can see some scans from the outside that made it through. I have since updated the WAF rules to block them as well.

    Updates

    As I mentioned in my post about backing up data, I automate the updates for all my LXCs, VMs, and container images. While this approach does carry the risk of introducing breaking changes, the time and effort saved by automating these updates outweigh the potential downsides for me at this stage. Manual maintenance just isn’t practical for my setup right now.

    Since I do daily backups I can recover real fast.

    The Cycle of Security

    This process of monitoring, analyzing, and refining creates an ongoing cycle of security improvements. It’s a proactive and dynamic approach that keeps my server well-protected against evolving threats.

    If you’re using a similar setup or have additional tips for securing WordPress hosting, I’d love to hear your thoughts. Sharing strategies and experiences is one of the best ways to stay ahead of attackers.

    That said, I’m genuinely curious if any attackers reading this will now take it as a challenge to get around my defenses. For that very reason, I stay vigilant, regularly auditing my Grafana logs at home. Security is a constant effort, and in my case, we have SIEM at home, son!

  • My Home Server: “PrettyLittleKitten” – A Personal Tech Haven

    My Home Server: “PrettyLittleKitten” – A Personal Tech Haven

    Hardware

    RAM

    My server is equipped with a total of 128GB of DDR5 RAM, made up of two Kingston FURY Beast kits (each consisting of 2 x 32GB, 6000 MHz, DDR5-RAM, DIMM).

    The RAM operates at around 3600 MHz and consistently maintains 32GB in active usage:

    Cooling

    I kept the fan coolers as they were and opted for an all-in-one liquid cooling solution: the Arctic Liquid Freezer III – 280. No particular reason, really—I just thought it was a cool choice (pun intended).

    PC Case

    This setup was originally intended to be a gaming-only PC, so I chose a sleek and clean-looking case: the Fractal Design North XL. While it’s an aesthetically pleasing choice, the one downside for use as a server is its limited storage capacity.

    CPU

    I chose the AMD Ryzen 7 7800X3D (AM5, 4.20 GHz, 8-Core), which is fantastic for gaming. However, as a server for my needs, I regret that it doesn’t have a better built-in GPU. Intel’s iGPUs are far superior for media transcoding, and using an integrated GPU instead of an external one would save a significant amount of energy.

    GPU

    I do have a dedicated GPU, the ASUS TUF Gaming AMD Radeon RX 7900 XTX OC Edition 24GB, which I chose primarily for its massive VRAM. This allows me to run larger models locally without any issues. However, when it comes to media transcoding, AMD GPUs fall short compared to other options, as highlighted in the Jellyfin – Selecting Appropriate Hardware guide.

    Mainboard

    I chose the MSI MAG B650 TOMAHAWK WIFI (AM5, AMD B650, ATX) as it seemed like a great match for the CPU. However, my GPU is quite large and ends up covering the only other PCI-E x16 slot. This limits my ability to install a decent hardware RAID card or other large expansion cards.

    Storage

    For the main OS, I selected the WD_BLACK SN770 NVMe SSD 2 TB, providing fast and reliable performance. To handle backups and media storage, I added a Seagate IronWolf 12 TB (3.5”, CMR) drive.

    For fast and redundant storage, I set up a ZFS mirror using two Intenso Internal 2.5” SSD SATA III Top, 1 TB drives. This setup ensures that critical data remains safe and accessible.

    Additionally, I included an external Samsung Portable SSD T7, 1 TB, USB 3.2 Gen.2 for extra media storage, rounding out the setup.

    Software

    For my main OS, I stick to what I know best—Proxmox. It’s absolutely perfect for home or small business servers, offering flexibility and reliability in a single package.

    I run a variety of services on it, and the list tends to evolve weekly. Here’s what I’m currently hosting:

    • Nginx Proxy Manager: For managing reverse proxies.
    • n8n: Automation tool for workflows.
    • Bearbot: A production-grade Django app.
    • Vaultwarden: A lightweight password manager alternative.
    • MySpeed: Network speed monitoring.
    • Another Nginx Proxy Manager: Dedicated to managing public-facing apps.
    • Code-Server: A browser-based IDE for developing smaller scripts.
    • Authentik: Single-Sign-On (SSO) solution for all local apps.
    • WordPress: This blog is hosted here.
    • Logs: A comprehensive logging stack including Grafana, Loki, Rsyslog, Promtail, and InfluxDB.
    • Home Assistant OS: Smart home management made easy.
    • Windows 11 Gaming VM: For gaming and other desktop needs.
    • Karlflix: A Jellyfin media server paired with additional tools to keep my media library organized.

    And this list is far from complete—there’s always something new to add or improve!

    Performance

    The core allocation may be displayed incorrectly, but otherwise, this is how my setup looks:

    Here’s the 8-core CPU usage over the last 7 days. As you can see, there’s plenty of headroom, ensuring the system runs smoothly even with all the services I have running:

    Energy costs for my server typically range between 20-25€ per month, but during the summer months, I can run it at 100% capacity during the day using the solar energy generated by my panels. My battery also helps offset some of the power usage during this time.

    Here’s a solid representation of the server’s power consumption:

    I track everything in my home using Home Assistant, which allows me to precisely calculate the energy consumption of each device, including my server. This level of monitoring ensures I have a clear understanding of where my energy is going and helps me optimize usage effectively.

    Conclusion

    Hosting a server locally is a significant investment—both in terms of hardware and energy costs. My setup cost €2405, and I spend about €40 per month on energy, including domain and backup services. While my solar panels make running the server almost free during summer, winter energy costs can be a challenge.

    That said, hosting locally has its advantages. It provides complete control over my data, excellent performance, and the flexibility to upgrade or downgrade hardware as needed. These benefits outweigh the trade-offs for me, even though the energy consumption is higher compared to a Raspberry Pi or Mini-PC.

    I could have gone a different route. A cloud server, or even an alternative like the Apple Mac Mini M4, might have been more efficient in terms of cost and power usage. However, I value upgradability and privacy too much to make those sacrifices.

    This setup wasn’t meticulously planned as a server from the start—it evolved from a gaming PC that was sitting unused. Instead of building a dedicated server from scratch or relying on a Mini-PC and NAS combination, I decided to repurpose what I already had.

    Sure, there are drawbacks. The fans are loud, energy costs add up, and it’s far from the most efficient setup. But for me, the flexibility, control, and performance make it worthwhile. While hosting locally might not be the perfect solution for everyone, it’s the right choice for my needs—and I think that’s what really matters.

  • Inception-Level Data Safety: Backing Up Your Proxmox Backups with Borg on Hetzner

    Inception-Level Data Safety: Backing Up Your Proxmox Backups with Borg on Hetzner

    Today, I want to walk you through how I handle backups for my home server. My primary method is using Proxmox’s built-in backup functionality, which I then sync to a Hetzner Storage Box for added security.

    When it comes to updates, I like to live on the edge. I enable automatic (security) updates on nearly all of my systems at home using UnattendedUpgrades. For containers, I usually deploy a Watchtower instance to keep them updated automatically. While this approach might make some people nervous—fearing a broken system after an update—I don’t sweat it. I back up daily and don’t run any mission-critical systems at home (except for this blog, of course 😉).

    For specific files or directories, like Vaultwarden, I take an extra layer of precaution by creating additional backups within the LXC container itself. These backups are synced to a Nextcloud instance I also manage through Hetzner, but in a different datacenter. Hetzner’s “Storage Shares” offer a great deal—€5 gets you 1TB of managed Nextcloud storage. While not the fastest, they’re reliable enough for my needs.

    I won’t dive into the details here, but my approach for these backups is pretty straightforward: I use ZIP files and rclone to upload everything to Nextcloud.

    Here is my script, maybe it helps you in some way:

    #!/bin/bash
    
    # Variables
    BITWARDEN_DIR="/root/bitwarden"
    BACKUP_DIR="/root/bitwarden-backup"
    NEXTCLOUD_REMOTE="nextcloud:Vaultwarden"
    TIMESTAMP=$(date '+%Y%m%d-%H%M')
    
    # Ensure backup directory exists
    mkdir -p $BACKUP_DIR
    
    # Create a single tarball of the entire Vaultwarden directory
    echo "Creating a full backup of the Vaultwarden directory..."
    tar -czvf $BACKUP_DIR/vaultwarden_full_backup-${TIMESTAMP}.tar.gz -C $BITWARDEN_DIR .
    
    # Sync the backup to Nextcloud
    echo "Uploading backup to Nextcloud..."
    rclone copy $BACKUP_DIR $NEXTCLOUD_REMOTE
    
    # Clean up local backup directory
    echo "Cleaning up local backups..."
    rm -rf $BACKUP_DIR
    
    echo "Backup completed successfully!"

    Basically, all you need to do is create an App Password and follow the Rclone guide for setting up with WebDAV. It’s straightforward and works seamlessly for this kind of setup.

    Backups in Proxmox

    Proxmox makes backups incredibly simple with its intuitive functionality. I back up pretty much everything—except for my Gaming VM. It’s a Windows 11 experiment where I’ve passed through my AMD RX7900XT for gaming. Ironically, instead of gaming, I end up spending more time tweaking backups and writing about them. Let’s just say that gaming setup hasn’t exactly gone as planned.

    I rely on Snapshot mode for my backups, and you can explore all its features and settings right here. As I mentioned earlier, I tend to restore backups more frequently than most people, and I’ve never faced any issues with inconsistencies. It’s been consistently reliable for me!

    For retention, I keep it straightforward by saving only the last two backups. Since I also back up my backups (as you’ll see later), this minimalist approach is more than sufficient for my needs and saves me some space.

    I left the rest of the settings as they are. The note templates are useful if you’re managing a large or multiple instances, but for my setup, I don’t use them.

    Trigger warning: For now, I’m storing these backups on a single internal Seagate IronWolf (12 TB). I know, not ideal. These drives are pretty pricey, but one day I plan to add another and set up a ZFS mirror or RAID for better redundancy. For now, I’m relying on this one drive—fingers crossed, it’s been rock solid so far!

    Borg(Backup)

    The first thing I heard when I proudly told my friends that I was finally taking the golden 3-2-1 backup rule seriously was: “Why not restic?”

    The simple answer? I Googled “backups to Hetzner Storage Box,” and the first result was an article explaining exactly what I wanted to do—using Borg 🤷‍♂️. Before I even considered trying restic, I had already set up encrypted incremental backups with Borg. Feel free to share what you use and why you might have switched, but for now, this setup works perfectly for me!

    Hetzner Storage Box

    Just to clarify, I’m not talking about Hetzner Storage Share 😁. I’m using their 5TB Storage Box and opted for Finland 🇫🇮 as the location since I already have other Karlcom-related stuff in their German datacenter. It helps keep things spread out a bit!

    Essentially, it’s a big, affordable storage backend with multiple options for uploading data. You could mount it using the “Samba/CIFS” option, but I decided against that. Instead, I went with a more secure SSH connection to send my backups there.

    Setup

    First, you’ll need to upload your SSH key to the Hetzner Storage Box. You can follow this step by step guide.

    Once that’s done, the next step is to install and Configure BorgBackup, which you can also follow the simple guide I linked to.

    I know, it seems like you came here just to find links to set this up somewhere else. But don’t worry—I’ve got some cool stuff to share with you next. Here’s my backup script:

    /usr/local/bin/proxmox_borg_backup.sh
    #!/bin/bash
    
    # Variables
    BORG_REPO="ssh://[email protected]:23/home/backups/central"
    
    BORG_PASSPHRASE=''
    BACKUP_SOURCE="/mnt/pve/wd_hdd_internal/dump"                               
    LOG_FILE="/var/log/proxmox_borg_backup.log"                                 
    MAX_LOG_SIZE=10485760
    RID=`uuidgen`
    CHECK_ID="ggshfo8-9ca6-1234-1234-326571681"
    
    # start
    curl -fsS -m 10 --retry 5 "https://ping.yourdomain.de/ping/$CHECK_ID/start?rid=$RID"
    
    # Export Borg passphrase
    export BORG_PASSPHRASE
    
    # Rotate log file if it exceeds MAX_LOG_SIZE
    if [ -f "$LOG_FILE" ] && [ $(stat -c%s "$LOG_FILE") -gt $MAX_LOG_SIZE ]; then
        mv "$LOG_FILE" "${LOG_FILE}_$(date +"%Y-%m-%d_%H-%M-%S")"
        touch "$LOG_FILE"
    fi
    
    # Check for BorgBackup installation
    if ! command -v borg &> /dev/null; then
        echo "ERROR: BorgBackup is not installed or not in PATH." >> "$LOG_FILE"
        exit 1
    fi
    
    # Check for SSH connection
    if ! ssh -q -o BatchMode=yes -o ConnectTimeout=5 -p 23 -i ~/.ssh/backup u123456@ u123456.your-storagebox.de exit; then
        echo "ERROR: Unable to connect to Borg repository." >> "$LOG_FILE"
        exit 1
    fi
    
    # Logging start time
    {
      echo "==== $(date +"%Y-%m-%d %H:%M:%S") Starting Proxmox Backup ===="
    
      # Check if the backup source exists
      if [ ! -d "$BACKUP_SOURCE" ]; then
          echo "ERROR: Backup source directory $BACKUP_SOURCE does not exist!"
          exit 1
      fi
    
      # Create a new Borg backup
      echo "Creating Borg backup..."
      borg create --stats --compression zstd \
          "$BORG_REPO::backup-{now:%Y-%m-%d}" \
          "$BACKUP_SOURCE" >> "$LOG_FILE" 2>&1
    
    
      if [ $? -ne 0 ]; then
          echo "ERROR: Borg backup failed!"
          exit 1
      fi
    
      # Prune old backups to save space
      echo "Pruning old backups..."
      borg prune --stats \
          --keep-daily=7 \
          --keep-weekly=4 \
          --keep-monthly=6 \
          "$BORG_REPO"
    
      if [ $? -ne 0 ]; then
          echo "ERROR: Borg prune failed!"
          exit 1
      fi
    
      echo "==== $(date +"%Y-%m-%d %H:%M:%S") Proxmox Backup Completed ===="
    } >> "$LOG_FILE" 2>&1
    
    # finished
    curl -fsS -m 10 --retry 5 "https://ping.yourdomain.de/ping/$CHECK_ID?rid=$RID"

    The curl requests at the top and bottom of the script are for my Healthchecks.io instance—I even wrote a blog post about it here.

    Before moving on, you should definitely test this script. Depending on the size of your setup, the initial backup could take several hours. However, if it doesn’t fail within the first 10 seconds, that’s usually a good sign. To be sure it’s running smoothly, check the log file to confirm it started correctly:

    /var/log/proxmox_borg_backup.log
    ==== 2025-01-10 01:39:07 Starting Proxmox Backup ====
    Creating Borg backup...
    ------------------------------------------------------------------------------
    Repository: ssh://u123456@ u123456.your-storagebox.de:23/home/backups/central
    Archive name: backup-2025-01-10
    Archive fingerprint: z724gf2789hgf972hf9uh...
    Time (start): Fri, 2025-01-10 01:39:08
    Time (end):   Fri, 2025-01-10 05:36:41
    Duration: 3 hours 57 minutes 32.92 seconds
    Number of files: 72
    Utilization of max. archive size: 0%
    ------------------------------------------------------------------------------
                           Original size      Compressed size    Deduplicated size
    This archive:               62.03 GB             61.98 GB             61.60 GB
    All archives:               62.03 GB             61.98 GB             61.60 GB
    
                           Unique chunks         Total chunks
    Chunk index:                   24030                40955
    ------------------------------------------------------------------------------
    Pruning old backups...
    ------------------------------------------------------------------------------
                           Original size      Compressed size    Deduplicated size
    Deleted data:                    0 B                  0 B                  0 B
    All archives:               62.03 GB             61.98 GB             61.60 GB
    
                           Unique chunks         Total chunks
    Chunk index:                   24030                40955
    ------------------------------------------------------------------------------
    ==== 2025-01-10 05:36:42 Proxmox Backup Completed ====

    Security of BORG_PASSPHRASE

    I decided to include the passphrase for encryption and decryption directly in the script because it fits within my threat model. My primary concern isn’t someone gaining access to my local Proxmox server and restoring or deleting my backups—my focus is on protecting against snooping by cloud providers or malicious admins.

    Having the passphrase in the script works for me. Sure, there are other ways to handle this, but for the script to run automatically, you’ll always need to store the passphrase somewhere on your system. At the very least, it has to be accessible by root. This setup strikes the right balance for my needs.

    Systemd timers

    I created a system service to handle this backup process. For long-running jobs, it’s generally better to use systemd timers instead of cron, as they’re less prone to timeouts. I found this post particularly helpful when setting it up.

    Here’s the service that actually runs my bash script:

    /etc/systemd/system/proxmox_borg_backup.service
    [Unit]
    Description=Proxmox BorgBackup Service
    After=network.target
    
    [Service]
    Type=oneshot
    ExecStart=/usr/local/bin/proxmox_borg_backup.sh

    And here’s the systemd timer that handles scheduling the service:

    /etc/systemd/system/proxmox_borg_backup.timer
    [Unit]
    Description=Run Proxmox BorgBackup Daily at 3 AM
    
    [Timer]
    OnCalendar=*-*-* 03:00:00
    Persistent=true
    
    [Install]
    WantedBy=timers.target

    Now, instead of enabling the service directly, you enable and start the timer. The timer will take care of starting the service according to the schedule you’ve defined. This setup ensures everything runs smoothly and on time!

    Bash
    systemctl enable proxmox_borg_backup.timer
    systemctl start proxmox_borg_backup.timer 
    systemctl status proxmox_borg_backup.timer

    That’s it! You’re all set. You can check the log file we created or use the journalctl command to review any errors or confirm successful runs. Happy backing up! 🎉

    Bash
    journalctl -xeu proxmox_borg_backup.timer
    
    # or 
    
    tail -n 50 /var/log/proxmox_borg_backup.log

    Conclusion

    You should now have an easy and efficient solution to back up your Proxmox backups to a Hetzner Storage Box using Borg Backup. Both Borg and Restic support a variety of storage targets, so you can adapt this approach to suit your needs. In my setup, Borg performs incremental backups, uploading only new data, which helps keep storage costs low while maintaining security.

    A word of caution: don’t lose your secrets—your encryption key or passphrase—because without them, you won’t be able to restore your data. Trust me, I’ve been there before! Thankfully, I had local backups to fall back on.

    On Hetzner, I schedule daily backups at noon, after all my backup jobs have completed. I retain only the last three days, which works perfectly for me, though your needs might differ. Just remember that snapshot storage counts toward your total storage capacity—so if you have 1TB, the space used by snapshots will reduce the available storage for new data.

    Thank you for reading! May your backups always be safe, your disks last long, and your systems run smoothly. Wishing you all the best—love you, byeeeeee! ❤️🚀

  • Before “PrettyLittleKitten”: A Brief Affair with the Mac Mini M4

    Before “PrettyLittleKitten”: A Brief Affair with the Mac Mini M4

    Before I built my beloved server, affectionately named “PrettyLittleKitten“, I had a brief fling with the brand-new Mac Mini M4. Spoiler alert: it was a short-lived relationship.

    Let me start with the good stuff: processing power-to-power usage ratio. It’s absolutely unmatched. The Mac Mini M4 is a beast in terms of efficiency—an essential factor for me. I wanted hardware that could handle Jellyfin with smooth hardware acceleration while still hosting all my containers.

    The Hardware

    On paper (and in practice as a desktop), the Mac Mini M4 shines. It offers:

    • 4 Thunderbolt USB-C ports, making storage expansion a breeze. Pair it with an external NVMe enclosure, and you can achieve speeds close to that of internal storage.
    • Hardware that punches way above its price point, making it a reasonable investment for many use cases.

    The Disappointment

    Here’s where the romance fell apart. While the Mac Mini M4 is brilliant as a desktop, using it as a server is a whole different ball game—and not a fun one.

    The iCloud Conundrum

    First up: the dreaded iCloud account requirement. This wasn’t a total shock (it’s Apple, after all), but it made me long for the simplicity of Debian and Proxmox, where everything is blissfully offline.

    I went ahead and set it up with my personal iCloud account—big mistake. To run the Mac Mini as I wanted, it needed to stay logged in indefinitely. And here’s the kicker: to achieve that, I had to disable authentication entirely. Translation? If anyone got their hands on my Mini, they’d have full access to my iCloud account. Yikes.

    Pro tip: Use a burner iCloud account if you’re planning to go down this route. (Is this what you want, Apple?!)

    Dummy HDM

    Then there’s the issue of fooling the Mac into thinking it’s doing desktop work. Without a connected display, macOS doesn’t fully utilize the GPU or cores, which impacts performance. Enter the Dummy HDMI Plug—a little device to trick the system into thinking a monitor is attached. At ~€40, it’s not a dealbreaker, but definitely annoying.

    Power Saving Woes

    You’ll also need to disable power-saving features. While the Mac Mini M4 consumes very little power in idle, turning off power-saving negates some of its efficiency benefits.

    Recap of Mac Mini Server Challenges

    If you’re still tempted to use the Mac Mini M4 as a server, here’s your checklist:

    • Dummy HDMI Plug: €40 (because macOS needs to “see” a monitor).
    • Burner iCloud Account: Necessary to avoid risking your real account.
    • Disable Authentication: Say goodbye to security.
    • Disable Power Saving: Because macOS doesn’t believe in idle servers.

    Final Thoughts

    If you’re determined, Evan Bartlett has written an excellent guide on setting up the Mac Mini as a server. However, as someone coming from the Linux world—where operating systems are designed for server use—it just didn’t feel right. Forcing macOS, an OS that clearly does not want to be a server, felt morally and ethically wrong.

    Here’s hoping Big Siri AI will be kind to me when it inevitably takes over. 🙇‍♂️🍏

    Bonus: Check this website’s response headers to see that it runs on PrettyLittleKitten

  • Why HedgeDoc Reigns as the King of Self-Hosted Note-Taking Apps

    Why HedgeDoc Reigns as the King of Self-Hosted Note-Taking Apps

    This is going to be a bold, highly opinionated take on how note-taking apps should be. For the non-technical folks, discussing text editors and note-taking apps with IT people is like walking straight into a heated geopolitical debate at the family Thanksgiving table—it’s passionate, intense, and probably never-ending. Gobble Gobble.

    I have tested a lot of note taking apps:

    There are probably even more apps I have used in the past, but these are the ones that left a lasting impression on me. First off, let me just say—I love taking notes in Markdown. Any app that doesn’t support Markdown is pretty much useless to me. I’m so much faster at writing styled notes this way, without the hassle of clicking around or memorizing weird shortcut commands.

    For me, HedgeDoc hit the sweet spot. It’s got just the right features and just the right amount of organization. I’m not looking for an app to micromanage my entire life—I just want to take some damn notes!

    Live editing has also become a game-changer for me. I often have multiple screens open, sometimes even on different networks, and being instantly up-to-date while copy-pasting seamlessly between them is invaluable. Before HedgeDoc, I was using Obsidian synced via Nextcloud, but that was neither instant nor reliable on many networks.

    And let’s talk about security. With HedgeDoc, it’s a breeze. Their authorization system is refreshingly simple, and backing up your notes is as easy as clicking a button. You get a ZIP file with all your Markdown documents, which you could technically use with other editors—but why would you? HedgeDoc feels like it was made for you, and honestly, you’ll feel the love right back.

    I run HedgeDoc inside a container on my server, and it’s rock-solid. It just works. No excessive resource use, no drama—just a tool that quietly does its job.

    Now, let’s dive in! I’m going to show you how to host HedgeDoc yourself. Let’s get started!

    Prerequisites

    Here’s what you’ll need to get started:

    • A Linux distribution: Any modern Linux distro that supports Docker will work, but for today, we’ll go with Alpine.
    • A server with a public IP address: While not strictly mandatory, this is highly recommended if you want to access your note-taking app from anywhere.
    • A reverse proxy: Something like Caddy or Nginx to handle HTTPS and make your setup accessible and secure.

    Got all that? Great—let’s get started!

    Setup

    Here’s a handy script to install Docker on a fresh Alpine setup:

    init.sh
    #!/bin/sh
    
    # Exit on any error
    set -e
    
    echo "Updating repositories and installing prerequisites..."
    cat <<EOF > /etc/apk/repositories
    http://dl-cdn.alpinelinux.org/alpine/latest-stable/main
    http://dl-cdn.alpinelinux.org/alpine/latest-stable/community
    EOF
    
    apk update
    apk add --no-cache curl openrc docker docker-compose
    
    echo "Configuring Docker to start at boot..."
    rc-update add docker boot
    service docker start
    
    echo "Verifying Docker installation..."
    docker --version
    if [ $? -ne 0 ]; then
        echo "Docker installation failed!"
        exit 1
    fi
    
    echo "Verifying Docker Compose installation..."
    docker-compose --version
    if [ $? -ne 0 ]; then
        echo "Docker Compose installation failed!"
        exit 1
    fi
    
    echo "Docker and Docker Compose installed successfully!"

    To make the script executable and run it, follow these steps:

    Bash
    chmod +x init.sh
    ./init.sh

    If everything runs without errors, Docker should now be installed and ready to go. 🎉

    To install HedgeDoc, we’ll follow the steps from their official documentation. It’s straightforward and easy

    I prefer to keep all my environment variables and secrets neatly stored in .env files, separate from the actual Compose file.

    .env
    POSTGRES_USER=hedgedoctor
    POSTGRES_PASSWORD=super_secure_password
    POSTGRES_DB=hedgedoc
    
    CMD_DB_URL=postgres://hedgedoctor:super_secure_password@database:5432/hedgedoc
    CMD_ALLOW_FREEURL=true
    CMD_DOMAIN=docs.yourdomain.de
    CMD_PROTOCOL_USESSL=true
    CMD_ALLOW_ANONYMOUS=false
    CMD_ALLOW_EMAIL_REGISTER=true # <- remove after you registered

    To keep things secure, it’s a good idea to set CMD_ALLOW_ANONYMOUS to false, so anonymous users can’t edit your documents. For added security, you can create your own account and then disable CMD_ALLOW_EMAIL_REGISTER to prevent outsiders from signing up, effectively locking down HedgeDoc.

    One great benefit of using the env_file directive in your Docker Compose setup is that it keeps your Compose files clean and tidy:

    docker-compose.yml
    services:
      database:
        image: postgres:13.4-alpine
        env_file:
          - .env
        volumes:
          - database:/var/lib/postgresql/data
        restart: always
    
      app:
        image: quay.io/hedgedoc/hedgedoc:latest
        env_file:
          - .env
        volumes:
          - uploads:/hedgedoc/public/uploads
        ports:
          - "3000:3000"
        restart: always
        depends_on:
          - database
    
    volumes:
      database:
      uploads:

    After running docker compose up -d, you should be all set! This setup assumes you already have a reverse proxy configured and pointing to the public domain where you’re hosting your HedgeDoc. If you need help setting that up, I’ve written a guide on it in another blog post.

    Keep in mind, with the settings in the .env file above, HedgeDoc won’t work unless it’s served via HTTPS through the reverse proxy using the domain you specified.

    Once everything’s in place, you should see the HedgeDoc login screen and be able to “Register” your account:

    Don’t forget to head back to your .env file and comment out that specific line once you’re done:

    .env
    ...
    # CMD_ALLOW_EMAIL_REGISTER=true # <- remove after you registered

    This ensures that no one else can create accounts on your HedgeDoc instance.

    Personally, I always set my notes to “Private” (you can do this in the top right). That way, even if I decide to let others use the instance later, I don’t have to worry about any old notes where I might have called them a stinky doodoo face (as one does):

    You can still share your documents with others, but you’ll need to change the setting to “Locked.” Anything more restrictive will prevent people from viewing your notes.

    Imagine sending your crush a beautifully crafted, markdown-styled love letter, only for them to get blocked because of your overly strict settings. Yeah… couldn’t be me.

    Conclusion

    I conclude —our notes are ready, no need for more WordPress blog posts. Now it’s time to hit the gym because it’s chest day, and let’s be honest, chest day is the best day! 💪

  • Effortless Cron Job Monitoring: A Guide to Self-Hosting with Healthchecks.io

    Effortless Cron Job Monitoring: A Guide to Self-Hosting with Healthchecks.io

    Do you ever find yourself lying awake at night, staring at the ceiling, wondering if your beloved cronjobs ran successfully? Worry no more! Today, we’re setting up a free, self-hosted solution to ensure you can sleep like a content little kitten 🐱 from now on.

    I present to you Healthchecks.io. According to their website:

    Simple and Effective Cron Job Monitoring

    We notify you when your nightly backups, weekly reports, cron jobs, and scheduled tasks don’t run on time.

    How to monitor any background job:

    1. On Healthchecks.io, generate a unique ping URL for your background job.
    2. Update your job to send an HTTP request to the ping URL every time the job runs.
    3. When your job does not ping Healthchecks.io on time, Healthchecks.io alerts you!

    Today, we’re taking the super easy, lazy-day approach by using their Docker image. They’ve provided a well-documented, straightforward guide for deploying it right here: Running with Docker.

    What I love most about Healthchecks.io? It’s built on Django, my all-time favorite Python web framework. Sorry, FastAPI—you’ll always be cool, but Django has my heart!

    Prerequisites:

    1. A Server: You’ll need a server to host your shiny new cronjob monitor. A Linux distro is ideal.
    2. Docker & Docker Compose: Make sure these are installed. If you’re not set up yet, here’s the guide.
    3. Bonus Points: Having a domain or subdomain, along with a public IP, makes it accessible for all your systems.

    You can run this on your home network without any hassle, although you might not be able to copy and paste all the code below.

    Need a free cloud server? Check out Oracle’s free tier—it’s a decent option to get started. That said, in my experience, their free servers are quite slow, so I wouldn’t recommend them for anything mission-critical. (Not sponsored, pretty sure they hate me 🥺.)

    Setup

    I’m running a Debian LXC container on my Proxmox setup with the following specs:

    • CPU: 1 core
    • RAM: 1 GB
    • Swap: 1 GB
    • Disk: 10 GB (NVMe SSD)

    After a month of uptime, these are the typical stats: memory usage stays pretty consistent, and the boot disk is mostly taken up by Docker and the image. As for the CPU? It’s usually just sitting there, bored out of its mind.

    First, SSH into your server, and let’s get started by creating a .env file to store all your configuration variables:

    .env
    PUID=1000
    PGID=1000
    APPRISE_ENABLED=True
    TZ=Europe/Berlin
    SITE_ROOT=https://ping.yourdomain.de
    SITE_NAME=Healthchecks
    ALLOWED_HOSTS=ping.yourdomain.de
    CSRF_TRUSTED_ORIGINS=https://ping.yourdomain.de
    DEBUG=False
    SECRET_KEY=your-secret-key

    In your .env file, enter the domain you’ll use to access the service. I typically go with something simple, like “ping” or “cron” as a subdomain. If you want to explore more configuration options, you can check them out here.

    For my setup, this basic configuration does the job perfectly.

    To generate secret keys, I usually rely on the trusty openssl command. Here’s how you can do it:

    Bash
    openssl rand -base64 64
    docker-compose.yml
    services:
      healthchecks:
        image: lscr.io/linuxserver/healthchecks:latest
        container_name: healthchecks
        env_file:
          - .env
        volumes:
          - ./config:/config
        ports:
          - 8083:8000
        restart: unless-stopped

    All you need to do now is run:

    Bash
    docker compose -up

    That’s it—done! 🎉

    Oh, and by the way, I’m not using the original image for this. Instead, I went with the Linuxserver.io variant. There is no specific reason for this —just felt like it! 😄

    Important!

    Unlike the Linuxserver.io guide, I skipped setting the superuser credentials in the .env file. Instead, I created the superuser manually with the following command:

    Bash
    docker compose exec healthchecks python /app/healthchecks/manage.py createsuperuser

    This allows you to set up your superuser interactively and securely directly within the container.

    If you’re doing a standalone deployment, you’d typically set up a reverse proxy to handle SSL in front of Healthchecks.io. This way, you avoid dealing with SSL directly in the app. Personally, I use a centralized Nginx Proxy Manager running on a dedicated machine for all my deployments. I’ve even written an article about setting it up with SSL certificates—feel free to check that out!

    Once your site is served through the reverse proxy over the domain you specified in the configuration, you’ll be able to access the front end using the credentials you created with the createsuperuser command.

    There are plenty of guides for setting up reverse proxies, and if you’re exploring alternatives, I’m also a big fan of Caddy—it’s simple, fast, and works like a charm!

    Here is a finished Docker Compose file with Nginx Proxy Manager:

    docker-compose.yml
    services:
      npm:
        image: 'jc21/nginx-proxy-manager:latest'
        container_name: nginx-proxy-manager
        restart: unless-stopped
        ports:
          - '443:443'
          - '81:81'
        volumes:
          - ./npm/data:/data
          - ./npm/letsencrypt:/etc/letsencrypt
    
      healthchecks:
        image: lscr.io/linuxserver/healthchecks:latest
        container_name: healthchecks
        env_file:
          - .env
        volumes:
          - ./healthchecks/config:/config
        restart: unless-stopped

    In Nginx Proxy Manager your proxied host would be “http://healthchecks:8000”

    If you did not follow my post you will need to expose port 80 on the proxy as well for “regular” Let’s Encrypt certificates without DNS challenge.

    Healthchecks.io

    If you encounter any errors while trying to access the UI of your newly deployed Healthchecks, the issue is most likely related to the settings in your .env file. Double-check the following to ensure they match your domain configuration:

    .env
    SITE_ROOT=https://ping.yourdomain.de
    ALLOWED_HOSTS=ping.yourdomain.de
    CSRF_TRUSTED_ORIGINS=https://ping.yourdomain.de

    Once you’re in, the first step is to create a new project. After that, let’s set up your first simple check.

    For this example, I’ll create a straightforward uptime monitor for my WordPress host. I’ll set up a cronjob that runs every hour and sends an “alive” ping to my Healthchecks.io instance.

    The grace period is essential to account for high latency. For instance, if my WordPress host is under heavy load, an outgoing request might take a few extra seconds to complete. Setting an appropriate grace period ensures that occasional delays don’t trigger false alerts.

    I also prefer to “ping by UUID”. Keeping these endpoints secret is crucial—if someone else gains access to your unique ping URL, they could send fake pings to your Healthchecks.io instance, causing you to miss real downtimes.

    Click on the Usage Example button in your Healthchecks.io dashboard to find ready-to-use, copy-paste snippets for various languages and tools. For this setup, I’m going with bash:

    Bash
    curl -m 10 --retry 5 https://ping.yourdomain.de/ping/67162f7b-5daa-4a31-8667-abf7c3e604d8
    • -m sets the max timeout to 10 seconds. You can change the value but do not leave this out!
    • –retry says it should retry the request 5 times before aborting.

    Here’s how you can integrate it into a crontab:

    Bash
    # A sample crontab entry. Note the curl call appended after the command.
    # FIXME: replace "/your/command.sh" below with the correct command!
    0 * * * * /your/command.sh && curl -fsS -m 10 --retry 5 -o /dev/null https://ping.yourdomain.de/ping/67162f7b-5daa-4a31-8667-abf7c3e604d8
    

    To edit your crontab just run:

    Bash
    crontab -e

    The curl command to Healthchecks.io will only execute if command.sh completes successfully without any errors. This ensures that you’re notified only when the script runs without issues.

    After you ran that command, your dashboard should look like this:

    Advanced Checks

    While this is helpful, you might often need more detailed information, such as whether the job started but didn’t finish or how long the job took to complete.

    Healthchecks.io provides all the necessary documentation built right into the platform. You can visit /docs/measuring_script_run_time/ on your instance to find fully functional examples.

    Bash
    #!/bin/sh
    
    RID=`uuidgen`
    CHECK_ID="67162f7b-5daa-4a31-8667-abf7c3e604d8"
    
    # Send a start ping, specify rid parameter:
    curl -fsS -m 10 --retry 5 "https://ping.yourdomain.de/ping/$CHECK_ID/start?rid=$RID"
    
    # Put your command here
    /usr/bin/python3 /path/to/a_job_to_run.py
    
    # Send the success ping, use the same rid parameter:
    curl -fsS -m 10 --retry 5 "https://ping.yourdomain.de/ping/$CHECK_ID?rid=$RID"

    As you can see here this will give me the execution time as well:

    Here, I used a more complex cron expression. To ensure it works as intended, I typically rely on Crontab.guru for validation. You can use the same cron expression here as in your local crontab. The grace period depends on how long you expect the job to run; in my case, 10 seconds should be sufficient.

    Notifications

    You probably don’t want to find yourself obsessively refreshing the dashboard at 3 a.m., right? Ideally, you only want to be notified when something important happens.

    Thankfully, Healthchecks.io offers plenty of built-in notification options. And for even more flexibility, we enabled Apprise in the .env file earlier, unlocking a huge range of additional integrations.

    For notifications, I usually go with Discord or Node-RED, since they work great with webhook-based systems.

    While you could use Apprise for Discord notifications, the simplest route is to use the Slack integration. Here’s the fun part: Slack and Discord webhooks are fully compatible, so you can use the Slack integration to send messages directly to your Discord server without any extra configuration!

    This way, you’re only disturbed when something really needs your attention—and it’s super easy to set up.

    Discord already provides an excellent Introduction to Webhooks that walks you through setting them up for your server, so I won’t dive into the details here.

    All you need to do is copy the webhook URL from Discord and paste it into the Slack integration’s URL field in Healthchecks.io. That’s it—done! 🎉

    With this simple setup, you’ll start receiving notifications directly in your Discord server whenever something requires your attention. Easy and effective!

    On the Discord side it will look like this:

    With this setup, you won’t be bombarded with notifications every time your job runs. Instead, you’ll only get notified if the job fails and then again when it’s back up and running.

    I usually prefer creating dedicated channels for these notifications to keep things organized and avoid spamming anyone:

    EDIT:

    I ran into some issues with multiple Slack notifications in different projects. If you get 400 errors just use Apprise. The Discord URL would look like this:

    discord://{WebhookID}/{WebhookToken}/
    
    for example:
    
    discord://13270700000000002/V-p2SweffwwvrwZi_hc793z7cubh3ugi97g387gc8svnh

    Status Badges

    In one of my projects, I explained how I use SVG badges to show my customers whether a service is running.

    Here’s a live badge (hopefully it’s still active when you see this):

    bearbot

    Getting these badges is incredibly easy. Simply go to the “Badges” tab in your Healthchecks.io dashboard and copy the pre-generated HTML to embed the badge on your website. If you’re not a fan of the badge design, you can create your own by writing a custom JavaScript function to fetch the status as JSON and style it however you like.

    Here is a code example:

    HTML
    <style>
        .badge {
            display: inline-block;
            padding: 10px 20px;
            border-radius: 5px;
            color: white;
            font-family: Arial, sans-serif;
            font-size: 16px;
            font-weight: bold;
            text-align: center;
            box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
            transition: background-color 0.3s ease;
        }
        .badge.up {
            background-color: #28a745; /* Green for "up" */
        }
        .badge.down {
            background-color: #dc3545; /* Red for "down" */
        }
        .badge.grace {
            background-color: #ffc107; /* Yellow for "grace" */
        }
    </style>
    </head>
    <body>
    <div id="statusBadge" class="badge">Loading...</div>
    
    <script>
        async function updateBadge() {
            // replace this 
            const endpoint = "https://ping.yourdmain.de/badge/XXXX-XXX-4ff6-XXX-XbS-2.json"
            const interval = 60000
            // ---
            
            try {
                const response = await fetch(endpoint);
                const data = await response.json();
    
                const badge = document.getElementById('statusBadge');
                badge.textContent = `Status: ${data.status.toUpperCase()} (Total: ${data.total}, Down: ${data.down})`;
    
                badge.className = 'badge';
                if (data.status === "up") {
                    badge.classList.add('up');
                } else if (data.status === "down") {
                    badge.classList.add('down');
                } else if (data.status === "grace") {
                    badge.classList.add('grace');
                }
            } catch (error) {
                console.error("Error fetching badge data:", error);
                const badge = document.getElementById('statusBadge');
                badge.textContent = "Error fetching data";
                badge.className = 'badge down';
            }
        }
    
        updateBadge();
        setInterval(updateBadge, interval);
    </script>
    </body>

    The result:

    It might not look great, but the key takeaway is that you can customize the style to fit seamlessly into your design.

    Conclusion

    We’ve covered a lot of ground today, and I hope you now have a fully functional Healthchecks.io setup. No more sleepless nights worrying about whether your cronjobs ran successfully!

    So, rest easy and sleep tight, little kitten 🐱—your cronjobs are in good hands now.

  • Bearbot.dev – The Final Form

    Bearbot.dev – The Final Form

    I wanted to dedicate a special post to the new version of Bearbot. While I’ve already shared its history in a previous post, it really doesn’t capture the full extent of the effort I’ve poured into this update.

    The latest version of Bearbot boasts a streamlined tech stack designed to cut down on tech debt, simplify the overall structure, and laser-focus on its core mission: raking in those sweet stock market gains 🤑.

    Technologie

    • Django (Basic, not DRF): A high-level Python web framework that simplifies the development of secure and scalable web applications. It includes built-in tools for routing, templates, and ORM for database interactions. Ideal for full-stack web apps without a heavy focus on APIs.
    • Postgres: A powerful, open-source relational database management system known for its reliability, scalability, and advanced features like full-text search, JSON support, and transactions.
    • Redis: An in-memory data store often used for caching, session storage, and real-time messaging. It provides fast read/write operations, making it perfect for performance optimization.
    • Celery: A distributed task queue system for managing asynchronous tasks and scheduling. It’s commonly paired with Redis or RabbitMQ to handle background jobs like sending emails or processing data.
    • Bootstrap 5: A popular front-end framework for designing responsive, mobile-first web pages. It includes pre-designed components, utilities, and customizable themes.
    • Docker: A containerization platform that enables the packaging of applications and their dependencies into portable containers. It ensures consistent environments across development, testing, and production.
    • Nginx: A high-performance web server and reverse proxy server. It efficiently handles HTTP requests, load balancing, and serving static files for web applications.

    To streamline my deployments, I turned to Django Cookiecutter, and let me tell you—it’s been a game changer. It’s completely transformed how quickly I can get a production-ready Django app up and running.

    For periodic tasks, I’ve swapped out traditional Cron jobs in favor of Celery. The big win here? Celery lets me manage all asynchronous jobs directly from the Django Admin interface, making everything so much more efficient and centralized.

    Sweet, right ?

    Features

    Signals

    At its core, this is Bearbot’s most important feature—it tells you what to trade. To make it user-friendly, I added search and sort functionality on the front end. This is especially handy if you have a long list of signals, and it also improves the mobile experience. Oh, and did I mention? Bearbot is fully responsive by design.

    I won’t dive into how these signals are calculated or the reasoning behind them—saving that for later.

    Available Options

    While you’ll likely spend most of your time on the Signals page, the Options list is there to show what was considered for trading but didn’t make the cut.

    Data Task Handling

    Although most tasks can be handled via the Django backend with scheduled triggers, I created a more fine-tuned control system for data fetching. For example, if fetching stock data fails for just AAPL, re-running the entire task would unnecessarily stress the server and APIs. With this feature, I can target specific data types, timeframes, and stocks.

    User Management

    Bearbot offers complete user management powered by django-allauth, with clean and well-designed forms. It supports login, signup, password reset, profile updates, multifactor authentication, and even magic links for seamless access.

    Datamanagement

    Thanks to Django’s built-in admin interface, managing users, data, and other admin-level tasks is a breeze. It’s fully ready out of the box to handle just about anything you might need as an admin.

    Keeping track of all the Jobs

    When it comes to monitoring cronjobs—whether they’re running in Node-RED, n8n, Celery, or good old-fashioned Cron—Healthchecks.io has always been my go-to solution.

    If you’ve visited the footer of the bearbot.dev website, you might have noticed two neat SVG graphics:

    Those are dynamically loaded from my self-hosted Healthchecks instance, giving a quick visual of job statuses. It’s simple, effective, and seamlessly integrated!

    On my end it looks like this:

    I had to remove a bunch of Info, otherwise anyone could send uptime or downtime requests to my Healtchecks.io

    How the Signals work

    Every trading strategy ever created has an ideal scenario—a “perfect world”—where all the stars align, and the strategy delivers its best results.

    Take earnings-based trading as an example. The ideal situation here is when a company’s earnings surprise analysts with outstanding results. This effect is even stronger if the company was struggling before and suddenly outperforms expectations.

    Now, you might be thinking, “How could I possibly predict earnings without insider information?” There are a lot of things you could consider that indicate positive earnings like:

    • Launching hyped new products that dominate social media conversations.
    • Announcing major partnerships.
    • Posting a surge in job openings that signal strategic growth.

    There are a lot of factors that support facts that a company is doing really good.

    Let’s say you focus on one or two specific strategies. You spend considerable time researching these strategies and identifying supporting factors. Essentially, you create a “perfect world” for those scenarios.

    Bearbot then uses statistics to calculate how closely a trade aligns with this perfect world. It scans a range of stocks and options, simulating and comparing them against the ideal scenario. Anything scoring above a 96% match gets selected. On average, this yields about 4-5 trades per month, and each trade typically delivers a 60-80% profit.

    Sounds like a dream, right? Well, here’s the catch: it’s not foolproof. There’s no free lunch on Wall Street, and certainly no guaranteed money.

    The remaining 4% of trades that don’t align perfectly? They can result in complete losses—not just the position, but potentially your entire portfolio. Bearbot operates with tight risk tolerance, riding trades until the margin call. I’ve experienced this firsthand on a META trade. The day after opening the position, news of a data breach fine broke, causing the stock to plummet. I got wiped out because I didn’t have enough cash to cover the margin requirements. Ironically, Bearbot’s calculations were right—had I been able to hold through the temporary loss, I would’ve turned a profit. (Needless to say, I’ve since implemented much better risk management. You live, you learn.)

    If someone offers or sells you a foolproof trading strategy, it’s a scam. If their strategy truly worked, they’d keep it secret and wouldn’t share it with you. Certainly not for 100€ in some chat group.

    I’m not sharing Bearbot’s strategy either—and I have no plans to sell or disclose its inner workings. I built Bearbot purely for myself and a few close friends. The website offers no guidance on using the signals or where to trade, and I won’t answer questions about it.

    Bearbot is my personal project—a fun way to explore Django while experimenting with trading strategies 😁.

  • Scraproxy: A High-Performance Web Scraping API

    Scraproxy: A High-Performance Web Scraping API

    After building countless web scrapers over the past 15 years, I decided it was time to create something truly versatile—a tool I could use for all my projects, hosted anywhere I needed it. That’s how Scraproxy was born: a high-performance web scraping API that leverages the power of Playwright and is built with FastAPI.

    Scraproxy streamlines web scraping and automation by enabling browsing automation, content extraction, and advanced tasks like capturing screenshots, recording videos, minimizing HTML, and tracking network requests and responses. It even handles challenges like cookie banners, making it a comprehensive solution for any scraping or automation project.

    Best of all, it’s free and open-source. Get started today and see what it can do for you. 🔥

    👉 https://github.com/StasonJatham/scraproxy

    Features

    • Browse Web Pages: Gather detailed information such as network data, logs, redirects, cookies, and performance metrics.
    • Screenshots: Capture live screenshots or retrieve them from cache, with support for full-page screenshots and thumbnails.
    • Minify HTML: Minimize HTML content by removing unnecessary elements like comments and whitespace.
    • Extract Text: Extract clean, plain text from HTML content.
    • Video Recording: Record a browsing session and retrieve the video as a webm file.
    • Reader Mode: Extract the main readable content and title from an HTML page, similar to “reader mode” in browsers.
    • Markdown Conversion: Convert HTML content into Markdown format.
    • Authentication: Optional Bearer token authentication using API_KEY.

    Technology Stack

    • FastAPI: For building high-performance, modern APIs.
    • Playwright: For automating web browser interactions and scraping.
    • Docker: Containerized for consistent environments and easy deployment.
    • Diskcache: Efficient caching to reduce redundant scraping requests.
    • Pillow: For image processing, optimization, and thumbnail creation.

    Working with Scraproxy

    Thanks to FastAPI it has full API documentation via Redoc

    After deploying it like described on my GitHub page you can use it like so:

    #!/bin/bash
    
    # Fetch the JSON response from the API
    json_response=$(curl -s "http://127.0.0.1:5001/screenshot?url=http://10.107.0.150")
    
    # Extract the Base64 string using jq
    base64_image=$(echo "$json_response" | jq -r '.screenshot')
    
    # Decode the Base64 string and save it as an image
    echo "$base64_image" | base64 --decode > screenshot.png
    
    echo "Image saved as screenshot.png"

    Make sure jq is installed

    The API provides images in base64 format, so we use the native base64 command to decode it and save it as a PNG file. If everything went smoothly, you should now have a file named “screenshot.png”.

    Keep in mind, this isn’t a full-page screenshot. For that, you’ll want to use this script:

    #!/bin/bash
    
    # Fetch the JSON response from the API
    json_response=$(curl -s "http://127.0.0.1:5001/screenshot?url=http://10.107.0.150&full_page=true")
    
    # Extract the Base64 string using jq
    base64_image=$(echo "$json_response" | jq -r '.screenshot')
    
    # Decode the Base64 string and save it as an image
    echo "$base64_image" | base64 --decode > screenshot.png
    
    echo "Image saved as screenshot.png"

    Just add &full_page=true, and voilà! You’ll get a clean, full-page screenshot of the website.

    The best part? You can run this multiple times since the responses are cached, which helps you avoid getting blocked too quickly.

    Conclusion

    I’ll be honest with you—I didn’t go all out on the documentation for this. But don’t worry, the code is thoroughly commented, and you can easily figure things out by taking a look at the app.py file.

    That said, I’ve used this in plenty of my own projects as my go-to tool for fetching web data, and it’s been a lifesaver. Feel free to jump in, contribute, and help make this even better!