The future happened while we weren't looking

  • I’m retired. Which is interesting. I am free to do whatever I want with my time and energy. There is much to do, as there always is, but still – I am free… perhaps I should figure out what I want to do. I’ve done a bit of building with AI, which was a safe choice and is indeed fun to do, but I want to stretch a bit and see what else might be the want.

    Right now, I’m on the fence if documenting the journey would be interesting reading, or just another whisper into the void, but what the heck, it might be the want anyway. Writing, that is. If I can tell this story, I can probably tell other ones too.

    Where to start? So many cliches to choose from. For me, the thing driving this is time. I have some left in my body, but that’s a predictable glide path with a significant number of whammies, which I like most people try not to think about. But I have an unknown and uncharted amount left in my mind. Writing certainly exercises my mind (who knew?) but I’ve only really written about technology in some form or other.

    Write what you know, they say. And that brings us back to the title of this post. Not that I’m some kind of philosopher on par with Socrates but rather that it’s encouraging to know that the same guy who thought that, thought quite a bit of pretty important things up too. So yeah, what do I know? The second part of that question being that would be valuable to share? And that editor is hereby dismissed, so here it is, readers be damned. There’s probably no one out there reading anyway. So, just me then… onwards I go.

  • If you manage more than one or two home Linux servers, you already know the routine. Every morning, logwatch wakes up, collects a pile of system data, and drops a flat text file into your inbox. If you have three, four, or five machines running, checking those logs turns into an absolute chore. Half the time you just scan the headers, notice nothing looks on fire, and close the window.

    That’s bad security hygiene. But let’s be honest—reading regular operational noise day in and day out is mind-numbing.

    So I started thinking. The future happened while we weren’t looking, and now we have incredibly capable local Large Language Models (LLMs) that can run completely offline right on our own hardware. Could I offload the boring task of daily log parsing to an AI running on my home Intel NUC server? Could it act like a virtual, around-the-clock Ubuntu System Administrator, reading the noise and only buzzing my phone when an actual infrastructure anomaly hits?

    Yes, yes it can. Partnering with Google AI, which did all the heavy lifting for the coding, I mapped out a complete multi-server extraction framework.

    The best part? It costs zero dollars in subscription APIs, and not a single line of my server log data ever leaves my local network. Here is how I made that happen.

    The Architectural Philosophy: Why Pull vs. Push?

    When designing a multi-node tracking setup, you have two real choices for collecting files: you can have your remote servers push their data to your central node, or have your central node pull the data from them.

    I explicitly chose a Single-Pass Pull Architecture.

    If a remote server gets compromised, the very first thing an attacker does is look for API tokens or SSH keys to pivot across your network. By implementing a pull architecture, my remote nodes remain completely passive. They do not hold any connection keys or write permissions to my central NUC. The central monitoring node initiates a hardened outbound connection, pulls the data into an isolated staging queue, processes it, and clears the remote space. If a sub-node drops offline or suffers a breach, the rest of the infrastructure remains entirely safe.

    (more…)

  • I built a thing using google AI mode. In a couple sessions. Then I asked it to recap the sessions. For your reading pleasure here is what it said.

    Pages: 1 2

  • Or more accurately, how I used Google AI mode to do it. Then after iterating a few times, changing the approach, and debugging it, I had it generate this generic version, and this blog post, including everything after this intro block. Honestly, it would have taken me more than a day to do this. Using Google AI mode and some iterations it was done in a few hours. It all started with this question: “is it possible to use AI tools to review raw logwatch logs and filter them, opting to only be notified if something actually required manual human attention.” It was really interesting to iterate over various approaches. Pretty cool. Anyway, over to you, AI…

    Edit: There were some bugs – permissions on the local logwatch files didn’t work (UMask didn’t do it) and in the end I needed to edit the cron.daily 00logwatch to both not send email, and chown the result file to the user pulling the files. And there is a bug in parse_logs.py line 98, in
    base_name = os.path.splitext(filename).upper()
    needs to be
    base_name = os.path.splitext(filename)[0].upper()

    Also looks like the server name for flagged errors isn’t getting into the email, and it’s flagging non-errors (ssh attempts for unallowed users, successful ssh for allowed users) as actionable log messages. More revisions needed! I’ll talk to the AI later. This is going to show up in the annual performace review!

    Edit 2: Many iterations later, I have a much cleaner version that works consistently and has lots of error checking. Now, if only I could get it to generate a clean blog post.

    Pages: 1 2

  • The Flint2 aka GL-MT6000 router is my new router of choice…running OpenWRT based firmware, it is customizable, not locked down, not idiot proof. Thats great if you want, or need the ability to set up your home network the way you want. That’s me. Anyway, cool. I got one, configured it, everything works as expected. So, next obvious question – what can I do with this Linux machine? Can I monitor anything? Can I integrate it into my existing Munin based monitoring running on my home Linux server?
    Yes, yes I can. This post describes how to make that happen.

    Pages: 1 2