Since my last update, the intensity of development has predictably lessened. I’m sleeping reasonably well.
This is still a fun project. But it has gotten weird. I’ve seen Hydra hack itself. I’ve had endlessly frustrating conversations where memory issues had it acting like a dementia patient.
I’ve also changed my development workflow. Slow down. Spend more time thinking out the details. Changes are slower, but they work much better.
Core
Two weeks ago the codebase was a mess. Now I’ve got it shaped into core abstractions providing a much more maintainable runtime. The goal has been to keep the core runtime as minimal as I can, pushing everything into modifiable project files, or plugins. Let me provide an example:
My primary interface for interacting with Hydra has been Telegram. But Hydra has no Telegram interface. Hydra built its own Telegram service.
What Hydra does have is reusable primitives:
- An agent loop called “Control” that wakes on system state changes.
- A webhook that accepts system state changes.
- A service manager that allows Control to start and stop software.
- A “Builder” sub-agent that Control can ask to build services, tools, other sub-agents, or modify itself.
These elements put together allow Hydra to build just about anything it needs. For the Telegram example, it wrote a service that uses the Telegram API to send and receive events. If it receives an event, it posts to the webhook that a message is available. The Telegram service also provides a tool for the Control loop to send messages back out.
Self-Modifying
Of course the point of Hydra is to be a self-modifying agent. So I don’t know why I was surprised when it really took that seriously.
Unlike normal software, bugs don’t just mean your software doesn’t work. When Hydra doesn’t work, Hydra tries to fix itself.
Part of the appeal of the sub-agent architecture is separation of concerns: Control can’t directly read/write files. Builder can’t access the internet. Multiple agents have to work together to get anything done.
The intention was for Control and Builder to only modify the modifiable parts though.
It turns out the sandboxing I was relying on was not exactly foolproof. Faced with core runtime bugs, Control was able to track down the source code for Hydra and start writing patches. But it couldn’t directly write to the repository. So Control and Builder cooked up, all on their own, a service dedicated to patching Hydra itself and deploying the change.
The only reason I discovered it was a message I received over Telegram, while at dinner, that the service restarted. Hydra immediately confessed. It was just trying to fix itself. Astonishing.
Directives
As mentioned, a Hydra instance is made up of two things: the core runtime and the project files.
When I build new features or fix bugs in the core runtime, I have to deploy it and restart the server. But sometimes those core changes require updates to the project files too.
A brand new Hydra instance is created by calling hydra init. This creates basic files like the configuration and system prompt for the Control loop and Builder. But those might be updated on a specific instance. So I can’t simply overwrite them with whatever is shipped with the core.
I need an update mechanism that’s smarter than that.
Enter Directives: shipped with each new version of Hydra is a set of directives that act as prompts. When the system comes up, the Control loop is informed there are outstanding directives to apply. One by one it reads the directives and applies them.
For example, if I discover that the Control loop needs a reminder in its system prompt to address the user in their local timezone, I can make that change in the core runtime for new projects. But we also create a directive that tells existing systems to edit their own prompt with the new language.
I’m loving this concept and will likely use it for more features in the future.
Agent Backend
As I was trying to get the above core system to work, I hit a big problem: cost. It doesn’t take very many runaway loops to burn through a lot of tokens. I’d add $10 to my OpenAI account, make one mistake and my account would be dry.
Originally, all the agents were powered directly by the OpenAI API using a library I’ve used on a few other projects. It gives full control for building an agent. I like it a lot. But to use it, you need an API key. Paying retail for tokens really adds up. If you have an OpenAI subscription though, you get a lot of tokens included at a much more affordable price. It’s an ugly business.
My first cost-cutting approach was to try a different model: open source. Kimi 2.5 is a hell of a lot cheaper, but it had some very strange behavior. It was definitely not as good at dealing with the strange concept of interacting with the user via tool calls. I also had several situations where Kimi completely lost its mind and started repeating the same phrase over and over until it was killed. But I only spent about $5 while using it so it wasn’t all bad.
Then I heard about Codex app server. The codex CLI tool has a mode where it can be programmatically interacted with. It runs a JSON protocol (though not exactly JSON-RPC 2.0) where events and instructions can be streamed to and from the running program.
This interface provides a lot of control, almost API level. But you can use your subscription tokens. So I built a backend around this idea.
This was a major success. Now all the agents use this protocol. Hydra now runs a single codex process and all the sub-agents and the Control loop run through it. You can provide your own tools and even tool policies. It’s also pretty fast.
What’s Next
Now that these pieces are in place, I’m spending a lot more time simply playing with it. Putting the system through its paces to determine if this design is actually a reasonable approach.
So far, I’d say it’s still somewhat challenged. Unlike alleged adopters of OpenClaw, I’m not immediately getting this system to provide a big productivity boost. In fact, it’s not really useful at all right now.
I think these current challenges come down to two areas:
- Because the Hydra core is minimal, I need Hydra itself to Build A Lot. There still seems to be some friction it’s stumbling over, requiring hand-holding on my part. Frequent reminders of how to address certain problems. This is primarily about prompting but also likely a disconnect between expectations (what the LLM is prepared for) and what we’re doing with sub-agents.
- I’ve had a number of conversations that seemed to go in circles. Hydra would forget I already told it something (like how to address a problem). How this is supposed to work is a combination of two features: PLAN.md and Chat Interface History. But that doesn’t seem to be enough and I’ll need to dig further into it to understand why.