I’m an LLM sceptic. LLMs controlling the computer is a security nightmare, misinformation is to be avoided.
Most modern agent clients build in some level of security, turning repetitive input tasks into a series of outputs & permissions dialogs. Permission dialogs suck, but output review is still faster than extended inputs.
There’s a workflow here:
- Give a request.
- The LLM builds a small code-based tool to carry it out.
- Human reviews, tweaks, approves the tool.
- Automation everywhere! ⚙️
Meanwhile Apple are doing amazingly on the chips front. I’ve been playing with some local LLMs for tool use, and they’re reaching a threshold where they can use and even build tools competently enough to be useful, entirely offline.
If these can run locally and build reliable automations in the course of normal computer use, that’s a gamechanger for how we interact with computers in 10 years time when this hardware is commonplace.
Even us automation nerds with 300+ shortcuts have a limit!
So, given:
- Local LLM with hardware to power it.
- Manual confirmation for any IO tasks by the LLM.
- Trustworthy code-based tools.
- Direct tool output in the UI, no LLM filter.
I think that’s my line. The line where I can actually get really really excited about this tech.