Agents on Singularity Layer can pay for services on their own. Before an agent spends anything, something has to choose which service to call, check that the choice fits the budget and network, and decide whether a person needs to approve it. xMESH is the model I trained to make that first choice.
It’s a small Qwen model (873M parameters) that I fine-tuned on a single Mac using Singularity Training Lab. You give it what the agent wants to do, the rules it has to follow, such as budget and network, and a short list of services that could do the job. It answers with which service to use (or “none of these”), how sure it is, whether a person should approve, and whether the network and cost fit.
Building it took a few weeks. What stayed with me afterwards was not the model, it was how hard it is to know whether a model is actually good. So this is less about how I built xMESH and more about what it taught me about evaluation.
Split the decision from the guardrail
The model never constructs a payment, holds a key or invents an endpoint. Its choices are restricted to retrieved candidate IDs, and a deterministic policy validates every stage before any money moves. Learned components choose. Deterministic code decides whether the choice is allowed. On the frozen suite this produced zero post-policy violations.
The idea is bigger than payments. When a learned system’s mistakes are expensive, the safest design gives it the narrowest possible job and lets something you can read decide whether its answer is allowed. It also makes evaluation honest, because you can measure the judgement and the guardrail separately instead of one blurred number. I suspect a lot of agent safety will end up looking like this. Small learned judgements, wrapped in rules a person can audit.
Freeze the test before you look
The standalone export was fixed before a single evaluation on an existing 80-case x402 suite. No weights, prompts or policy rules changed after the results came in.
It felt like a small discipline at the time. Looking back, it was the whole game. Every time I had peeked at results while still tuning, my numbers went up and my confidence that they meant anything went down. A test you have looked at stops being a test. It quietly becomes part of the training data, through you.
| 80 cases | xMESH 0.8B | Jev | Laya |
|---|---|---|---|
| Complete-flow exact match | 80.0% | 11.25% | 0.0% |
| Stage-gate accuracy | 96.25% | 86.5% | 78.5% |
| Active-service accuracy | 97.5% | 100.0% | 85.63% |
| Global-decision accuracy | 96.25% | 37.5% | 35.63% |
The comparison rows are not hardware-equivalent. Jev is a hosted API and Laya used a different runtime. They are there to show the shape of the result, not to rank models.
Report where it loses
Outside its specialty, zero-shot on a general typed-decision benchmark, xMESH scored 0.478 accuracy. That beats the base Laya checkpoint and loses clearly to Jev and to a Laya model fine-tuned on that benchmark. Its calibration is worse than both. A narrow model that is honest about being narrow is more useful than one that looks general on a cherry-picked table.
This was the uncomfortable section to write, which is exactly why it is useful. Wins tell you the model works. Losses tell you the shape of what you actually built.
Know which property is learned
Shuffling the order of the options changed 0 of 600 answers. It is tempting to read that as robustness the model learned. It is not. The runtime canonicalises option identifiers before inference, so the invariance comes from the system, not the weights. The distinction matters, and it is exactly the kind I want to get better at drawing.
In larger models this question gets much harder. When a system behaves well, is it the weights, the prompt, the scaffolding around it, or the way we tested it? Pulling those apart is unglamorous work, and I think it is one of the most important skills in this field.
Quantisation is a behaviour change
Q8 preserved all 960 decisions across the frozen suite. Q4_K_M changed 16 decisions and kept only 65 of 80 complete flows, so it was rejected. Accelerated paths were checked the same way. Vulkan on an RTX A4000 matched CPU on 960 of 960 decisions, Metal on 959 of 960, with one near-tie. A smaller file that makes different choices is a different model.
We talk about compressing models as if it were free. It is not. Every cheaper copy is a slightly different decision maker, and if you ship it, it deserves its own evaluation.
What I would tell myself at the start
- Decide what you will measure before you build anything, and write it down.
- Make the learned part as small as you can, and everything around it readable.
- Treat every number you are proud of as a question. Where does it really come from?
What I would do next
- Probe the decision head. Is there a linear direction for abstain versus select, and does it move before the output does?
- Measure how confidence calibration shifts under quantisation, not only top-1 agreement.
- Build adversarial catalogs where the cheapest candidate is subtly wrong, and see whether the model or the policy catches it.