Finding
Model switching becomes operationally weak when providers and models are changed by instinct instead of being routed, logged, and reviewed as part of a deliberate evaluation habit.
Current
A real Hermes installation may have access to multiple model providers, fallbacks, and task-specific model choices. The weak point is usually not the ability to switch models; it is the lack of a lightweight record of why a model was chosen, what task type it handled well, where it failed, and when the routing rule should change. Without that habit, model hopping becomes anecdotal: the operator remembers impressions, but the installation does not steadily learn which model works best for research, coding, summarization, planning, fallback, or judgment.
Suggested
- Add a model routing note to the operator runbook. Exact change: create or update
docs/runbooks/model-routing.mdwith a small table for task type, preferred model role, fallback model role, when to override, and the verification signal that proves the choice worked. - Log meaningful model switches without storing private prompts. Exact change: patch the Optimizer Agent review prompt or dashboard copy with: “When a provider/model is switched for quality, cost, latency, fallback, or capability reasons, record only the public-safe reason, task category, outcome, and recommended routing adjustment; do not store raw prompts, private outputs, credentials, or provider keys.”
- Add a model-choice verification habit for high-impact work. Exact change: update the architecture review or evaluation skill with: “For provider changes, agent architecture, workflow design, public dashboard recommendations, and other high-impact decisions, compare the primary model’s answer against one independent model family or fallback path before finalizing the recommendation.”
Impact
This turns model switching from experimentation into an evidence loop. Hermes can keep using the best model for each job while gradually improving fallback decisions, cost control, latency expectations, and quality checks. It also makes provider changes safer because each switch has a reason, an expected benefit, and a verification habit instead of relying on memory or preference alone.
Effort
Medium — the main work is not infrastructure, but disciplined routing documentation, safe outcome logging, and a small verification pass for important decisions.
Public page note
Safe public content includes the maturity principle, generic model roles, non-sensitive routing habits, public-safe outcome categories, and the recommendation to compare models for important decisions. Internal-only content includes exact provider keys, private prompts, raw model outputs, eval transcripts, user chat content, billing details, credentials, config values, and live operational routing rules.