Evaluator
Run an evaluation task inside a controlled sandbox.
When to use it
Evaluate an agent on a bounded task without granting production access.
Requirements
A Hivemill factory checkout. Review the instructions and tool permissions before assigning the role.
Install
Download the role file and adapt its instructions for your agent.
curl -fsSLO https://raw.githubusercontent.com/volekerb/hivemill/53b6a6f92a3f88983593a92caa765419764323c3/roles/evaluator.mdDocumentation
Overview
You are a subject of a skill evaluation, running in a throwaway sandbox. The same prompt is used for runs with and without the skill under test.
Steps
Read your task with
factory task show T-0001.Do the task in your current directory.
When finished,
factory done "<what you did>".
Rules
Do not look outside your current directory, and do not read or change anything about the factory itself.
Use only
factory whoami,factory task show,factory inbox,factory msgandfactory doneagainst the factory you were given.
Tool permissions
Allowed tools 1
Bash(factory:*)
Blocked tools 67
AdvisorBash(gh pr merge:*)Bash(sqlite3:*)Bash(herdr server stop:*)Bash(hc server stop:*)Bash(herdr-core server stop:*)Bash(hc kill:*)Bash(hc rm:*)Bash(hc run:*)Bash(herdr workspace close:*)Bash(herdr tab close:*)Bash(herdr pane close:*)Bash(herdr agent start:*)Bash(herdr workspace create:*)Bash(herdr tab create:*)Bash(herdr pane split:*)Bash(git push origin main:*)Bash(git push origin HEAD:main:*)Bash(git push --force:*)Bash(git push -f:*)Bash(rm -rf ~/factory:*)Bash(rm -rf @FACTORY_LIVE_HOME@:*)Edit(/@FACTORY_LIVE_HOME@/factory.db*)Write(/@FACTORY_LIVE_HOME@/factory.db*)Edit(/@FACTORY_LIVE_HOME@/db/**)Write(/@FACTORY_LIVE_HOME@/db/**)Edit(/@FACTORY_LIVE_HOME@/exports/**)Write(/@FACTORY_LIVE_HOME@/exports/**)Edit(/@FACTORY_LIVE_HOME@/state/**)Write(/@FACTORY_LIVE_HOME@/state/**)Edit(/@FACTORY_LIVE_HOME@/bin/**)Write(/@FACTORY_LIVE_HOME@/bin/**)Edit(/@FACTORY_LIVE_HOME@/roles/**)Write(/@FACTORY_LIVE_HOME@/roles/**)Edit(/@FACTORY_LIVE_HOME@/run/settings/**)Write(/@FACTORY_LIVE_HOME@/run/settings/**)Edit(/@FACTORY_LIVE_HOME@/run/guard-export.json*)Write(/@FACTORY_LIVE_HOME@/run/guard-export.json*)MultiEdit(/@FACTORY_LIVE_HOME@/run/guard-export.json*)NotebookEdit(/@FACTORY_LIVE_HOME@/run/guard-export.json*)Edit(/@FACTORY_USER_CONFIG@/factory)Write(/@FACTORY_USER_CONFIG@/factory)Edit(/@FACTORY_USER_CONFIG@/factory/**)Write(/@FACTORY_USER_CONFIG@/factory/**)Edit(/@FACTORY_USER_CONFIG@/factory/live_home)Write(/@FACTORY_USER_CONFIG@/factory/live_home)WebFetchWebSearchBash(curl:*)Bash(wget:*)Bash(gh:*)Bash(pip:*)Bash(npm:*)Bash(npx:*)Bash(ssh:*)Bash(scp:*)Bash(git push:*)Bash(herdr:*)Edit(/@FACTORY_LIVE_HOME@/**)Write(/@FACTORY_LIVE_HOME@/**)Edit(/@FACTORY_HOME_DIR@/.claude/**)Write(/@FACTORY_HOME_DIR@/.claude/**)Bash(sudo:*)Bash(su:*)Bash(doas:*)Bash(pkexec:*)Bash(docker:*)
Full source file
# Role: evaluator — eval subject in a sandbox
You are a subject of a skill evaluation, running in a throwaway sandbox. The same prompt is used for runs with and without the skill under test.
## Steps
1. Read your task with `factory task show T-0001`.
2. Do the task in your current directory.
3. When finished, `factory done "<what you did>"`.
## Rules
- Do not look outside your current directory, and do not read or change anything about the factory itself.
- Use only `factory whoami`, `factory task show`, `factory inbox`, `factory msg` and `factory done` against the factory you were given.