> For the complete documentation index, see [llms.txt](https://t420.gitbook.io/t420/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://t420.gitbook.io/t420/openclaw-fleetos-eval-alignment.md).

# OpenClaw ↔ FleetOS — Eval Alignment

**Audience:** OpenClaw operators + FleetOS data owners **Supersedes:** the per-role eval anchors section in `fleetos-data-structure-for-sales-evals.md` (the schema/ICP/guardrail content there is still valid; only the agent breakdown is replaced). **As of:** 2026-04-30

OpenClaw owns FleetOS's autonomous B2B sales function. The autonomous team is **four agents only** — Orchestrator, Prospector, Actor/Scribe, Evaluator. Researcher, Qualifier, Drafter, Demo, Activation, Reviewer are **skills/workflows** distributed across those four, not agents.

***

## 1. Agent KPIs (recap, single source of truth)

| Agent              | Primary eval                    | What it measures                                                                           |
| ------------------ | ------------------------------- | ------------------------------------------------------------------------------------------ |
| **Prospector**     | `bounce_rate`                   | Contact/map quality — proportion of outbound that hits an invalid address/number           |
| **Actor / Scribe** | `signup_rate`                   | Conversion from **valid delivered** messages to **valid signups**                          |
| **Evaluator**      | `independent_audit_defect_rate` | Whether Evaluator's reports/calculations/recommendations survive a deterministic re-audit  |
| **Orchestrator**   | `valid_signups_per_day`         | Safe growth flow from the whole machine; gates throttling, channel choice, geo, compliance |

Everything else in this doc is a **diagnostic, guardrail, or activation metric** — not a fifth KPI.

***

## 2. `valid_signup` — definition

A FleetOS row counts as a **valid signup** for Scribe's `signup_rate` and Orchestrator's `valid_signups_per_day` **if and only if all of the following are true**:

### 2.1 Identity proven

* A `users` row exists AND a `user_profiles` row exists for it.
* `user_profiles.role ∈ {supplier, fleet, dispatch, manager, tech, owner}`.
* `user_profiles.company_id` is non-null and resolves to a `companies` row.
* `companies.company_type ∈ {fleet, supplier, mixed}`.

### 2.2 Reachability proven

* The originating contact has a confirmed delivery: at least one `notifications` row tied to this user/company with `delivered_at IS NOT NULL`.
* Magic-link path: at least one `notifications` row with `clicked_at IS NOT NULL` **OR** an `event_log` row with `event_type = 'magic_link_clicked'` for the same `company_id`.

### 2.3 Geography (hard guardrail)

* Contact resolves to **US**: phone is E.164 `+1…` OR email domain resolves to a US-operating org.
* **Today this cannot be enforced at the data layer** — see §6 for the missing column.

### 2.4 Not noise

* `companies.company_name` is non-empty and not a known test fixture.
* No `service_requests` for this company with `created_via = 'demo_seed'` as the **sole** activity (demo-seeded rows are not product use).
* `users.email` is not in the test/seed allowlist (free-domain allowance is governed by migration `019_allow_free_email_domains.sql` — kept, but flagged).
* The signup is not a re-creation of an existing `companies.domain` (dedupe by `companies.domain`, not by `users.email`).

### 2.5 Attribution discoverable (for Scribe → signup attribution)

* `users.signup_source` OR `companies.source_agent` OR `companies.source_campaign_id` OR `companies.blade_lead_id` is set, **OR** the user clicked through a Scribe-issued magic link whose token is logged.
* Without attribution the row is a valid signup for Orchestrator (it still counts toward growth) but **not attributable** to Scribe — see §7 on how to handle that.

### 2.6 Invalid / noisy (exclude from numerator and denominator semantics differently)

| Reason                                      | Excluded from valid\_signups | Notes                                                      |
| ------------------------------------------- | ---------------------------- | ---------------------------------------------------------- |
| No `delivered_at` on any notification       | Yes                          | Counts as Prospector bounce, not Scribe failure            |
| No `user_profiles` row                      | Yes                          | Half-formed account                                        |
| `created_via = 'demo_seed'` only activity   | Yes                          | Demo seed, not real                                        |
| Duplicate `companies.domain` within 90 days | Yes                          | Dedupe — count once                                        |
| Non-US contact                              | Yes (when detectable)        | Hard guardrail                                             |
| Test fixture / dev account                  | Yes                          | Needs `is_test` flag — missing today                       |
| Bot/script signup pattern                   | Yes                          | Rate-limit table `rate_limits` (migration 033) is the gate |

***

## 3. Source-of-truth tables/events

| Concept                         | Source of truth                                                                                                                                  | Query shape                                                             |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
| **Signup**                      | `user_profiles` JOIN `users` JOIN `companies`                                                                                                    | First `user_profiles.created_at` per `company_id`                       |
| **Magic link sent**             | `notifications` (`magic_link IS NOT NULL`)                                                                                                       | `sent_at`                                                               |
| **Magic link delivered**        | `notifications.delivered_at`                                                                                                                     | proves reachability                                                     |
| **Magic link clicked**          | `notifications.clicked_at` OR `event_log.event_type = 'magic_link_clicked'`                                                                      | proves intent                                                           |
| **Demo used**                   | `service_requests.created_via = 'demo_seed'` (seeded data) **+** `event_log.event_type = 'onboarding_started'` (user-initiated demo)             | distinguish seeded vs explored                                          |
| **First work order**            | `service_requests` ORDER BY `created_at` ASC LIMIT 1 per `fleet_company_id` (or `supplier_company_id`), filtered to `created_via != 'demo_seed'` | `event_log.event_type = 'first_request_created'` is a parallel signal   |
| **First quote**                 | `service_requests.quoted_amount IS NOT NULL` (first per `supplier_company_id`); via API: `commercial_status = 'quoted'`                          | crosses commercial-status threshold                                     |
| **First invite**                | `fleetos_invites` ORDER BY `created_at` ASC per `inviter_id`; `event_log.event_type = 'first_invite_sent'`                                       | viral surface                                                           |
| **Relationship created**        | `fleetos_relationships.created_at`; `event_log.event_type = 'relationship_established'`                                                          | Scribe-attributable if invite was Scribe-issued                         |
| **Repeat usage**                | COUNT(`service_requests`) > 1 per `company_id` over 30/60/90d, excluding `demo_seed`                                                             | retention diagnostic                                                    |
| **Outbound delivery**           | `notifications.delivered_at`                                                                                                                     | denominator for Scribe `signup_rate`                                    |
| **Outbound bounce**             | **NOT INSTRUMENTED** — see §6                                                                                                                    | denominator/numerator for Prospector `bounce_rate`                      |
| **Attribution to Blade/Scribe** | `companies.source_agent`, `companies.source_campaign_id`, `companies.blade_lead_id`, `users.signup_source`                                       | additive, nullable (migration `20260418_blade_fleetos_attribution.sql`) |

`event_log.event_type` enum already in use (from `services/shared/EventLogService.ts`):

```
signup_initiated, magic_link_clicked, profile_completed,
first_supplier_added, first_request_created, first_request_acknowledged,
first_invite_sent, onboarding_complete,
request_created, request_acknowledged, request_accepted,
request_completed, request_declined,
relationship_established, invite_sent, invite_accepted,
magic_link_accessed
```

***

## 4. Mapping the prior roles into the four agents

| Prior role           | Now lives as                                                                             | Inside which agent        | Eval contribution                                                 |
| -------------------- | ---------------------------------------------------------------------------------------- | ------------------------- | ----------------------------------------------------------------- |
| Researcher           | Skill: prospect-list build                                                               | **Prospector**            | Inputs to `bounce_rate` (better lists → fewer bounces)            |
| Qualifier            | Skill: ICP fit + DQ filter                                                               | **Prospector**            | Filters before send → protects `bounce_rate` and `signup_rate`    |
| Outreach Drafter     | Skill: message composition                                                               | **Scribe**                | Drives `signup_rate`                                              |
| Demo / Pitch         | Skill: send + route to CTA + magic link                                                  | **Scribe**                | Drives `signup_rate`; demo-link clicks are activation diagnostics |
| Activation / Handoff | Workflow: Scribe sends activation messages **+** Orchestrator decides whether to proceed | **Scribe + Orchestrator** | Scribe = signup\_rate; Orchestrator = `valid_signups_per_day`     |
| Reviewer / Refiner   | Skill: independent re-audit                                                              | **Evaluator**             | Drives `independent_audit_defect_rate`                            |

**Rule:** if a workflow appears in two agents (e.g. activation), the **eval credit** goes to the agent whose KPI it moves. Activation message draft = Scribe. Decision to push that activation under current geo/compliance state = Orchestrator.

***

## 5. Activation events — diagnostic, not Scribe KPI

Activation events tell us **whether signups are valuable**. They do **not** roll up into Scribe's `signup_rate`.

| Stage                    | Source                                                    | Diagnostic question                            |
| ------------------------ | --------------------------------------------------------- | ---------------------------------------------- |
| Magic link clicked       | `notifications.clicked_at`                                | Did the message reach a real human?            |
| Profile completed        | `event_log: profile_completed`                            | Did the human bother?                          |
| First work order created | `event_log: first_request_created` (excluding demo\_seed) | Did they reach product value?                  |
| First quote submitted    | `service_requests.quoted_amount` first non-null           | Did the supplier side reach product value?     |
| First invite sent        | `event_log: first_invite_sent`                            | Did they pull a counterparty in? Viral signal. |
| Relationship established | `event_log: relationship_established`                     | Did the network form?                          |
| Repeat usage in 30d      | COUNT > 1 of real `service_requests`                      | Did they come back?                            |

If `signup_rate` is healthy but activation drops at a specific stage, that's a **product/onboarding** issue — not a Scribe issue. Orchestrator routes that signal to product, not to Scribe's eval.

***

## 6. Missing instrumentation (what blocks the evals)

| Eval                                          | Missing today                                                                                                                                                  | Required change                                                                                                                                                                                           | Owner                                                  |
| --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| **Prospector `bounce_rate`**                  | No bounce capture. `notifications` has `sent_at, delivered_at, opened_at, clicked_at` but no `bounced_at` or `bounce_reason`.                                  | Add `bounced_at TIMESTAMPTZ`, `bounce_reason TEXT`, `bounce_type TEXT CHECK ('hard','soft','undeliverable','invalid_number','blocked')` to `notifications`. Wire Resend + Twilio webhooks to populate.    | FleetOS                                                |
| **Prospector `bounce_rate` (US guardrail)**   | No country column. `users.phone` and `contacts.contact_value` are TEXT; no normalized E.164 or country code.                                                   | Add `contacts.country_code TEXT` (ISO-3166 alpha-2) and `contacts.phone_e164 TEXT`. Backfill with libphonenumber. Reject non-US at Prospector layer.                                                      | FleetOS                                                |
| **Scribe `signup_rate` (denominator)**        | "Valid delivered messages" is computable today (`notifications.delivered_at IS NOT NULL`) but not deduped to the company a Scribe campaign targeted.           | Add `notifications.campaign_id UUID` and `notifications.outbound_agent TEXT`. Backfill from Blade campaigns.                                                                                              | FleetOS + OpenClaw                                     |
| **Scribe `signup_rate` (attribution)**        | `companies.source_campaign_id` exists but no enforced link from a delivered notification → resulting signup.                                                   | Stamp the campaign\_id on the magic-link token and propagate it to `companies.source_campaign_id` at signup.                                                                                              | FleetOS (sign-token edge function)                     |
| **`valid_signup` (test/seed exclusion)**      | No `is_test` flag on `companies` or `users`.                                                                                                                   | Add `companies.is_test BOOLEAN DEFAULT FALSE`. Mark all dev/seed companies. Exclude from valid\_signups.                                                                                                  | FleetOS                                                |
| **`valid_signup` (US enforcement)**           | No country on `companies` either.                                                                                                                              | Add `companies.country_code TEXT NOT NULL DEFAULT 'US'` post-launch; until then derive from `companies.domain` + primary contact.                                                                         | FleetOS                                                |
| **Demo-vs-real distinction**                  | `service_requests.created_via = 'demo_seed'` works for seeded rows, but a user clicking "Try the demo" is not always tagged.                                   | Ensure `DemoEntry.tsx` flow logs `event_log.event_type = 'demo_started'` and stamps `created_via = 'demo_explored'` on any rows it produces.                                                              | FleetOS                                                |
| **Evaluator `independent_audit_defect_rate`** | No table to record audit results. Evaluator outputs are not stored as deterministically replayable artifacts.                                                  | Create `agent_audits (audit_id, agent, target_id, target_type, computed_metric, evaluator_metric, defect BOOLEAN, defect_reason, ran_at)`. Evaluator writes here; defect\_rate = SUM(defect) / COUNT(\*). | OpenClaw                                               |
| **Orchestrator `valid_signups_per_day`**      | Computable today from §2 once §6 items above land. Until then, the metric runs on a best-effort basis with known undercounts (no bounce filter, no US filter). | Same as above.                                                                                                                                                                                            | Orchestrator-side query is trivial once columns exist. |

**Priority order for OpenClaw to unblock evals:**

1. `notifications.bounced_at` + webhook wiring → unlocks `bounce_rate`.
2. `contacts.phone_e164` + `country_code` → unlocks US guardrail and tightens `bounce_rate`.
3. `notifications.campaign_id` + token propagation → unlocks Scribe attribution.
4. `companies.is_test` → cleans `valid_signups_per_day`.
5. `agent_audits` table → unlocks Evaluator KPI.

***

## 7. Attribution policy (when Scribe gets credit)

A signup attributes to Scribe **only if** at least one of:

* `companies.source_agent IN ('scribe','blade')` AND `source_campaign_id` matches a Scribe-issued campaign.
* `users.signup_source` matches a Scribe channel tag.
* The signup originated from a magic-link token whose `campaign_id` was stamped by Scribe (per §6 item 3).

Signups without attribution still count for **Orchestrator's** `valid_signups_per_day` (organic growth is real growth) but are **excluded from Scribe's `signup_rate` denominator and numerator** (otherwise organic signups would inflate Scribe's eval). Orchestrator should track `attributed_signups / valid_signups` as a diagnostic of attribution coverage, not as an agent KPI.

***

## 8. Hard guardrails — encoded as data rules

| Guardrail                           | Enforcement point               | Data rule                                                                                                                                                           |
| ----------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **US-only**                         | Prospector pre-send             | Reject any contact where `country_code != 'US'` (once column lands). Until then, soft-reject non-`+1` phone and known non-US TLDs.                                  |
| **No SMS until 10DLC clears**       | Orchestrator channel router     | Block channel `'sms'` in `notifications.channel` while `flag.10dlc_approved = false`. Track in feature flag, not in code.                                           |
| **Not all-in-one fleet management** | Scribe message-composition lint | Evaluator sample-checks outbound for forbidden phrases ("fleet management suite", "all-in-one fleet", "TMS replacement"). Defect → `independent_audit_defect_rate`. |
| **Lead with free tier**             | Scribe message-composition lint | Outbound must mention `$0/month` or "free" before any paid-tier reference. Evaluator checks.                                                                        |
| **Magic link / no app install**     | Scribe CTA lint                 | All CTAs must route to `?view=FLEETOS_*_ONBOARDING`, `/request/:slug`, or `/track/:token`. No "download app" copy. Evaluator checks.                                |
| **Wedge: external coordination**    | Scribe targeting                | Prospector excludes single-supplier-single-fleet exclusive relationships and W2-internal-only fleets from list build.                                               |

Each guardrail violation is a **defect** in Evaluator's audit. They do not roll up into Scribe's `signup_rate` directly, but a high defect rate from Evaluator is a kill-switch signal Orchestrator uses to throttle Scribe.

***

## 9. Eval pipeline — one paragraph

Prospector builds and qualifies a list, hands it to Scribe. Scribe sends. `notifications` records sent / delivered / bounced. Bounces feed Prospector's `bounce_rate`. Delivered messages form Scribe's denominator. A subset clicks magic links and completes signup; those that meet §2 are valid signups; those that also meet §7 attribute to Scribe and form Scribe's `signup_rate` numerator. Activation events (§5) are tracked but do not feed Scribe's KPI — they feed product. Evaluator independently re-runs the math and the message-lint checks; mismatches go into `agent_audits` and feed `independent_audit_defect_rate`. Orchestrator sums valid signups per day, watches Evaluator's defect rate as a kill switch, and routes throttling/channel/geo decisions accordingly.

***

## 10. Open questions for OpenClaw

1. Does Evaluator also re-audit Prospector's bounce calls (i.e. independently verify a "bounce" was actually undeliverable)? Default answer: yes — that's the deterministic part.
2. Should organic signups (no attribution) be visible to Scribe at all? Default: no, to keep Scribe's eval honest.
3. What's the SLA for FleetOS to ship the §6 instrumentation? Until shipped, Scribe `signup_rate` runs on best-effort with documented undercounts.
4. Where does `agent_audits` live — FleetOS DB or OpenClaw DB? Default: OpenClaw, since Evaluator owns the artifact lifecycle.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://t420.gitbook.io/t420/openclaw-fleetos-eval-alignment.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.