agentforce-eval

Category: Tools & Productivity | Uploader: aquivalabs | Downloads: 0 | Version: v1.0（Latest）
Evaluate Agentforce agents and prompt templates with Promptfoo — real multi-turn conversations, prompt template output testing, portable test format, your choice of LLM judge
Changelog: Source: GitHub https://github.com/aquivalabs/my-org-butler
Directory Structure

Current level: .claude/skills/agentforce-eval/
📁 providers/
- 📄 sf-agent-api.mjs 2.5 KB
- 📄 sf-generations-api.mjs 2.5 KB
📄 README.md 2.9 KB
📄 SKILL.md 7.2 KB
SKILL.md

---
name: agentforce-eval
description: Evaluate Agentforce agents and prompt templates with Promptfoo — real multi-turn conversations, prompt template output testing, portable test format, your choice of LLM judge
argument-hint: "run|port"
---

# Agentforce Evaluation with Promptfoo

You are an expert at evaluating Salesforce Agentforce agents and prompt templates using [Promptfoo](https://promptfoo.dev). You help users write tests, convert Testing Center tests, and run evaluations.

## Why this exists

Agentforce Testing Center fakes multi-turn conversations. It injects hardcoded history — telling the agent "pretend this conversation already happened." The agent never actually said those things. There is no real session state.

Prompt templates have no built-in regression testing at all.

This skill uses Promptfoo to run real API calls against both agents and prompt templates. Every call gets its own assertion.

## How it works

Two providers, two APIs:

```
Agent tests:
  Promptfoo CLI → sf-agent-api.mjs → Agent API
                   POST /services/data/{version}/actions/custom/generateAiAgentResponse/{agent}

Prompt template tests:
  Promptfoo CLI → sf-generations-api.mjs → Generations API
                   POST /services/data/{version}/einstein/prompt-templates/{template}/generations
```

No custom Apex needed. Both providers authenticate via `sf org display` and call standard REST APIs.

## Setup

The `agentforce-eval/.env` file is auto-generated by `scripts/create-scratch-org.sh`. It contains:

```env
AGENT_NAME=MyOrgButler
API_VERSION=v65.0
OPENAI_API_KEY=sk-...
CONTENT_DOCUMENT_ID=069...
```

- `AGENT_NAME`: the API name of the Agentforce agent
- `API_VERSION`: Salesforce API version
- `OPENAI_API_KEY`: for the LLM judge (o3-mini by default)
- `CONTENT_DOCUMENT_ID`: auto-populated by scratch org script for file-based prompt template tests

The `agentforce-eval/` directory also needs a `package.json` with `{"private": true, "type": "module"}` for ESM support.

## Test format

### Agent tests (multi-turn)

Turns sharing a `conversationId` are part of the same real conversation — the provider maintains session state automatically.

```yaml
providers:
  - id: "file://../.claude/skills/agentforce-eval/providers/sf-agent-api.mjs"
    label: "MyOrgButler"

evaluateOptions:
  maxConcurrency: 1    # required for multi-turn — turns must run in order

tests:
  - description: "Set a display preference"
    vars:
      utterance: "From now on, always sort my opportunities by close date"
      conversationId: preference
    assert:
      - type: javascript
        value: "output.success === true"
      - type: llm-rubric
        value: "Confirms it will remember the sorting preference"

  - description: "Check if preference was applied"
    vars:
      utterance: "Show me my opportunities"
      conversationId: preference
    assert:
      - type: javascript
        value: "output.success === true"
      - type: llm-rubric
        value: "Shows opportunities sorted by close date, soonest first"
```

### Prompt template tests

Each test specifies `promptTemplateName` and the template's input variables. For SObject inputs, declare the type via `sobjectInputs`.

```yaml
providers:
  - id: "file://../.claude/skills/agentforce-eval/providers/sf-generations-api.mjs"
    label: "Einstein Generations API"

tests:
  - description: "Merge memories preserving specificity"
    vars:
      promptTemplateName: ConsolidateMemory
      consolidatedMemory: "User prefers tables with Amount, Stage, CloseDate"
      newMemory: "User also wants Industry field when showing accounts"
    assert:
      - type: javascript
        value: "output.success === true"
      - type: llm-rubric
        value: "Preserves specific field names and object contexts"

  - description: "Summarize attached document"
    vars:
      promptTemplateName: AnswerFromFile
      userQuestion: "Give me a TL;DR of this document"
      file: "${CONTENT_DOCUMENT_ID}"
      sobjectInputs:
        file: "SOBJECT://ContentDocument"
    assert:
      - type: javascript
        value: "output.success === true"
      - type: llm-rubric
        value: "Provides a concise summary of the document content"
```

### Prompt template response format

```json
{
  "success": true,
  "text": "Generated text",
  "promptTemplateName": "ConsolidateMemory",
  "safetyScores": {
    "safetyScore": 0.01,
    "toxicityScore": 0.0,
    "hateScore": 0.0
  }
}
```

Safety scores are available for assertions: `output.safetyScores.toxicityScore < 0.5`

## Convention

Test files live in `agentforce-eval/` at the project root:

```
agentforce-eval/
  agent-regression.yaml    # agent tests (multi-turn + single-turn)
  prompt-regression.yaml   # prompt template tests
  .env                     # config + API keys (gitignored, auto-generated)
  package.json             # ESM support
```

YAML files reference providers via relative path:

```yaml
providers:
  - id: "file://../.claude/skills/agentforce-eval/providers/sf-agent-api.mjs"
  - id: "file://../.claude/skills/agentforce-eval/providers/sf-generations-api.mjs"
```

## Commands

### `/agentforce-eval run` — Run evaluations

Run agent tests:

```bash
cd agentforce-eval && npx promptfoo@latest eval -c agent-regression.yaml --env-file .env
```

Run prompt template tests:

```bash
cd agentforce-eval && npx promptfoo@latest eval -c prompt-regression.yaml --env-file .env
```

Run all suites:

```bash
cd agentforce-eval && for f in *.yaml; do npx promptfoo@latest eval -c "$f" --env-file .env; done
```

View results: `npx promptfoo@latest view`

### `/agentforce-eval port` — Convert Testing Center tests to Promptfoo

Read the Testing Center XML at `regressions/aiEvaluationDefinitions/*.aiEvaluationDefinition-meta.xml` and convert to Promptfoo YAML.

#### Conversion rules

- `<utterance>` → `vars.utterance`
- `<conversationHistory>` with `role=user` → earlier test cases with the same `conversationId`
- `<conversationHistory>` with `role=agent` → drop (agent responses are real in Promptfoo)
- `<expectation name="topic_assertion">` → skip (not visible through API)
- `<expectation name="actions_assertion">` → skip (not visible through API)
- `<expectation name="bot_response_rating">` → `llm-rubric` assertion

## Agent provider response format

```json
{
  "success": true,
  "text": "Agent response text",
  "sessionId": "uuid",
  "error": "error message if failed"
}
```

Use `output.success`, `output.text`, `output.sessionId` in assertions.

## Key differences from Testing Center

| Capability               | Testing Center          | Promptfoo                           |
|--------------------------|-------------------------|-------------------------------------|
| Multi-turn conversations | Fake (injected history) | Real (replayed API calls)           |
| Per-turn assertions      | Final turn only         | Every turn                          |
| Prompt template testing  | Not supported           | Via Generations API provider        |
| Response quality judge   | SF built-in, opaque     | Your choice (o3-mini, Claude, etc.) |
| Topic/action assertions  | Yes                     | No (runtime internal)               |
| Test format              | XML, deploy to test     | YAML, run from CLI                  |
Comments 0
Latest | Hot
Please login before commenting.
No comments yet. Be the first one!
agentforce-eval

Directory Structure

SKILL.md

Report

Notice
