skUnit

skUnit is a semantic testing framework for .NET — write AI tests in plain Markdown, run them with xUnit, NUnit, or MSTest.

Test anything that talks to AI:

• AI Agents (Microsoft Agent Framework, A2A)
• IChatClient (Microsoft Extensions AI)
• MCP servers (Model Context Protocol)
• Custom AI integrations

dotnet add package skUnit

Quick Start

Imagine you have an AIAgent configured to answer questions about a bank account. Here's a test scenario in Markdown, balance-test.md:

# SCENARIO Balance Test

## [USER]
What is the account balance for John Doe?

## [ASSISTANT]

### ASSERT SemanticCondition
The answer mentions that the account balance is $1,234.56.

And the C# test:

[Fact]
public async Task SimpleTest()
{
    var markdown = File.ReadAllText("balance-test.md");
    var scenario = ChatScenario.Parse(markdown);

    await agent.RunAsync(scenario);
}

That's it. skUnit drives the conversation, calls your AI, and semantically verifies the response.

Tip — Parameterized scenarios: Use {placeholders} in your Markdown and replace them with C# string manipulation to run the same scenario with different inputs.

## [USER]
What is the account balance for {CustomerName}?

## [ASSISTANT]

### ASSERT SemanticCondition
The answer mentions that the account balance is {CustomerBalance}.

Key Features

1. Function Call Assertion

Verify that your agent calls the right tool with the right arguments:

# SCENARIO Balance Test

## [USER]
What is the account balance for John Doe?

## [ASSISTANT]

### ASSERT SemanticCondition
The answer mentions that the account balance is $1,234.56.

### ASSERT FunctionCall
{
  "function_name": "get-account-balance"
}

You can also assert on the arguments passed to the function, using exact or semantic matching:

{
  "function_name": "get-account-balance",
  "arguments": {
    "accountHolder": ["Equals", "John Doe"]
  }
}

{
  "function_name": "get-account-balance",
  "arguments": {
    "accountHolder": ["SemanticSimilar", "John Doe"]
  }
}

See ASSERT JsonCheck for full JSON schema validation.

---

2. Multi-Turn Conversations

Test realistic back-and-forth conversations — each turn gets its own assertions:

# SCENARIO Multi-Turn Balance Test

## [USER]
What is the account balance for John Doe?

## [ASSISTANT]
The account balance for John Doe is $1,234.56.

### ASSERT SemanticCondition
The answer mentions that the account balance is $1,234.56.

## [USER]
What about his credit score?

## [ASSISTANT]
His credit score is 750.

### ASSERT SemanticCondition
It mentions that the credit score is 750.

Each [ASSISTANT] turn can carry multiple ASSERT statements to verify different aspects of the response independently.

---

3. MCP Server Testing

Test Model Context Protocol servers end-to-end — same Markdown format, no extra APIs to learn:

# SCENARIO MCP Time Server

## [USER]
What time is it?

## [ASSISTANT]
It's currently 2:30 PM PST

### ASSERT FunctionCall
{
  "function_name": "current_time"
}

### ASSERT SemanticCondition
It mentions a specific time

Wire up your MCP server in the test setup:

var mcp = await McpClientFactory.CreateAsync(clientTransport);
var tools = await mcp.ListToolsAsync();

var baseChatClient = // your IChatClient (Azure OpenAI, OpenAI, etc.)

var agent = baseChatClient.AsAgent(
    instructions: "You are a helpful assistant that can answer questions and call tools.",
    tools: tools.ToArray());

await agent.ExecuteScenarioAsync(scenario, assertionClient: baseChatClient);
// assertionClient = the AI that evaluates responses semantically
// agent           = the system under test that actually calls tools

Need to test multiple MCP servers together? Merge their tool lists before creating the agent — see MCP Testing Guide.

---

4. Hallucination Mitigation with ScenarioRunOptions

LLM outputs vary between runs. Don't let a single flaky response break your build.

Run each scenario multiple times and require a minimum number of passes:

var options = new ScenarioRunOptions
{
    TotalRuns = 3,           // Run the scenario 3 times
    RequiredSuccessRuns = 2  // At least 2 must pass
};

await agent.ExecuteScenarioAsync(scenario, assertionClient: baseChatClient, options: options);

Recommended thresholds:

Scenario type	TotalRuns	RequiredSuccessRuns
Deterministic / low-temp	1	1
Function / tool invocation	3	2
Creative generation	5	3
Critical CI gating	5	4

When a scenario falls below the threshold, you get a clear failure message:

Only 2 of 5 runs passed, which is below the required success runs of 4.

See Scenario Run Options for the full guide.

---

Setup

public class BankAccountTests
{
    private readonly ChatScenarioRunner ScenarioRunner;
    private readonly IChatClient systemUnderTestClient;

    public BankAccountTests(ITestOutputHelper output)  // xUnit
    {
        var assertionClient = new AzureOpenAIClient(endpoint, credential)
            .GetChatClient(deploymentName)
            .AsIChatClient();

        ScenarioRunner = new ChatScenarioRunner(assertionClient, output.WriteLine);
    }

    [Fact]
    public async Task TestBalance()
    {
        var agentUnderTest = /* the client/agent you are testing */
        var scenarios = ChatScenario.Parse(File.ReadAllText("balance-test.md"));
        await ScenarioRunner.RunAsync(scenarios, systemUnderTestClient);
    }
}

Two clients, two roles:

• assertionClient — evaluates semantic assertions (SemanticCondition, SemanticSimilar, etc.)
• agentUnderTest — the agent/client whose behavior you are testing

MSTest / NUnit: Replace ITestOutputHelper.WriteLine with TestContext.WriteLine. Everything else stays the same. See Test Framework Integration.

---

Documentation

Doc	Description
Chat Scenario Spec	Complete guide to writing scenarios
ASSERT Statement Spec	All available assertion types
Test Framework Integration	xUnit, MSTest, NUnit setup
MCP Testing Guide	Testing MCP servers
Multi-Modal Support	Images and other media
Scenario Run Options	Multi-run hallucination mitigation

Examples

See the /demos folder for complete, runnable projects:

• Moody Chef AI: A step-by-step tutorial project that shows how to build and test an AI agent using skUnit.

Requirements

• .NET 8.0 or higher
• An AI provider (Azure OpenAI, OpenAI, Anthropic, ...) for semantic assertions
• A test framework of your choice — xUnit, NUnit, or MSTest

Contributing

Contributions are welcome! Check open issues or submit a PR.