test(go/ai/prompt): add tests for template variable substitution

[Zereker]

Summary

• Fix template variable substitution bug in LoadPrompt where variables were replaced with empty values at load time
• Defer template rendering to execution time using WithMessagesFn
• Add convertDotpromptMessages helper function
• Add regression test TestLoadPromptTemplateVariableSubstitution

Problem

When using LoadPrompt to load .prompt files, the template was rendered at load time with an empty DataArgument. This caused all template variables (like {{name}}, {{topic}}, etc.) to be replaced with empty values immediately.

As a result, subsequent calls to Execute() or Render() with actual input values had no effect - the template was already "baked" with empty values.

Example

// greeting.prompt content:
// Hello {{name}}, welcome to {{place}}!

prompt := genkit.LookupPrompt(g, "greeting")

// BUG: Variables not substituted!
result, _ := prompt.Execute(ctx, ai.WithInput(map[string]any{
    "name":  "Alice",
    "place": "Wonderland",
}))
// Expected: "Hello Alice, welcome to Wonderland!"
// Actual: "Hello , welcome to !"  (empty values)

Solution

Defer template rendering to execution time by using WithMessagesFn. The closure:

1. Captures the raw template text at load time
2. Compiles and renders the template with actual input values at execution time
3. Properly handles multi-role messages (<<<dotprompt:role:XXX>>> markers)
4. Properly handles history insertion (<<<dotprompt:history>>> markers)

Test Plan

• Added TestLoadPromptTemplateVariableSubstitution regression test
• Verified TestMultiMessagesRenderPrompt still passes (multi-role support)
• All existing ai package tests pass

Fixes #3924

[hugoaguirre]

Hi @Zereker
Thanks for both of your contributions (here and Dotprompt). I'll take a look at them.

We are making improvements in the core which are causing merge conflicts with your contribution. Would it be possible if you address the conflicts?

[Zereker]

Hi @hugoaguirre, I've rebased on the latest main and resolved the conflicts. Ready for review!

[hugoaguirre]

Hi @Zereker, I've tried to reproduce the issue with the latest changes in main and I was able to see the prompt rendering correctly. This is the code that I used to reproduce the issue:

greeting.prompt contents:

---
description: "A greeting prompt with variables"
---
Hello {{name}}, welcome to {{place}}!

Genkit code:

func PromptFromZereker(ctx context.Context, g *genkit.Genkit) {
	prompt := genkit.LoadPrompt(g, "./prompts/greeting.prompt", "greetings")
	if prompt == nil {
		log.Fatal("empty prompt")
	}

	resp, err := prompt.Execute(ctx,
		ai.WithInput(map[string]any{
			"name":  "Alice",
			"place": "Wonderland",
		}))
	if err != nil {
		log.Fatalf("error executing prompt: %v", err)
	}
	fmt.Printf("request: %#v\n", resp.Request.Messages[0].Text())
	log.Print(resp.Text())
}

Output:

request: "Hello Alice, welcome to Wonderland!"
2025/12/16 21:11:11 Thank you for the warm welcome! What an intriguing place to find myself. I'm already feeling a delightful sense of wonder and perhaps a touch of delightful confusion, which I hear is quite common here.

So, tell me, where shall our adventure begin? Are there any White Rabbits I should follow, or perhaps a curious riddle to solve? I'm quite ready for whatever Wonderland has in store!

Could you point me in the right direction to reproduce the issue you are reporting?

You can copy paste this sample in go/samples/prompts/main.go and run it

[Zereker]

Hi @hugoaguirre,

Thanks for testing! I found that the {{role "system"}} Handlebars syntax is already used in the codebase:

• go/samples/prompts/prompts/multi-msg.prompt
• go/samples/coffee-shop/main.go
• go/ai/prompt_test.go (TestMultiMessagesPrompt)

However, there's an issue with how this syntax is handled in LoadPrompt.

How to Reproduce

Add this test to go/ai/prompt_test.go:

func TestHandlebarsRoleMarkers(t *testing.T) {
    tempDir := t.TempDir()
    mockPromptFile := filepath.Join(tempDir, "test.prompt")
    content := `---
model: test/chat
---
{{role "system"}}
You are a helpful assistant.

{{role "user"}}
Hello {{name}}, welcome to {{place}}!
`
    if err := os.WriteFile(mockPromptFile, []byte(content), 0644); err != nil {
        t.Fatal(err)
    }

    prompt := LoadPrompt(registry.New(), tempDir, "test.prompt", "test")
    opts, err := prompt.Render(context.Background(), map[string]any{
        "name":  "Alice",
        "place": "Wonderland",
    })
    if err != nil {
        t.Fatal(err)
    }

    // Verify messages are correctly separated
    if len(opts.Messages) != 2 {
        t.Errorf("Expected 2 messages, got %d", len(opts.Messages))
    }
    if opts.Messages[0].Role != RoleSystem {
        t.Errorf("Expected first message to be system role")
    }
}

Expected:

Messages: 2
[0] Role: system, Text: "You are a helpful assistant."
[1] Role: user, Text: "Hello Alice, welcome to Wonderland!"

Actual (on main):

Messages: 1
[0] Role: user, Text: "\nYou are a helpful assistant.\n\n\nHello Alice, welcome to Wonderland!"

Comparison

The existing test TestMultiMessagesRenderPrompt uses <<<dotprompt:role:system>>> format (internal markers), which works correctly. But the Handlebars syntax {{role "system"}} that users write does not work properly - all roles are merged into a single user message.

Format	Messages	Result
<<<dotprompt:role:system>>>	2	✅ Correctly separated
{{role "system"}}	1	❌ Merged into single user message

Root Cause

In LoadPrompt, line 711 calls:

dpMessages, err := dotprompt.ToMessages(parsedPrompt.Template, &dotprompt.DataArgument{})

This renders the template at load time with an empty DataArgument, before the Handlebars {{role "..."}} syntax is processed into internal markers.

My Fix

My fix defers template rendering to execution time by using WithMessagesFn:

1. Compile the template at load time (but don't render)
2. Render at execution time with actual input values via WithMessagesFn closure
3. Convert the rendered dotprompt.Message list to ai.Message with correct roles

This ensures both template variables ({{name}}) and role markers ({{role "system"}}) are properly processed.

[hugoaguirre]

Hi @Zereker,
Thanks for the clarification. I'll make some internal validations and will get back to you.

[mcicoria]

Note: This is resolved by syncing to the latest version of the main branch at cd3835a but I'm leaving it here in case it's helpful

I'm bumping this rather than reporting a separate issue. This should resolve another issue which I don't see explicitly reported where rendering a prompt multiple times with different inputs does not update the rendered template after the first input.

How to Reproduce

// TestPromptMultipleRenders tests that the prompt correctly handles
// multiple sequential renders with different inputs, ensuring each render uses
// the correct input value.
func TestPromptMultipleRenders(t *testing.T) {
	ctx := context.Background()

	tempDir := t.TempDir()
	mockPromptFile := filepath.Join(tempDir, "test.prompt")
	content := `---
model: test/chat
input:
  schema:
    input: string
---
Here is the input: {{input}}
`
	if err := os.WriteFile(mockPromptFile, []byte(content), 0644); err != nil {
		t.Fatal(err)
	}

	g := genkit.Init(ctx, genkit.WithPromptDir(tempDir))

	prompt := genkit.LookupPrompt(g, "test")
	if prompt == nil {
		t.Fatal("Prompt 'test.prompt' not found")
	}

	// Test multiple sequential renders with different inputs
	inputs := []string{
		"input-test-abc-1",
		"input-test-def-2",
		"input-test-ghi-3",
	}

	for i, input := range inputs {
		inputMap := map[string]any{
			"input": input,
		}

		actionOpts, err := prompt.Render(ctx, inputMap)
		if err != nil {
			t.Fatalf("Failed to render prompt with input %d (%q): %v", i+1, input, err)
		}

		if actionOpts == nil {
			t.Fatalf("Render() returned nil action options for input %d", i+1)
		}

		if len(actionOpts.Messages) == 0 {
			t.Fatalf("Render() returned no messages for input %d", i+1)
		}

		// Collect all text
		var renderedText strings.Builder
		for _, msg := range actionOpts.Messages {
			for _, part := range msg.Content {
				if part.IsText() {
					renderedText.WriteString(part.Text)
					renderedText.WriteString(" ")
				}
			}
		}
		text := renderedText.String()

		// Verify current input appears
		if !strings.Contains(text, input) {
			t.Errorf("Input %d (%q) not found in render %d. Text snippet: %q", i+1, input, i+1, text[:min(200, len(text))])
		}

		// Verify previous inputs do NOT appear
		for j, prevInput := range inputs {
			if j < i && strings.Contains(text, prevInput) {
				t.Errorf("BUG: Previous input %d (%q) found in render %d when it should only contain input %d (%q).",
					j+1, prevInput, i+1, i+1, input)
			}
		}
	}
}

Expected:

[0] Here is the input: input-test-abc-1
[1] Here is the input: input-test-def-2
[2] Here is the input: input-test-ghi-3

Actual:

[0] Here is the input: input-test-abc-1
[1] Here is the input: input-test-abc-1
[2] Here is the input: input-test-abc-1

[Zereker]

Hi @mcicoria, thanks for bumping this and providing the detailed reproduction case!

The issue you reported (multiple renders always using the first input) is caused by a template sharing bug in the dotprompt Compile() method. I've already fixed this in a separate PR: google/dotprompt#363 (merged on 2025-12-29).

After updating genkit to use the latest dotprompt version (v0.0.0-20251229072418-d79986469d4c), your test case passes:

=== RUN   TestPromptMultipleRenders
--- PASS: TestPromptMultipleRenders (0.00s)

Note: This PR (#3925) addresses a different issue — the {{role "..."}} Handlebars syntax not being processed correctly in LoadPrompt, where all messages get merged into a single user message instead of being properly separated by role.

[Zereker]

Update: Rebased and Simplified

This PR has been rebased on the latest main branch. Since PR #4035 has already fixed the core issue (deferring template rendering to execution time), this PR now only adds a regression test.

Why This Test is Still Valuable

The existing tests in #4035 verify that:

• Multi-role messages work correctly (TestMultiMessagesPrompt)
• Template variables are substituted (TestDeferredSchemaResolution, TestDataPromptExecute)

However, none of them test the core scenario of issue #3924:

What happens when the same prompt is rendered multiple times with different input values?

Our TestLoadPromptTemplateVariableSubstitution test specifically covers this:

// First render
actionOpts1, _ := prompt.Render(ctx, map[string]any{"name": "Alice", "place": "Wonderland"})

// Second render with DIFFERENT values
actionOpts2, _ := prompt.Render(ctx, map[string]any{"name": "Bob", "place": "Paradise"})

// Critical assertion: second render must NOT contain first render's values
if strings.Contains(text2, "Alice") {
    t.Errorf("BUG: Second render contains 'Alice' from first input!")
}

This test ensures that if someone accidentally regresses the fix (e.g., pre-rendering templates at load time), the test will catch it immediately.

Test Coverage

Sub-test	Scenario
single role	Template variables + consecutive renders + isolation verification
multi role	Multi-role + template variables + consecutive renders + isolation verification

I believe this regression test adds value as a safeguard against future regressions of issue #3924.

[Zereker]

@hugoaguirre Could you please approve the workflow runs when you have a chance? This PR adds a regression test for the template variable substitution fix. Thanks!

[hugoaguirre]

Hi @Zereker thanks for taking your time on adding this regression test!

I missed this PR since I was working on some plugin changes. I'll get back to you with my review 👍🏽

[hugoaguirre]

LGTM, we should cover the {{ role %s}} syntax as well