[smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1)

[github-actions]

🔍 Smoke Test Investigation - Run #18727962258

Summary

The Smoke GenAIScript workflow failed in the detection job due to an invalid OpenAI model name configured in the GenAIScript engine. The workflow is using openai:gpt-4.1 which is not a valid OpenAI model, causing GenAIScript to crash with a TypeError when attempting to access properties of an undefined result.

Failure Details

• Run: #18727962258
• Commit: 9d6dacf - "Update dictation prompt instructions with documentation-based glossary ([docs] Update dictation prompt instructions #2152)"
• Trigger: workflow_dispatch
• Duration: 3.9 minutes
• Failed Job: detection (1.1 minutes)
• Workflow: Smoke GenAIScript

Root Cause Analysis

Primary Issue: Invalid Model Name

The GenAIScript engine configuration in .github/workflows/shared/genaiscript.md:6 specifies:

GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4.1"

Problem: gpt-4.1 is not a valid OpenAI model name.

Valid OpenAI model names include:

• gpt-4o (latest GPT-4 optimized model)
• gpt-4-turbo
• gpt-4
• gpt-3.5-turbo

Error Chain

1. GenAIScript attempts to resolve model openai:gpt-4.1
2. OpenAI API returns an error or null because the model doesn't exist
3. GenAIScript continues execution with undefined result
4. When trying to set GitHub Action outputs, it attempts to read .text property from undefined
5. Crash: TypeError: Cannot read properties of undefined (reading 'text')

Stack Trace

2025-10-22T19:49:26.0407893Z Cannot read properties of undefined (reading 'text')
2025-10-22T19:49:26.0408573Z TypeError: Cannot read properties of undefined (reading 'text')
2025-10-22T19:49:26.0409433Z     at githubActionSetOutputs ((redacted))
2025-10-22T19:49:26.0410525Z     at async Command.runScriptWithExitCode ((redacted))
2025-10-22T19:49:26.0549645Z ##[error]Process completed with exit code 255.

Failed Jobs and Errors

Job Execution Summary

1. ✅ activation - succeeded (4s)
2. ✅ agent - succeeded (1.9m)
3. ❌ detection - failed (1.1m)
4. ✅ create_issue - succeeded (7s)
5. ⏭️ missing_tool - skipped

Detection Job Failure

The detection job runs GenAIScript for threat detection analysis. From the logs:

2025-10-22T19:49:25.9916102Z   model: 'openai:gpt-4.1',
2025-10-22T19:49:25.9918993Z 2025-10-22T19:49:25.986Z genaiscript:host:node alias: large.model = openai:gpt-4.1 (source: script)
2025-10-22T19:49:25.9920566Z 2025-10-22T19:49:25.987Z genaiscript:modelalias large: openai:gpt-4.1 (script)
2025-10-22T19:49:25.9921343Z 2025-10-22T19:49:25.987Z genaiscript:models resolving model for 'openai:gpt-4.1'

GenAIScript successfully validates OPENAI_API_KEY but fails when attempting to use the invalid model.

Investigation Findings

Configuration Location

The invalid model is configured in:

• File: .github/workflows/shared/genaiscript.md
• Line: 6
• Variable: GH_AW_AGENT_MODEL_VERSION

Affected Workflows

All workflows that import shared/genaiscript.md are affected:

• smoke-genaiscript.md - Currently failing (this investigation)
• Any other workflows using the GenAIScript shared configuration

Why This Wasn't Caught Earlier

• OPENAI_API_KEY validation passes (the key exists)
• GenAIScript doesn't validate model names during initialization
• Error only occurs when attempting to use the model
• Poor error handling in GenAIScript's output setting code

Related Historical Context

Similar issue was reported in #2142, but that was caused by missing OPENAI_API_KEY. This is a related but distinct issue - the API key exists but the model name is invalid.

Pattern Database: This is the first occurrence of the GENAISCRIPT_INVALID_MODEL pattern.

Recommended Actions

🔴 CRITICAL - Immediate Fix

• Update model name in .github/workflows/shared/genaiscript.md:6

  # Change from:
  GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4.1"
  
  # Change to (recommended):
  GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4o"

🟡 HIGH PRIORITY - GenAIScript Error Handling

• Fix GenAIScript's githubActionSetOutputs function (upstream issue)
  • Add null/undefined checks before accessing .text property
  • Provide meaningful error message when model resolution fails
  • File issue with GenAIScript project: https://github.com/microsoft/genaiscript

🟢 MEDIUM PRIORITY - Prevention

• Add model name validation in GenAIScript configuration
• Document valid model names in engine configuration comments
• Add pre-flight check to validate model exists before running workflow
• Create smoke test specifically for engine configuration validation

🔵 LOW PRIORITY - Documentation

• Update GenAIScript engine documentation with list of valid models
• Add troubleshooting guide for model configuration errors
• Document the relationship between model names and required API keys

Prevention Strategies

1. Configuration Validation

Add a validation step at the beginning of GenAIScript workflows:

- name: Validate model configuration
  run: |
    VALID_MODELS="gpt-4o gpt-4-turbo gpt-4 gpt-3.5-turbo"
    MODEL_NAME="${GH_AW_AGENT_MODEL_VERSION#openai:}"
    if [[ ! " $VALID_MODELS " =~ " $MODEL_NAME " ]]; then
      echo "Error: Invalid model '$MODEL_NAME'"
      echo "Valid models: $VALID_MODELS"
      exit 1
    fi

2. Schema Validation

Use JSON schema or similar to validate workflow configuration files before they're used.

3. Better Error Messages

Enhance GenAIScript to detect invalid model names and provide suggestions:

Error: Model 'gpt-4.1' not found
Did you mean: gpt-4o, gpt-4-turbo, gpt-4?

4. Smoke Test Coverage

Add specific smoke tests for each engine configuration to catch these issues before they affect production workflows.

Technical Details

Debug Logs Analysis

From the detection job logs, we can see GenAIScript attempting to resolve the model:

2025-10-22T19:49:25.9922308Z 2025-10-22T19:49:25.987Z genaiscript:models candidate openai:gpt-4.1
2025-10-22T19:49:25.9922980Z 2025-10-22T19:49:25.987Z genaiscript:models resolving openai:gpt-4.1
2025-10-22T19:49:25.9923735Z 2025-10-22T19:49:25.989Z genaiscript:config:env parsing token for 'openai:gpt-4.1:'
2025-10-22T19:49:25.9924493Z 2025-10-22T19:49:25.989Z genaiscript:config:env processing openai

The model resolution appears to succeed internally, but the actual API call fails because the model doesn't exist.

MCP Configuration Note

Failed to load MCP configuration: MCP configuration file not found: /tmp/gh-aw/mcp-config/mcp-servers.json

This is a separate (minor) issue - MCP configuration is optional for this workflow.

Impact Assessment

Severity: 🔴 HIGH

• All GenAIScript-based workflows are failing
• Threat detection is not running (security impact)
• Smoke tests cannot validate GenAIScript engine functionality

Urgency: 🔴 IMMEDIATE

• Simple one-line fix required
• Blocking critical functionality

Scope:

• Affects: All workflows using shared/genaiscript.md
• Not affected: OpenCode, Claude, and other engine workflows

Reproduction Steps

1. Configure GenAIScript with model: openai:gpt-4.1
2. Ensure OPENAI_API_KEY is set (so validation passes)
3. Run any GenAIScript workflow
4. Observe failure when GenAIScript attempts to use the invalid model
5. See TypeError when accessing undefined result

Related Issues

• [smoke-detector] 🔍 Smoke Test Investigation - GenAIScript OPENAI_API_KEY Missing #2142 - Similar GenAIScript error with missing OPENAI_API_KEY (different root cause)

---

Investigation Metadata

• Investigator: Smoke Detector (Investigation Agent)
• Investigation Run: #18728054085
• Pattern: GENAISCRIPT_INVALID_MODEL (first occurrence)
• Pattern File: /tmp/gh-aw/cache-memory/patterns/genaiscript_invalid_model.json
• Investigation Record: /tmp/gh-aw/cache-memory/investigations/2025-10-22-18727962258.json
• Created: 2025-10-22T19:52:00Z

AI generated by Smoke Detector - Smoke Test Failure Investigator

AI generated by Smoke Detector - Smoke Test Failure Investigator