Skip to content

Empty commit basically#4088

Merged
cloutiertyler merged 7 commits intomasterfrom
tyler/fix-llm-benchmark
Jan 22, 2026
Merged

Empty commit basically#4088
cloutiertyler merged 7 commits intomasterfrom
tyler/fix-llm-benchmark

Conversation

@cloutiertyler
Copy link
Contributor

Empty commit to fix llm benchmark.

@cloutiertyler
Copy link
Contributor Author

/update-llm-benchmark

@clockwork-labs-bot
Copy link
Collaborator

LLM Benchmark Results (ci-quickfix)

Language Mode Category Tests Passed Task Pass %
Rust rustdoc_json basics 25/27 83.3% ⬆️ +38.9%
Rust rustdoc_json schema 26/34 75.3% ⬆️ +48.8%
Rust rustdoc_json total 51/61 79.7% ⬆️ +43.4%
Rust docs basics 5/27 11.1%
Rust docs schema 8/34 20.5%
Rust docs total 13/61 15.4%
C# docs basics 27/27 100.0% ⬆️ +38.9%
C# docs schema 25/34 73.7% ⬆️ +42.2%
C# docs total 52/61 88.0% ⬆️ +40.4%

Compared against master branch baseline

Generated at: 2026-01-21T19:53:18.713Z

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

  • Total failures analyzed: 31

SpacetimeDB Benchmark Test Failures Analysis

Group 1: Rust / rustdoc_json Failures

t_002_scheduled_table

  • The generated code:

    use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
    
    #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        scheduled_id: u64,
        scheduled_at: ScheduleAt,
    }
    
    #[reducer]
    pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {
        // Triggered by the scheduler; no-op body for demo.
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        if ctx.db.tick_timer().count() == 0 {
            ctx.db.tick_timer().insert(TickTimer {
                scheduled_id: 0,
                scheduled_at: ScheduleAt::RepeatMicros(50_000),
            });
        }
    }
  • The golden example:

    use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
    use std::time::Duration;
    
    #[table(name = tick_timer, scheduled(tick))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        pub scheduled_id: u64,
        pub scheduled_at: ScheduleAt,
    }
    
    #[reducer]
    pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        let every_50ms: ScheduleAt = Duration::from_millis(50).into();
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: every_50ms,
        });
    }
  • The error: publish_error: spacetime publish failed (exit=1)

  • Explain the difference:

    • The LLM didn't declare scheduled_id and scheduled_at as pub, which is necessary for struct fields to be accessible outside their module. Additionally, the scheduled-reducer usage (in the form of ScheduleAt::RepeatMicros) was not correct; it should be an interval created with Duration::from_millis.
  • Root cause: The documentation may lack clarity on the necessity of field visibility (using pub) and proper handling of scheduled intervals.

  • Recommendation:

    • Explicitly document that struct fields need to be pub for external usage.
    • Include information about the correct intervals for ScheduleAt.

t_017_scheduled_columns

  • The generated code:

    use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
    
    #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        scheduled_id: u64,
        scheduled_at: ScheduleAt,
    }
    
    #[reducer(scheduled)]
    pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {
        // scheduled tick handler
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: ScheduleAt::repeat_micros(50_000),
        });
    }
  • The golden example:

    use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
    use std::time::Duration;
    
    #[table(name = tick_timer, scheduled(tick))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        pub scheduled_id: u64,
        pub scheduled_at: ScheduleAt,
    }
    
    #[reducer]
    pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        let interval = Duration::from_millis(50);
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: ScheduleAt::Interval(interval.into()),
        });
    }
  • The error: publish_error: spacetime publish failed (exit=1)

  • Explain the difference: The LLM's tick function missed a proper definition for the scheduled handler. The scheduled_id and scheduled_at were not declared as pub, and ScheduleAt::repeat_micros does not align with expected constructors (should use Interval).

  • Root cause: Lack of documentation surrounding struct field visibility and proper API usage for scheduled mechanics.

  • Recommendation:

    • Clarify visibility requirements for struct fields in documentation.
    • Provide clearer examples for ScheduleAt usage to avoid confusion.

Group 2: Rust / docs Failures

t_000_empty_reducers

  • The generated code:

    use spacetimedb::ReducerContext;
    
    #[spacetimedb::reducer]
    pub fn empty_reducer_no_args(_ctx: &ReducerContext) {}
    
    //... other reducers
  • The golden example:

    use spacetimedb::{reducer, ReducerContext};
    
    #[reducer]
    pub fn empty_reducer_no_args(ctx: &ReducerContext) -> Result<(), String> {
        Ok(())
    }
    
    //... other reducers with return type
  • The error: schema_parity: describe failed

  • Explain the difference: The LLM did not return a Result or Ok(()) for each reducer definition. Reducers are expected to return a Result to reflect success or failure.

  • Root cause: Inadequate documentation surrounding return types for reducer functions.

  • Recommendation:

    • Explicitly state that all reducers must return a Result type in documentation.

t_001_basic_tables

  • The generated code:

    #[spacetimedb::table(name = user)]
    pub struct User {
        #[primary_key]
        id: i32,
        name: String,
        age: i32,
        active: bool,
    }
  • The golden example:

    use spacetimedb::table;
    
    #[table(name = user)]
    pub struct User {
        #[primary_key]
        pub id: i32,
        pub name: String,
        pub age: i32,
        pub active: bool,
    }
  • The error: schema_parity: describe failed

  • Explain the difference: The LLM-generated code did not mark the struct fields as pub, leading to accessibility issues.

  • Root cause: Lack of clarity regarding which struct fields need to be public for proper database access.

  • Recommendation:

    • Emphasize the need for pub on table struct fields in documentation.

t_002_scheduled_table (Failure repeated in docs)

  • Explanation and recommendations are the same as in earlier analysis for the same scenario.

t_003_struct_in_table

  • The generated code:

    #[derive(spacetimedb::SpacetimeType)]
    pub struct Position {
        x: i32,
        y: i32,
    }
    
    #[spacetimedb::table(name = entity)]
    pub struct Entity {
        #[primary_key]
        id: i32,
        pos: Position,
    }
  • The golden example:

    use spacetimedb::{table, SpacetimeType};
    
    #[derive(SpacetimeType, Clone, Debug)]
    pub struct Position {
        pub x: i32,
        pub y: i32,
    }
    
    #[table(name = entity)]
    pub struct Entity {
        #[primary_key]
        pub id: i32,
        pub pos: Position,
    }
  • The error: schema_parity: describe failed

  • Explain the difference: Fields in Position and Entity structs were not marked as pub, leading to accessibility issues.

  • Root cause: Insufficient documentation on the need for visibility modifiers on struct fields for types used in tables.

  • Recommendation:

    • Require pub visibility for fields in structs in documentation examples.

Group 3: C# / docs Failures

t_014_elementary_columns

  • The generated code:

    using SpacetimeDB;
    
    public static partial class Module
    {
        [SpacetimeDB.Table(Name = "Primitive", Public = true)]
        public partial struct Primitive
        {
            [SpacetimeDB.PrimaryKey]
            public int Id;
            public int Count;
            public long Total;
            public float Price;
            public double Ratio;
            public bool Active;
            public string Name;
        }
    
        [SpacetimeDB.Reducer]
        public static void Seed(ReducerContext ctx) {
            // Insertion logic here...
        }
    }
  • The golden example:

    using SpacetimeDB;
    
    public static partial class Module
    {
        [Table(Name = "Primitive")]
        public partial struct Primitive
        {
            [PrimaryKey] public int Id;
            public int Count;
            public long Total;
            public float Price;
            public double Ratio;
            public bool Active;
            public string Name;
        }
    
        [Reducer]
        public static void Seed(ReducerContext ctx) {
            // Insertion logic here...
        }
    }
  • The error: no such table: primitive

  • Explain the difference: The LLM incorrectly altered the table declaration, setting Public = true which is unnecessary.

  • Root cause: Documentation may contain unclear parameters regarding visibility settings while defining tables.

  • Recommendation:

    • Update documentation to clarify that Public parameter is not mandatory.

Conclusion

For all observed failures, the root cause mainly lies within inconsistencies regarding the required syntax, visibility, and expected return types. Recommendations focus on improving the clarity of existing documentation to reduce confusion and ensure compliance with struct and API definitions.

clockwork-labs-bot and others added 3 commits January 21, 2026 19:54
This prevents CI failures when the rolling nightly updates between
when hashes are saved and when CI checks run. The pinned version
can be updated intentionally by changing PINNED_NIGHTLY constant.
@cloutiertyler
Copy link
Contributor Author

/update-llm-benchmark

@clockwork-labs-bot
Copy link
Collaborator

LLM Benchmark Results (ci-quickfix)

Language Mode Category Tests Passed Task Pass %
Rust rustdoc_json basics 25/27 83.3% ⬆️ +38.9%
Rust rustdoc_json schema 23/34 65.3% ⬆️ +38.8%
Rust rustdoc_json total 48/61 75.2% ⬆️ +38.9%
Rust docs basics 5/25 11.1%
Rust docs schema 3/19 7.5% ⬇️ -13.0%
Rust docs total 8/44 9.5% ⬇️ -5.9%
C# docs basics 27/27 100.0% ⬆️ +38.9%
C# docs schema 25/34 73.7% ⬆️ +42.2%
C# docs total 52/61 88.0% ⬆️ +40.4%

Compared against master branch baseline

Generated at: 2026-01-22T00:11:03.837Z

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

  • Total failures analyzed: 34

Analysis of SpacetimeDB Benchmark Test Failures

Rust / rustdoc_json Failures

Compile/Publish Errors

Failure Group: t_002_scheduled_table and t_017_scheduled_columns

Generated Code:

use spacetimedb::{ReducerContext, ScheduleAt, Table};

#[spacetimedb::table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    scheduled_id: u64,
    scheduled_at: ScheduleAt,
}

#[spacetimedb::reducer(init)]
pub fn init(ctx: &ReducerContext) {
    ctx.db.tick_timer().insert(TickTimer {
        scheduled_id: 0,
        scheduled_at: ScheduleAt::RepeatMicros(50_000),
    });
}

Expected Example:

use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
use std::time::Duration;

#[table(name = tick_timer, scheduled(tick))]
pub struct TickTimer {
    #[primary_key]
    pub scheduled_id: u64,
    pub scheduled_at: ScheduleAt,
}

#[reducer]
pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> {
    Ok(())
}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) -> Result<(), String> {
    ctx.db.tick_timer().insert(TickTimer {
        scheduled_id: 0,
        scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()),
    });
    Ok(())
}

Error Message:
publish_error: spacetime publish failed (exit=1)

Explanation of Differences:

  • The reducer and table macros were not correctly defined with required attributes such as #[reducer].
  • The generated code used ScheduleAt::RepeatMicros instead of ScheduleAt::Interval(Duration::from_millis(...)).

Root Cause:
The documentation is unclear regarding the required attributes for reducers and tables, especially with respect to scheduling functionality.

Recommendation:
Update documentation to include examples of proper usage patterns for scheduled tables and reducers:

  • Specify correct syntax for the ScheduleAt enum options.
  • Illustrate with more examples showing usage of the #[table] and #[reducer] attributes.

Other Failures

Failure Group: t_003_struct_in_table and t_012_spacetime_product_type

Generated Code:

use spacetimedb::{ReducerContext, Table, SpacetimeType};

#[derive(SpacetimeType, Clone)]
pub struct Position {
    x: i32,
    y: i32,
}

#[spacetimedb::table(name = entity)]
pub struct Entity {
    #[primary_key]
    id: i32,
    pos: Position,
}

#[spacetimedb::reducer(init)]
pub fn init(_ctx: &ReducerContext) {}

Expected Example:

use spacetimedb::{table, SpacetimeType};

#[derive(SpacetimeType, Clone, Debug)]
pub struct Position {
    pub x: i32,
    pub y: i32,
}

#[table(name = entity)]
pub struct Entity {
    #[primary_key]
    pub id: i32,
    pub pos: Position,
}

Error Message:
schema_parity: reducers differ - expected [], got ["add_entity()", "init()"]

Explanation of Differences:

  • Missing pub in struct fields (x, y, id), which hindered schema detection in the DB.
  • Lack of Debug and Clone traits in output.

Root Cause:
Documentation should specify visibility requirements for struct properties, especially for those that are part of a schema.

Recommendation:
Revise docs to explain the importance of visibility modifiers (pub) in struct definitions. Provide clear examples where struct attributes must be public to function properly.


Failures Specific to Scheduled Columns and Constraints

Failure Group: t_016_sum_type_columns

Generated Code:

use spacetimedb::{ReducerContext, Table};

#[derive(spacetimedb::SpacetimeType)]
pub enum Shape {
    Circle(i32),
    Rectangle(Rect),
}

#[spacetimedb::table(name = drawing)]
pub struct Drawing {
    #[primary_key]
    id: i32,
    a: Shape,
    b: Shape,
}

Expected Example:

use spacetimedb::{reducer, table, ReducerContext, SpacetimeType};

#[derive(SpacetimeType, Clone, Debug)]
pub enum Shape {
    Circle(i32),
    Rectangle(Rect),
}

#[table(name = drawing)]
pub struct Drawing {
    #[primary_key]
    pub id: i32,
    pub a: Shape,
    pub b: Shape,
}

Error Message:
Error: no such table: 'drawings'

Explanation of Differences:

  • Missing pub in struct fields (id, a, b), which prevents proper schema definition.

Root Cause:
The documentation is unclear about the need for visibility specifiers in enums and structs used within the DB context.

Recommendation:
Add details to documentation highlighting that all fields in struct and enum definitions should be marked as public for proper database interaction. Provide consistent examples.


Rust / docs Failures

Timeout Issues

Failure Group: Multiple Test Timeouts

Common Problem:
Several tests including t_001_basic_tables and t_020_ecs experienced timeouts during execution.

Root Cause:
This likely relates to complex processing without sufficient optimization or poorly defined database schema causing query execution to take too long.

Recommendation:

  1. Include benchmarks or performance expectations in documentation for various queries.
  2. Suggest practices or patterns for optimizing database schema and queries to avoid timeouts.

C# / docs Failures

Other Failures

Failure Group: t_014_elementary_columns and t_016_sum_type_columns

Generated Code:

using SpacetimeDB;

public static partial class Module
{
    [SpacetimeDB.Table(Name = "Primitive", Public = true)]
    public partial struct Primitive
    {
        [SpacetimeDB.PrimaryKey]
        public int Id;
        public int Count;
        public long Total;
        public float Price;
        public double Ratio;
        public bool Active;
        public string Name;
    }
}

Expected Example:

using SpacetimeDB;

public static partial class Module
{
    [Table(Name = "Primitive")]
    public partial struct Primitive
    {
        [PrimaryKey] public int Id;
        public int Count;
        public long Total;
        public float Price;
        public double Ratio;
        public bool Active;
        public string Name;
    }
}

Error Message:
Error: no such table: 'primitive'

Explanation of Differences:

  • Extraneous Public = true parameter in the attribute definition.
  • Missing proper struct visibility modifiers.

Root Cause:
Documentation does not clarify the effects of the visibility modifier on table definitions.

Recommendation:
Revise the documentation to clarify:

  1. The specific usage of attribute parameters and their implications.
  2. Clarify that struct properties must be public to be correctly recognized in the DB schema.

This analysis identifies specific issues in implementation and documentation that led to benchmark failures in SpacetimeDB. Addressing these will enhance clarity and improve user experience in building SpacetimeDB applications.

clockwork-labs-bot and others added 2 commits January 22, 2026 00:11
…ents

The context builder was including absolute file paths which caused hash
mismatches when the tool was compiled/run in different environments
(different workspace paths between CI runners). This fix uses relative
paths (stripped of the base directory prefix) so hashes are stable.

This will require re-running the benchmark after merge to update the
saved hashes with the new stable values.
@cloutiertyler
Copy link
Contributor Author

/update-llm-benchmark

@clockwork-labs-bot
Copy link
Collaborator

LLM Benchmark Results (ci-quickfix)

Language Mode Category Tests Passed Task Pass %
Rust rustdoc_json basics 25/27 83.3% ⬆️ +38.9%
Rust rustdoc_json schema 23/34 65.3% ⬆️ +38.8%
Rust rustdoc_json total 48/61 75.2% ⬆️ +38.9%
Rust docs basics 5/27 11.1%
Rust docs schema 7/29 15.5% ⬇️ -5.0%
Rust docs total 12/56 13.1% ⬇️ -2.3%
C# docs basics 27/27 100.0% ⬆️ +38.9%
C# docs schema 22/34 63.7% ⬆️ +32.2%
C# docs total 49/61 83.5% ⬆️ +35.8%

Compared against master branch baseline

Generated at: 2026-01-22T01:34:07.820Z

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

  • Total failures analyzed: 35

Analysis of SpacetimeDB Benchmark Test Failures

Rust / rustdoc_json Failures

Compile/Publish Errors

1. Test: t_002_scheduled_table

The generated code:

use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};

#[table(name = tick_timer, schedule(column = scheduled_at, reducer = tick))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    scheduled_id: u64,
    scheduled_at: ScheduleAt,
}

#[reducer]
pub fn tick(_ctx: &ReducerContext) {
    // Scheduled reducer invoked by tick_timer
}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
    ctx.db.tick_timer().insert(TickTimer {
        scheduled_id: 0,
        scheduled_at: ScheduleAt::RepeatMicros(50_000),
    });
}

The golden example:

use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
use std::time::Duration;

#[table(name = tick_timer, scheduled(tick))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    scheduled_id: u64,
    scheduled_at: ScheduleAt,
}

#[reducer]
pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> {
    Ok(())
}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) -> Result<(), String> {
    ctx.db.tick_timer().insert(TickTimer {
        scheduled_id: 0,
        scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()),
    });
    Ok(())
}

Error: publish_error: spacetime publish failed (exit=1)

Explain the difference:

  • The incorrect schedule attribute used improper syntax (column and reducer mentioned separately).
  • The tick reducer should accept a TickTimer row parameter.
  • The scheduled_at assignment is incorrect (incorrect type used).

Root cause:
Documentation does not clarify the need to use the scheduled attribute correctly and specify function signatures for reducers expected by the API.

Recommendation:
Update documentation to “Must use attribute scheduled(tick) and ensure the reducer functions accept the required parameters as specified.” Example:

#[table(name = tick_timer, scheduled(tick))]
// correct usage

2. Test: t_017_scheduled_columns

The generated code:

use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};

#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    scheduled_id: u64,
    scheduled_at: ScheduleAt,
}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
    if ctx.db.tick_timer().count() == 0 {
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: ScheduleAt::repeat_us(50_000),
        });
    }
}

#[reducer(scheduled)]
pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {}

The golden example:

use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
use std::time::Duration;

#[table(name = tick_timer, scheduled(tick))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    pub scheduled_id: u64,
    pub scheduled_at: ScheduleAt,
}

#[reducer]
pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
    let every_50ms: ScheduleAt = Duration::from_millis(50).into();
    ctx.db.tick_timer().insert(TickTimer {
        scheduled_id: 0,
        scheduled_at: every_50ms,
    });
}

Error: publish_error: spacetime publish failed (exit=1)

Explain the difference:

  • Incorrect schedule attribute syntax causes failures in specifying a reducer.
  • The scheduled_at type is incorrectly set to repeat_us, not conforming to the Interval structure.

Root cause:
Lack of detailed guidelines in documentation for correctly annotating and defining reducer routes expected during schema description.

Recommendation:
Include a guide section in the documentation detailing all attributes for tables and reducers, specifically emphasizing parameter types and structure expectations.


Other Failures

3. Test: t_003_struct_in_table

The generated code:

use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};

#[derive(SpacetimeType)]
pub struct Position {
    x: i32,
    y: i32,
}

#[table(name = entity)]
pub struct Entity {
    #[primary_key]
    id: i32,
    pos: Position,
}

#[reducer]
pub fn add(ctx: &ReducerContext, id: i32, x: i32, y: i32) {
    ctx.db.entity().insert(Entity { id, pos: Position { x, y } });
}

The golden example:

use spacetimedb::{table, SpacetimeType};

#[derive(SpacetimeType, Clone, Debug)]
pub struct Position {
    pub x: i32,
    pub y: i32,
}

#[table(name = entity)]
pub struct Entity {
    #[primary_key]
    pub id: i32,
    pub pos: Position,
}

Error: schema_parity: reducers differ - expected [], got ["add()"]

Explain the difference:

  • The reducer function should not be necessary if there is no reduction logic within the example.
  • The Position struct lacks appropriate visibility modifiers, leading to nil results in the API.

Root cause:
Documentation does not clarify when reducers are expected and how to define and declare public fields necessary for SpacetimeDB API to function correctly.

Recommendation:
Clarify in the documentation that reducers should only be included when necessary, along with showcasing fields with visibility for structs.


4. Test: t_013_spacetime_sum_type

The generated code:

use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};

#[derive(SpacetimeType)]
pub struct Rect {
    width: i32,
    height: i32,
}

#[derive(SpacetimeType)]
pub enum Shape {
    Circle(i32),
    Rectangle(Rect),
}

#[table(name = result)]
pub struct ResultRow {
    #[primary_key]
    id: i32,
    value: Shape,
}

#[reducer]
pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
    ctx.db.result().insert(ResultRow {
        id,
        value: Shape::Circle(radius),
    });
}

The golden example:

use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};

#[derive(SpacetimeType, Clone, Debug)]
pub struct Rect {
    pub width: i32,
    pub height: i32,
}

#[derive(SpacetimeType, Clone, Debug)]
pub enum Shape {
    Circle(i32),
    Rectangle(Rect),
}

#[table(name = result)]
pub struct ResultRow {
    #[primary_key]
    pub id: i32,
    pub value: Shape,
}

#[reducer]
pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
    ctx.db.result().insert(ResultRow { id, value: Shape::Circle(radius) });
}

Error: sum_type_row_parity: spacetime sql failed: Error: no such table: result

Explain the difference:

  • Visibility modifiers are absent on the Rect and Shape structs.
  • The naming convention for the struct is inconsistent with usage.

Root cause:
Documentation does not specify the significance of visibility in structs and enums directly influencing the usage of their produced database entries.

Recommendation:
Amend documentation to highlight the importance of public fields and specifically demonstrate the effect of struct and enum visibility.


Additional Recommendations

  1. Documentation Structure: Enhance the overall organization and clarity in the documentation relating to syntax rules. Providing clear examples of common pitfalls with API usages can prevent such issues.

  2. Consistency: Ensure that examples maintain consistent use of visibility and thorough descriptions of error types relevant to expected outcomes.

  3. Error Handling: Clarify the expected patterns and structures for error handling within API calls and data flows.

By implementing these documentation changes and clarifications, debug routines should become more intuitive, resulting in fewer benchmark test failures.

@cloutiertyler cloutiertyler merged commit 3b9497e into master Jan 22, 2026
25 of 26 checks passed
@Centril Centril deleted the tyler/fix-llm-benchmark branch January 22, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants