Conversation
|
/update-llm-benchmark |
LLM Benchmark Results (ci-quickfix)
Compared against master branch baseline Generated at: 2026-01-21T19:53:18.713Z Failure Analysis (click to expand)Benchmark Failure AnalysisGenerated from: Summary
SpacetimeDB Benchmark Test Failures AnalysisGroup 1: Rust / rustdoc_json Failurest_002_scheduled_table
t_017_scheduled_columns
Group 2: Rust / docs Failurest_000_empty_reducers
t_001_basic_tables
t_002_scheduled_table (Failure repeated in docs)
t_003_struct_in_table
Group 3: C# / docs Failurest_014_elementary_columns
ConclusionFor all observed failures, the root cause mainly lies within inconsistencies regarding the required syntax, visibility, and expected return types. Recommendations focus on improving the clarity of existing documentation to reduce confusion and ensure compliance with struct and API definitions. |
This prevents CI failures when the rolling nightly updates between when hashes are saved and when CI checks run. The pinned version can be updated intentionally by changing PINNED_NIGHTLY constant.
|
/update-llm-benchmark |
LLM Benchmark Results (ci-quickfix)
Compared against master branch baseline Generated at: 2026-01-22T00:11:03.837Z Failure Analysis (click to expand)Benchmark Failure AnalysisGenerated from: Summary
Analysis of SpacetimeDB Benchmark Test FailuresRust / rustdoc_json FailuresCompile/Publish ErrorsFailure Group:
|
…ents The context builder was including absolute file paths which caused hash mismatches when the tool was compiled/run in different environments (different workspace paths between CI runners). This fix uses relative paths (stripped of the base directory prefix) so hashes are stable. This will require re-running the benchmark after merge to update the saved hashes with the new stable values.
|
/update-llm-benchmark |
LLM Benchmark Results (ci-quickfix)
Compared against master branch baseline Generated at: 2026-01-22T01:34:07.820Z Failure Analysis (click to expand)Benchmark Failure AnalysisGenerated from: Summary
Analysis of SpacetimeDB Benchmark Test FailuresRust / rustdoc_json FailuresCompile/Publish Errors1. Test: t_002_scheduled_tableThe generated code: use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
#[table(name = tick_timer, schedule(column = scheduled_at, reducer = tick))]
pub struct TickTimer {
#[primary_key]
#[auto_inc]
scheduled_id: u64,
scheduled_at: ScheduleAt,
}
#[reducer]
pub fn tick(_ctx: &ReducerContext) {
// Scheduled reducer invoked by tick_timer
}
#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
ctx.db.tick_timer().insert(TickTimer {
scheduled_id: 0,
scheduled_at: ScheduleAt::RepeatMicros(50_000),
});
}The golden example: use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
use std::time::Duration;
#[table(name = tick_timer, scheduled(tick))]
pub struct TickTimer {
#[primary_key]
#[auto_inc]
scheduled_id: u64,
scheduled_at: ScheduleAt,
}
#[reducer]
pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> {
Ok(())
}
#[reducer(init)]
pub fn init(ctx: &ReducerContext) -> Result<(), String> {
ctx.db.tick_timer().insert(TickTimer {
scheduled_id: 0,
scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()),
});
Ok(())
}Error: Explain the difference:
Root cause: Recommendation: #[table(name = tick_timer, scheduled(tick))]
// correct usage2. Test: t_017_scheduled_columnsThe generated code: use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
pub struct TickTimer {
#[primary_key]
#[auto_inc]
scheduled_id: u64,
scheduled_at: ScheduleAt,
}
#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
if ctx.db.tick_timer().count() == 0 {
ctx.db.tick_timer().insert(TickTimer {
scheduled_id: 0,
scheduled_at: ScheduleAt::repeat_us(50_000),
});
}
}
#[reducer(scheduled)]
pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {}The golden example: use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
use std::time::Duration;
#[table(name = tick_timer, scheduled(tick))]
pub struct TickTimer {
#[primary_key]
#[auto_inc]
pub scheduled_id: u64,
pub scheduled_at: ScheduleAt,
}
#[reducer]
pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {}
#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
let every_50ms: ScheduleAt = Duration::from_millis(50).into();
ctx.db.tick_timer().insert(TickTimer {
scheduled_id: 0,
scheduled_at: every_50ms,
});
}Error: Explain the difference:
Root cause: Recommendation: Other Failures3. Test: t_003_struct_in_tableThe generated code: use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};
#[derive(SpacetimeType)]
pub struct Position {
x: i32,
y: i32,
}
#[table(name = entity)]
pub struct Entity {
#[primary_key]
id: i32,
pos: Position,
}
#[reducer]
pub fn add(ctx: &ReducerContext, id: i32, x: i32, y: i32) {
ctx.db.entity().insert(Entity { id, pos: Position { x, y } });
}The golden example: use spacetimedb::{table, SpacetimeType};
#[derive(SpacetimeType, Clone, Debug)]
pub struct Position {
pub x: i32,
pub y: i32,
}
#[table(name = entity)]
pub struct Entity {
#[primary_key]
pub id: i32,
pub pos: Position,
}Error: Explain the difference:
Root cause: Recommendation: 4. Test: t_013_spacetime_sum_typeThe generated code: use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};
#[derive(SpacetimeType)]
pub struct Rect {
width: i32,
height: i32,
}
#[derive(SpacetimeType)]
pub enum Shape {
Circle(i32),
Rectangle(Rect),
}
#[table(name = result)]
pub struct ResultRow {
#[primary_key]
id: i32,
value: Shape,
}
#[reducer]
pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
ctx.db.result().insert(ResultRow {
id,
value: Shape::Circle(radius),
});
}The golden example: use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};
#[derive(SpacetimeType, Clone, Debug)]
pub struct Rect {
pub width: i32,
pub height: i32,
}
#[derive(SpacetimeType, Clone, Debug)]
pub enum Shape {
Circle(i32),
Rectangle(Rect),
}
#[table(name = result)]
pub struct ResultRow {
#[primary_key]
pub id: i32,
pub value: Shape,
}
#[reducer]
pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
ctx.db.result().insert(ResultRow { id, value: Shape::Circle(radius) });
}Error: Explain the difference:
Root cause: Recommendation: Additional Recommendations
By implementing these documentation changes and clarifications, debug routines should become more intuitive, resulting in fewer benchmark test failures. |
Empty commit to fix llm benchmark.