Nrichers/sc 15354/workflows fix notebook execution workflows

[nrichers]

Pull Request Description

What and why?

This PR solves the issue whereby notebook execution and LLM Markdown rendering fail when no generated template schema docs are present but are referenced in an {{< include >}} statement.

The solution:

• Allow check-in of baseline template schema docs by writers — there's always a file to build regardless of where we're building docs
• Make template schema docs generation during CI workflows fail harder if the docs cannot be generated — a failsafe to ensure we don't silently publish older baseline docs

The alternative solution would have been a comment marker of some kind that gets replaced with a file embed or a mostly blank template schema docs baseline, both with more downsides than the solution in this PR. Happy to talk through this, if necessary.

Changes:

• .gitignore — Removed _template-schema-generated.qmd from gitignore so baseline docs can be tracked
• README.md — Minor documentation updates
• scripts/generate_template_schema_docs.py — Modified with stricter validation to fail CI on generation errors
• site/Makefile — Changed pip to python -m pip for environment robustness
• _template-schema-generated.qmd — Added tracked baseline template schema docs
• templates/customize-document-templates.qmd — Removed comment that is no longer applicable

Fixes sc-15354

How to test

1. Verify that the stuff that fails in other PRs completes successfully:

• Notebook execution workflow: https://github.com/validmind/documentation/actions/runs/23829214298/job/69458857557?pr=1263
• Validate LLM markdown render step: https://github.com/validmind/documentation/actions/runs/23829214300/job/69458857537?pr=1263

2. Check that the template schema docs continue to be rendered: Customize email templates

What needs special review?

Dependencies, breaking changes, and deployment notes

Release notes

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[validbeck]

@nrichers I don't think this fixed the issue (with the notebooks, at least) — the notebooks were NOT executed, they were skipped because no files were changed:

Let me pull test with a fake changed notebook file to see if it runs correctly.

[nrichers]

Let me pull test with a fake changed notebook file to see if it runs correctly.

I am not that familiar with our notebook execution step, so I appreciate your extra testing here. The issue should be resolved, as there is now always a file to build for the {{< include >}}.

[validbeck]

I am not that familiar with our notebook execution step, so I appreciate your extra testing here. The issue should be resolved, as there is now always a file to build for the {{< include >}}.

The notebook executions always run for staging and prod, and for PR previews it's a very easy filter for changed files in the notebooks/EXECUTED directory:

    # See if site/notebooks/ has updates
    # Checks the current PR branch against the target branch
    - name: Filter changed files
      uses: dorny/paths-filter@v2
      id: filter
      with:
        base: ${{ github.event.pull_request.base_ref }}
        ref: ${{ github.head_ref }}
        filters: |
          notebooks:
            - 'site/notebooks/EXECUTED/**'

(Just pointing out that again, just because something is documented doesn't make it transparent to the person that didn't do the designing/documenting. :p)

[validbeck]

Nice, it did work: https://github.com/validmind/documentation/actions/runs/23913198659/job/69740245958?pr=1263

Let me remove the test notebook before you merge.

[validbeck]

Actually, I just noticed this race condition:

The parallel render of the notebooks used to take longer than the full site render (which is why we pulled them out of the render and ran them parallel to begin with, to minimize runtime and prevent failures if the notebooks failed), meaning that the executed versions of the notebooks would replace the non-executed versions.

Now, the LLM markdown render step brings the runtime of the full render to longer than the execution, meaning ... the executed notebooks will always be replaced by the non-executed versions.

Can we look into chaining these steps into a separate workflow that only runs if the full site render is successful?

    - name: Install pandoc
      run: |
        sudo apt-get update
        sudo apt-get install -y pandoc

    - name: Validate LLM markdown render
      run: bash llm/render.sh && bash llm/clean.sh
      working-directory: site

[nrichers]

Can we look into chaining these steps into a separate workflow that only runs if the full site render is successful?

Yes, but not in this PR. I added c7a6d11 to prevent our validate workflow from syncing the executed notebooks folder. If that works, I can update staging and prod workflows. Simplest, safest solution.

[github-actions]

Lighthouse check results

⚠️ WARN: Average accessibility score is 0.87 (required: >0.9) — Check the workflow run

Show Lighthouse scores

Folder depth level checked: 0

Commit SHA: 41d20de

Modify the workflow to check a different depth:

• 0: Top-level navigation only — /index.html, /guide/guides.html, ...
• 1: All first-level subdirectories — /guide/.html, /developer/.html, ...
• 2: All second-level subdirectories — /guide/attestation/*.html, ...

Page	Accessibility	Performance	Best Practices	SEO
/developer/validmind-library.html	0.85	0.68	1.00	0.82
/get-started/get-started.html	0.85	0.74	1.00	0.73
/guide/guides.html	0.81	0.68	1.00	0.82
/index.html	0.93	0.61	1.00	0.82
/releases/all-releases.html	0.86	0.69	1.00	0.73
/support/support.html	0.91	0.65	1.00	0.82
/training/training.html	0.85	0.62	0.96	0.73

[github-actions]

Execute training notebooks for PRs

✓ INFO: Live previews are available —

• Open model development series
• Open model validation series

[validbeck]

Looks like that worked — can you apply the same changes to staging & prod (and remove the test notebook)? 🙏🏻

[nrichers]

OK, applied the same --exclude directive for the executed notebooks when we sync to AWS S3 for staging and prod. Looks like we're done here, merging.

[github-actions]

PR Summary

This PR introduces several functional improvements in the documentation deployment and template schema generation processes:

1. GitHub workflow updates:

   • In the production, staging, and PR preview deployment workflows, an additional exclusion pattern (notebooks/EXECUTED/*) is added to the AWS S3 sync commands. This ensures that unwanted notebook artifacts are not uploaded.

2. Template schema generation enhancements:

   • The script responsible for generating the template schema documentation has been refactored (e.g., renaming the output variable from OUTPUT_FILE to TARGET_FILE).
   • Additional sanity checks have been incorporated including a minimum expected file size and validation of HTML structure to catch generation failures early.
   • Post-processing of the generated HTML is performed to remove unnecessary title and heading tags, and to clean up excessive blank lines before embedding into a Quarto document.

3. Documentation updates:

   • The README and associated documentation file have been updated to reflect that the auto-generated template schema now overwrites the baseline output and is injected into the customize-document-templates.qmd file.
   • A section in the repository previously ignored (the generated template schema file) is now tracked and included in the documentation build, ensuring that users see the latest schema content.

4. Build and dependency adjustments:

   • The Makefile now uses python -m pip install instead of plain pip install to improve environment consistency.

Together, these changes streamline the documentation deployment and ensure that the template schema documentation is reliably and accurately generated and updated whenever the backend JSON schema changes.

Test Suggestions

• Manually run the GitHub workflows (or use a staging branch) to verify that the S3 sync commands properly exclude the 'notebooks/EXECUTED/*' directory and that CloudFront invalidation is triggered as expected.
• Execute the updated schema generation script to ensure that it produces an output file with a size above the defined minimum, contains valid HTML (i.e., includes both and tags), and that the post-processing (removal of title and h1 tags) functions correctly.
• Confirm that the changes in the README and the inclusion of the generated file into customize-document-templates.qmd display the documentation correctly when rendered.

[github-actions]

Validate docs site

✓ INFO: A live preview of the docs site is available — Open the preview