Major upcoming functionality for `mystjs`

[rowanc1]

Over the last several months and at the last two team meetings @fwkoch, @stevejpurves and myself have been showing some of the new work on mystjs, which you can see/demo/try here: https://js.myst.tools

There is a ✨ton✨ of new functionality, including:

• interactive demos in the docs
• word/docx export
• intersphinx support
• hover wikipedia links
• auto linking of citations
• export to latex and PDF documents
• a new library for templating latex called jtex
  • 422 journal templates in the new myst-templates org (try jtex list)
  • For an overview of how to create templates, we put together a webinar/walkthrough yesterday
• the ability to create and deploy jupyter-book style websites like https://js.myst.tools
  • We have written about the overlaps with JupyterBook here
  • Can you use mystjs with JupyterBook? Yes!
  • Can you use mystjs with Sphinx extensions, no not really, however, they are compatible at the intersphinx/objects.inv layer and many of the common JupyterBook and sphinx design tools have been ported already.
• A major focus on performance and accessibility

Where we are today

A lot of the code for mystjs was created for the Curvenote CLI (MIT licensed), some background on that here, and currently, much of the code for mystjs is actually currently hosted in https://github.com/curvenote/curvenote. Over the past few months we have been working (primarily with @choldgraf, @fperez, @colliand and @mmcky) to upstream this functionality into mystjs, this has taken the form of pulling out relevant functionality into non-curvenote-based tools that can be both open in the community and source-code sense.

By end of October

This will create some major changes to the mystjs repository. Our plan is to move to a mono-repository (using turbo + npm workspaces), with the mystjs repo containing the following packages:

• citation-js-utils utility functions to deal with citations
• intersphinx a command line tool and package for reading and writing o
• jtex a templating library (see docs)
• mystjs the current MyST parser, with extensibility
• myst-cli this will provide CLI functionality for myst build mystdoc.md
• myst-cli-utils some shared utils between intersphinx, jtex, and myst-cli
• myst-common Some common utilities for working with ASTs
• myst-frontmater definitions and validation for scientific authorship/affiliation frontmatter (see docs)
• myst-templates types and validation for templates (including jtex, and some word templates coming soon!)
• myst-to-docx convert myst documents to word docs!
• myst-to-react create basic, ideally unthemed react components for content only
• myst-to-tex convert myst to latex, to be used in combination with jtex to create stand alone latex documents
• myst-transforms a number of transformations for use with myst AST to transform, e.g. links, citations, cross-references, admonitions
• simple-validators validation utilities, that print all sorts of nice warnings

All of these are already packages that are available on npm, and if you want to contribute or poke around in the next month, we welcome PRs in the current repo!

Major pieces of work left

• Split out the CLI from Curvenote specific functionality. The current CLI allows you to pull content from Curvenote APIs, authenticate, deploy etc. This won't be part of the mystjs CLI.
• Allow for templates for the website. Currently there are a few assumptions for the default Curvenote theme. The myst-to-react tools are pretty close to being able to be reused across templates, but the web-server/theme needs to be split out more effectively. (nested under this is another task to improve the routing for content pages, which has too many assumptions built in and needs to be generalized).

In Screenshots

Export of MyST markdown files to PDFs, with 422 journal templates in the new myst-templates org

Cross-references with content, which works across pages (powered by a myst-spec content API)

Interactive citations, figures, and table references.

MyST demo component, which can now download word docs as well completely in the browser. :)

Questions and Feedback?!

Some of the push for this is to have robust javascript tools that are accessible in the future by tools like Jupyter (e.g. extending jupyterlab-myst) and interactive scientific journals. In working through this there is certainly overlap with the existing tools in the JupyterBook/MyST ecosystem. We have been investing in standardizing MyST as a markup language and AST, to ensure compatibility, but I am sure this effort opens up questions/concerns/uncertainty/feedback. I think what probably would be appropriate would be a blog article announcing some of this work in October, and it would be helpful to have some feedback on what we should address or spend time thinking through before a wider announcement. I would love feedback!

I am super excited by some of the developments recently in the MyST/JupyterBook community, and am glad that we can contribute to this community. Thanks and looking forward to feedback/questions! :)

[choldgraf]

Thanks @rowanc1 for sharing these ideas with the rest of the team. I'm also excited about seeing how far we can push the envelope on the JS side!

Agreed that a blog post is a good idea so that we can signal the intentions of the project in an intentional way. Perhaps that blog post should be closing action for this issue?

It also makes me think that we we should formalize the strategy and direction of the EBP organization, so that we can signal to external stakeholders and align our internal team on the direction in which we'd like to head. I've created an issue to track that here:

• Create a strategy and goals section in our meta/ repository #839

I also want to share two quick clarifications:

• MyST JS is a pre-alpha codebase that is not yet stable. So folks shouldn't consider this "a release" or anything...the docs there are an experimental prototype but are not yet a source of truth for the myst spec. The project is not yet being run inclusively so that it can prioritize quick development and prototyping (though we do welcome feedback). I opened up an issue to track this: Clarify that this is pre-alpha software jupyter-book/mystmd#61
• MyST JS has also had technical contributions from many other people other than the group @rowanc1 lists above (which have mostly been involved in project discussions about how to upstream pieces of curvenote into executablebooks...many thanks to the project's technical contributors as well.

[echarles]

As the foundations are being laid down, I'd like to ask about the choice of Remix.run as runtime engine. I think it is a good time to introduce that question and I understand you are extracting/refactoring the existing code base to be moved to executablebooks/myst repositories. There are other runtime engines I'd like to consider to render myst documents, like next.js, docusaurus. Is it possible to already factor the code so that the default current implementation is remix.run, while keeping the door open to other runtime engines in the future? I can guess this is not an easy architecture setup as remix.run may be used a lot to do the hard work, so just asking now.

[rowanc1]

Thanks for the feedback so far. A few of my responses below. :)

Agreed that a blog post is a good idea [...] [and] should be closing action for this issue?

Thanks @choldgraf, yes, I think that makes a lot of sense!

formalize the strategy and direction of the EBP organization

Love it, I am excited to contribute or review there, and help out where I can. There is probably some pre-work from the PIs of the project to do there, which I am sure some of the vision can come from the original grant proposal and be edited with the updated context.

One of the goals that we did set together at the team meeting was to make forward, collaborative progress on the MEP process over October, with the suggestion of three documents being created for review next month. This can hopefully dove-tail in with that work.

The project is not yet being run inclusively so that it can prioritize quick development and prototyping (though we do welcome feedback).

We do of course do try to be inclusive, and this is part of us trying harder to evolve the project in that direction! However as @choldgraf you point out, there is a big difference between a project run by a single stakeholder/company where most communication is internally focused and a community driven/governed project. We are aware of those differences (and are also getting coaching/pointers from @choldgraf on this!) and I am excited to get this into a place where the community can evolve it together. Thanks @choldgraf for calling out the stage we are at, and to your next 👇 point, we should do a better job of documenting where we are at in the current docs.

MyST JS is a pre-alpha codebase that is not yet stable

I will respond and address some of the concerns in updating the docs today to have appropriate advisory content, and respond on the issue you made.

has also had technical contributions from many other people

Yes of course, and sorry if it came across that way. My main intention on calling those folks out was that they are not represented in any of the issues or commits, and that a lot of the efforts going into this are not captured by github contribution graphs. Certainly there are many technical contributors, as well as users finding bugs or asking for support that have helped shape and evolve the project thus far. Looking forward to many more contributions in the future as well. 🚀

@echarles: I'd like to ask about the choice of Remix.run as runtime engine

Glad to see you looking into the code, thanks. :) I am loving working with Remix at the moment and it is a good fit for the style of websites like jupyterbook/docs-sites/scientific papers etc. The current way things are built is tied to remix to closely, and as you mention, there are a lot of other themes and frameworks (nextjs, vue, static-html, docusaurus, existing sphinx-themes...). Removing the "assumptions for the default Curvenote theme" is exactly this piece of work -- removing remix.run and treating it only as a templating layer that you can choose (or not) to use.

I think that we will get to a good starting place when we move things over to the @exectuablebooks community, but it certainly is only a starting place, and having other themes/frameworks to target, and ideally get contribution on how to shape that would be ideal. I am looking forward to working with you.

I can guess this is not an easy architecture setup as remix.run may be used a lot to do the hard work

I am hoping we can take a similar approach to how we have put effort into the templating of latex documents, where there aren't too many assumptions by mystjs tooling, and there is the possibility of large diversity of community approaches and contributions.

From a technical perspective, this means splitting out the JSON AST creation, and providing hooks to trigger the templating libraries. We are just working through some of the details on this now and I will both tag you on some of our experiments over in the curvenote repo this month if you have feedback, as well as open an issue in mystjs to have more discussion. The other package where there is something to do is in myst-to-react and how to deal with special images/links which each react framework often provides their own tools (e.g. for hosting/minifying images, or pre-loading pages for faster UX). I think this will be done through injecting those react components specifically through a small interface, or overriding the default components completely -- or just taking inspiration from the tools/hooks/providers and leaving it all up to the theme. Anyhow, still lots to figure out there - I will open an issue for more detailed discussion and to get your thoughts on this.

[echarles]

Super quick feedback after reading your reply, I would not put the theme aspects at the same level of complexity as the framework level taking back your words (unless with theme, you mean something like rendered). Prolly a good start is to have a common vocabulary.

Taking back the list you have put in #793 (comment) parsers, transformers, renderers, react components, latex, word, ... esm/cjs/jest debugging?!?, I would love to see a diagram with those components linked together (what depends on what, what feeds what).

This would allow to setup the language, as the scope/target. e.g. woudl the JSON AST be feeded to RemixRender or NextRenderer... (change the names as you want).

In terms of scope, we know React.js is not the only framework, but it sounds reasonable to target that only in first instance, while letting the door open for later technology like Vue.js (this is a common pattern in some projects). Same for Remix which could not be the only React technology to cover. In terms of myst-to-react, the question will also popup around the toolkit to choose (Material-UI, Chakra-UI...). There is that concept of Headless-UI to give more freedom to the user to integrate the components in its look-and-feel, but there also a choice needs to be made as there are many providers for such approach.

[rowanc1]

unless with theme, you mean something like rendered

Yes, I think so. This is very much in the JAM Stack style (javascript, apis, markdown), with the interface being between the J and A. That is, MyST's job is to provide renderable data interface to any template (I think this is clear for LaTeX, however, in the case of the web this is actually a really complex framework, with potentially many themes, etc. OR it could equally be a static html jinja template).

I would love to see a diagram with those components linked together

I will pull something together for discussion of where we are at now, and welcome feedback to improve and get on the same page about terminology. There is a diagram for the latex templating, which sort of illustrates this (maybe helpful?). Regardless, @fwkoch and I will pull some docs together over the coming weeks.

It sounds reasonable to target [react as the] first instance

This was my thinking as well, and is a broad place to start that gives significant differences to the things that are possible with JupyterBook today. I am not sure what the next one would be, and it might make sense to actually focus more on the static HTML templating with jinja even to gain community trust/understanding/buy in -- react/remix/nextjs are pretty far from how things are done today (mostly bootstrap, jinja, jquery), so that is also on my mind a lot lately.

the question will also popup around the toolkit to choose

Yep totally, the Curvenote theme/renderer is using headless-ui, remix and tailwind. Many, many choices out there for high-quality UI frameworks (currently impressed by radix, for example!). I think there are going to be a lot of opinions and options in this space, and MyST / EBP can add value by providing well documented and structured data that can easily be rendered (however complex that rendering process might also be!).

If you go to a page in the docs, you can add .json to the end of the URL and get a sense of the "API" layer. Which is what is coming out of mystjs, and is based on myst-spec, JATS, and schema.org. (compare rendered and the json). This also powers the citations/cross-references, which can render open, dynamic content with this protocol in other places.

[echarles]

Regardless, @fwkoch and I will pull some docs together over the coming weeks.

That would be super helpful. Thx!

currently impressed by radix, for example

Same here, I see it at a lot of different places.

[chrisjsewell]

Heya, firstly this looks great, theres some really awesome stuff here, but...

Agreed that a blog post is a good idea so that we can signal the intentions of the project in an intentional way. Perhaps that blog post should be closing action for this issue?

I feel if this is a blog post emitting from EBP, there should be a very clear message of what this means for JupyterBook.

For example, should users be expecting to switch from using the jupyter-book CLI to myst-cli in the future?
What about developers working on "python/sphinx based extensions", should they stop? (see also #837)

Honestly, for myself as a developer, I would like to know. Because at the moment I'm unclear if I should dedicate much more time to working on/maintaining the "Python side" of EBP, if it will eventually just become deprecated.

You say

Can I use JupyterBook and mystjs together? yes

I would say

no, its not clear what that means

There is certainly overlap, but non-technical users should be given one clear way of doing things.
Indeed, we are creating the MyST specification with the hope that it will indeed allow for multiple implementations of it.
But within the same organisation, I fear that may sow confusion, to effectively have "competing" tools.

Perhaps, there are clear use cases for using one or the other? Then they should be made clear.

There are three things that come to mind that I don't see implemented:

1. Configuration: jupyter-book (and underneath sphinx) has a wealth of configuration options that people use (https://jupyterbook.org/en/stable/customize/config.html)

2. Extensibility: The is no clear, user-facing, extensibility mechanism in mystjs, akin to sphinx extensions. If I assume you are still using https://github.com/executablebooks/markdown-it-docutils (?), then as its creator, it can tell you its really not particualrly great in the respects. This is exactly why I started to work on https://github.com/executablebooks/unified-myst/tree/main/packages/core-parse#extension-mechanism

3. Notebook execution (and caching). This page is very limited on details: https://js.myst.tools/guide/interactive-notebooks, can you in fact execute notebooks during the build process?
   Obviously a big part of jupyter-book, on top of just being a document renderer, is its integration with Jupyter Notebooks (https://jupyterbook.org/en/stable/content/executable/index.html)
   Is mystjs, just intended to be a document renderer (i.e. just the https://github.com/executablebooks/MyST-Parser part) or is it also envisaged to eventually take on more Notebook integration aspects (i.e. the https://github.com/ExecutableBookProject/MyST-NB part)
   Also, what about text-based notebooks? https://jupyterbook.org/en/stable/file-types/myst-notebooks.html#structure-of-myst-notebooks

Again, it comes back to the clarity of messaging to users and developers:
Its cool to show we are doing cool stuff, and I am certainly behind a potential "full" transition to mystjs
But I'm wary of muddying the waters with our user base at this perhaps early stage.

This thinking also obviously feeds into #839

[chrisjsewell]

(also another feature that I just thought of, that sphinx/jupyterbook offers but does not appear here, is multilingual support)

[choldgraf]

Thoughts from Chris H on how these two tools relate to each other

I'd like to help disambiguate some ambiguity I see above about the direction of the Python and JavaScript aspects of this stack. I don't think that it is helpful at this moment to compare them feature-for-feature, they are at very different stages of development and they also focus on different use-cases.

There is no intention to deprecate or move on from the Python / Sphinx stack in the short-term. Rather, I see investing time in the JavaScript implementation of MyST as a way to explore what is possible with a different implementation and technical stack, and to understand what implications this has for functionality that is important to the mission of the project, as well as for maintainability and sustainability of the project. Choosing a stack comes with a set of limitations, and we should explore what are the inherent limitations of Sphinx and what is easier to do with a different technical foundation.

In the short / medium term, I believe that Jupyter Book and MyST JS focus on two related but distinct use-cases. Here's a short overview from my perspective.

Jupyter Book focuses on multi-page books or documentation. It is more "informal" in nature, and is useful for people that do not need to abide by a particularly strict protocol for the structure of their outputs.

MyST JS focuses on single-document articles with more formal, semantic, structured outputs. This is more important for scholarly publishing and also makes it easier to render outputs in many different forms (like many different Latex template outputs). It also explores what it looks like to build a document engine with a more modern technical stack.

In the short term, I think we should develop both of these efforts simultaneously, using the MyST Specification as a unifying force to ensure they retain overlap in their core functionality and syntax, but allowing each to explore their respective focus-area independently. We should use project resources to support both of these efforts, though in the short term MyST JS is significantly less-developed and I think it makes sense to dedicated resources there to bring it up to speed.

In the medium- and long-term, it could be that MyST JS grows to have a kind of "feature parity" with Jupyter Book's Python implementation, or it could be that the ideas in MyST JS make their way into the Python side as well. If that is the case, then the team should discuss and decide whether it is in the long-term strategic interests of the project to just "choose" one of them, or to keep growing them independently. But I think that day is a ways off as we still have much to learn and to try.

In short: I think it's important that we both invest time and resources into our core infrastructure, but also that we keep our eye on the horizon to understand what future technologies will position us for the most impact. Thus far we have focused most project efforts on the Python implementation of MyST and Jupyter Book, and @rowanc1's proposal above represents another incremental push now in the direction of the JavaScript stack and a different use-case. Both are sticking around for the foreseeable future, and we should continue to assess the strategic value of each for the project as we learn more.

[chrisjsewell]

Thanks for your thoughts @choldgraf

Jupyter Book focuses on multi-page books or documentation ...
MyST JS focuses on single-document articles ...

Yep sounds good, and indeed perhaps something to this effect should be mentioned in any blog post and possibly on https://www.myst.tools/

I don't think that it is helpful at this moment to compare them feature-for-feature

I really feel that it is.
I stress that this is certainly not meant to be disparaging of mystjs, it's to aid its development.
The point is that of "upfront design" vs "iterative design"; both have their pros and cons, and essentially should be used in tandem.
We can't know and account for every feature that we might eventually want, but we do have a pretty good idea, from our existing work on jupyter-book etc
If we don't put in the due diligence, to think about features we might want in mystjs (and perhaps ones we don't), and how there may be ways to design for their eventual incorporation, my fear is we run the risk of creating a "sphinx 2.0", that gets very complex and difficult to use/maintain when we later try to push such features in.

I would note there has already been discussions around these kinds of thoughts in e.g. jupyter-book/mystmd#34

we should explore what are the inherent limitations of Sphinx and what is easier to do with a different technical foundation.

Here I would link to #837 (comment), for my analysis of the sphinx design.
I'm certainly not the biggest fan of sphinx 😅, but conversely there are aspects that are implemented for a reason, and we should also be thinking "what are the aspects of sphinx we should be implementing in any other foundation"

I hope that makes sense