Proposal: A Different Extension Model

[stianhoiland]

re: #17

Aka. "It's tempting to live in your editor, but have you tried living in your shell?"

Recently I dove deep into and embraced working on the command line--yes I know, I know, I'm late--and consequently I've experienced insights and dramatic changes to my notions of HCI and productivity. Like many, much of my usage of the command line revolves around the text editor, but I think my experience is different from most and worth reflecting on. The difference lies in the approach to building productivity by complimenting and extending tools. In the following I don't intend to make any claims to originality, only to cast light on, after having discovered for myself, the ways of the Old Ones.

I wouldn't bother you with this proposal if I didn't think that this project could potentially affect many people for a long time, nor if I didn't think that the alternative extension model I want to propose is substantially more useful and sustainable.

After my personal "zero to hero" on the command line, I now think that all productive development environments require scripting with an interpreted language and that the significance of this is both completely misunderstood and taken for granted. The common way to implement this is by embedding an interpreter and exposing key infrastructure in the host software, and thus allow the user to extend and modify its usage by writing scripts in the chosen interpreted language targeting the exposed API. The software itself hosts and owns the interpreter instance, and thus the extension mechanism.

Vim with VimScript, Emacs with Emacs Lisp, Neovim with Lua, and other software using Python, Ruby, JavaScript and in-house abominations.

These days this model and its compiled plugin counterpart are taken for granted, and I think the reason is GUI.

What unites programs in the GUI paradigm is the presentation of their output inside rectangles, input captured by interaction within those rectangles, and the manipulation of the display of those rectangles. That's the interaction platform of the GUI, and within this environment the GUI hosts its denizens and, let me say it again, unites them in being interactable rectangles.

What unites the programs in a GUI and underlies that platform is not scriptability, but graphical interaction, and therefore scriptability must be the concern of each program individually. The programs are not hosted within an environment that already provides a platform for interpreted programming/scripting, the capabilities of which they could have passively consumed. Instead, they are hosted within an environment that already provides graphical interaction, the capabilities of which they passively consume. The GUI is not founded on programmability/scriptability.

Enter the command line shell.

The shell is fundamentally and natively an interpreted environment. This is hard to overstate. From the bottom to the top the shell exists as an interpreter, within which it hosts its denizens. Whether writing singular command lines on the prompt or arbitrarily assembling command lines in files to be read and interpreted sequentially, the shell is programmability. As such there is no need for command line programs to concern themselves with hosting their own scriptability--this is already provided natively by their host environment. (Although of course they can, as we'll later understand, there is no reason to host their own.)

In fact, much can come from command line programs deriving their scriptability from the shell's hosted scripting environment rather than from their own isolated private scripting silos. Much like how GUIs provide a unified elegant graphical interaction paradigm, shells provide a unified elegant scripting/programmability paradigm. (Recall my conviction that all productive development environments nontrivially and unobviously require scripting.)

The extension model here is very different. Rather than exposing its infrastructure inwardly and hosting scripts that access its API, to be composed with and extended by other denizens of the shell's provided native programming environment, programs must instead expose themselves to the host environment, providing/serving their essential state and data in an easily digestible manner.

This is a model that favors "small" programs. If your program's data and state is huge and complex then serving it performantly and meaningfully becomes hard. I challenge you to think of a text editor in an almost Platonic way: What is essential to a text editor? What is the essential, nonreducible state and data that describes a text editor and nothing else? This and only this is what should be exposed to the other denizens of the shell. Anything else they could by definition gather or derive from elsewhere. To close out this thought: There is not only beauty in small programs--of which Edit could be one--but the size of programs is self-fulfilling. When your programs become big, they need to become bigger; and when your programs are small, they stay small. This is because big programs do not delegate their concerns to other programs, while small programs do.

An example of what much can come of a (small) program deriving its scriptability from the shell is that by having access to the program's essential state through the shell, a user can install any standalone interpreter, like Lua, Python, Ruby, PHP, etc., and write their extensions and scripts in whatever language they want, in effect extending the program's scriptability to support any scripting language the user fancies, today and in the future. Indeed, enabling the user to use any extension method.

The shell already provides a scriptable, interpreted environment, replete with a language specialized for composing programs--like say, Edit and whatever programs that do whatever you want to extend Edit with. And do not underestimate the utility of being able to slap together a cmd/PowerShell/Bash script to extend your text editor--I could write a whole 'nother one of these on that topic. You don't need to ship Lua. Not only do you already have an interpreted language--a TUI text editor necessarily runs in a shell--you also get an impressively more powerful extension model by architecting for its use. With there conceivably never being a time when Edit no longer use the console subsystem, there is no reason for Edit to start off on the wrong foot and pretend to be a GUI program (in this technical sense) that needs its own scripting silo.

Before this gets any longer, I will end it, though with more to say!

For empowering users now and in the future, I hope this proposal will be considered.

Relevant: #89, #90 (and others I might collect later)

[Valdsonjr]

This reminds me of the Kakoune philosophy (under 'A better unix citizen') and I would love something like that for this one.

More on the kakoune philosophy, including some shortcomings

[lhecker]

Similar to the proposals of using sockets, IPC/RPC, scripting languages, or external processes overall, the overall problem is that you can easily write a .so/.dll plugin that interfaces with shell processes, but you can't un-shell-process a shell-process if that's the only extension model that you expose.

Put differently, kakoune's model is nice, but I propose there's a missing abstraction layer in between the core editor and its scripting model. I'm currently 90% certain that I'll be implementing such a layer and I wouldn't mind, in fact I'd be happy, if we can then build something on top that's like what you propose, or the things that others propose. It's absolutely 100% possible to add such functionality in my future vision.

Crucially, what I must make absolutely clear is, I don't want "low overhead" for plugin calls, I want "no overhead". It has to be as if it was part of the editor to begin with.

That said, this is only my current vision and that may change over time, e.g. to what you propose. But that's how I currently feel about it.

[rhubarb-geek-nz]

a TUI text editor necessarily runs in a shell

You can run a text editor without a shell, all it needs is a parent process to execute it and a terminal/console for user input and output.

rhubarb-geek-nz     17915   17870  0 16:39 ?        00:00:00 sshd: rhubarb-geek-nz@pts/1
rhubarb-geek-nz     17916   17915  0 16:39 pts/1    00:00:00 github/edit/edit-1.0.0-x86_64-linux-gnu

You can even set an editor to be your default shell so simply logging in will run your editor.

This used to be a common practice for providing secure admin consoles where you login and the only thing you can do is run the program they give you.

Likewise git can use an editor for the commit comments, that does not make git a shell.

[stianhoiland]

@Valdsonjr Thanks so much for the links. I enjoyed reading them! Although I've never used Kakoune, it's been on my radar, both for its selection-first modal editing and its peculiar flavor of extensibility. Interestingly, even Kakoune makes its users fall into the temptation I mentioned in the alternative title of my proposal here: "It's tempting to live in your editor, but have you tried living in your shell?"

@rhubarb-geek-nz I actually had a vague niggle in the back of my mind when I wrote that sentence and you've pointed out to me why. (Also, I'm guessing that git shells out.)

You've also illustrated my point: The circumstances when an interactive command line program require their own scripting facilities are exceptional and are related to its isolation from the shell--a scenario/use case I would say in most cases isn't just not worth the additional complexity, but which also has unexpected and accidental compounding negative effects (see OP about small vs. large programs) which today are taken for granted (every app and their mom gets an interpreter tossed into the mix just in case!) These negative consequences are nontrivial, are not the easiest things to formulate, and have more to do with benefits insidiously and unwittingly lost through fragmentation than with any obvious, direct drawback. "Reimplement all the things (in subtly incompatible, nonintegrating ways)!"

@lhecker I'm not sure I followed this:

the overall problem is that you can easily write a .so/.dll plugin that interfaces with shell processes, but you can't un-shell-process a shell-process if that's the only extension model that you expose.

But re-reading it now, maybe I do understand, and I'll (hopefully) address it below. What do you mean by "un-shell-process"-ing?

You also wrote:

I propose there's a missing abstraction layer in between the core editor and its scripting model.

I'm unsure whether we're having this conversation on the same level. We're simultaneously talking about levels of abstraction wrt. to a plugin architecture/extension model and two different interaction paradigms. I want to meet you in seeing the value of the planned extension model aka. "small core editor + extension loader", and I want you to meet me in considering the value--if any--in a different paradigm altogether. Not because I want to push a different paradigm, but because I want you to be able to just consider it and thereby make a more informed decision about how to proceed with this part of the project. Edit doesn't have to go this route; I'll be implementing this in my own dream editor. But what I'm proposing is a Good Idea™ worthy of at least consideration, and, again, I wouldn't have bothered with this if I didn't think Edit could affect many people for a long time.

I'd be happy, if we can then build something on top that's like what you propose

This part I think I understand and it makes sense. With a "small core editor + extension loader" a plugin could be made which would sort of make Edit into such an application as Kakoune: The plugin would extend Edit with Kakoune-like extension points like piping text in and out of the active buffer, running arbitrary commands in spawned subshells, IPC with fifo pipes, etc. I like and agree with this idea! Hopefully this understands your idea.

But although this indeed touches on concrete functionality I've proposed, in the end it is only superficially aligned/similar because my proposal is paradigmatically or qualitatively different.

I won't necessarily be able to formulate that any better today. It isn't an easy thing to convey. It's an altogether different way of seeing these issues that is inherently concerned with extension and programmability as principles unto themselves (and therefore also concerned with "how to do it right"), not just wrt. a particular application like a text editor and its plugins. A particular issue/PR/commit (#102, #290, dd28e1c), as well as your comment here about "empty state+open folder+fuzzy finder" has me inspired to do a simple YouTube video with a demonstration of this thing that's so hard to formulate. Maybe that will help the discussion. Praying for some motivation to do that ;)

[stianhoiland]

I did the thing. Here's a link to my YouTube video The SHELL is the IDE. It's unlisted (but reachable with that link) until I make a better thumbnail and such. Hope to have some discussion.