Add --eot_line option

[shirayu]

Added --eot_line option.
This allows multi lines input.

For example, we can use the following prompt with --eot_line EOT .

Please follow the instructions.

1. aaa
2. bbb
3. ccc
EOT

[austinvhuang]

There's definitely a need for this. If you notice in the shell demo there's this tr '\n' ' ' step to workaround the the issue.

It does change the function interface slightly and presence/absence of whitespace like "\n" can have subtle effects on quality, so the details require a bit of care.

[austinvhuang]

This is really useful. let's bring this into dev - if folks have thoughts on the comments below or anything else feel free to chime in.

[austinvhuang]

can eot_line be const?

[austinvhuang]

@jan-wassenberg the way the CLI arg parsing works, are there certain characters that would be inexpressible (eg if you wanted some other whitespace like '\t' or ' '?

Another consideration to think about is, is end-of-prompt best specified as:

• a string?
• a character?
• a token ID?

I wonder if either a character or token ID might be preferable to string (and then '\n' becomes a special case), but open to having other perspectives chime in here.

OTOH the nice thing about using a line string is that:

• it can easily be entered in interactively (not always the case for a rare token/character)
• if you need to allow for the entire vocabulary, you can effectively use a unique-by-design string as a quasi-out-of-vocabulary sentinel token.

[jan-wassenberg]

I suppose we can post-process the string to implement escaping, then we can express anything, right?

[austinvhuang]

I think this makes inputs that don't end in "\n" inexpressible. This might be okay, but would be good to better understand the implications for quality.