Pasting into Python REPL broken in recent brew Python builds

[tzs]

Bug report

• ran brew update and can still reproduce the problem?
• ran brew doctor, fixed all issues and can still reproduce the problem?

brew config and brew doctor output appended at the end.

What you were trying to do (and why)

Paste multiple lines of text in a terminal into the Python REPL.

What happened (include command output)

Instead of each line being seen as a separate line, it seems to treat the whole paste as one big line. This is a change from earlier Brew builds of Python.

$ cat in
1+2
3+4
5+6

$ pbcopy < in
$ /usr/local/Cellar/python\@3.9/3.9.1_3/bin/python3
Python 3.9.1 (default, Dec 28 2020, 11:22:14) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3+4
5+6

  File "<stdin>", line 1
    1+2
3+4
5+6


       ^
SyntaxError: multiple statements found while compiling a single statement
>>> 

What you expected to happen

Here is what happens using the same version of Python, but installed via the installer from python.org instead of via Brew:

$ /usr/local/bin/python3
Python 3.9.1 (v3.9.1:1e5d33e9b9, Dec  7 2020, 12:10:52) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3
>>> 3+4
7
>>> 5+6
11
>>> 

That's also what happens using Apple's python3.

Step-by-step reproduction instructions (by running brew install commands)

1. Starting from a freshly installed Big Sur or Mojave system, install Homebrew.
2. brew install python3
3. Make a file containing the input shown in the aforementioned output.
4. Get the contents of that file onto the clipboard.
5. Run python3
6. paste

The same problem occurs if you install python@3.8 or python@3.7 via Brew, so this does not seem to be something specific to the relatively new python 3.9.

This problem did not occur 6 to 8 weeks ago.

It is not some sort of problem with the terminal itself. It occurs in Apple's Terminal, iTerm2, and xterm.

One thing that might provide a clue: if pasting something that contains tabs and using iTerm2, iTerm2 normally puts up a dialog asking what to do with the tabs. It is no longer asking when pasting into Brew python. That suggests that whatever has changed in the Brew python builds recently is something that iTerm2 can notice, suggesting the problem is something to do with how python is initializing the tty.

Before posting the issue, I brought this up on the discussion forums to get a sanity check in case I was just doing something stupid, and someone else was able to reproduce it on Brew python 3.8 and 3.9. Here is that discussion in case it might be helpful: Homebrew/discussions#428

Brew config output:

HOMEBREW_VERSION: 2.7.1
ORIGIN: https://github.com/Homebrew/brew
HEAD: fdb71a5cb3029da2ef74d7e09c40575bbfb4ec72
Last commit: 6 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: d924e37f57912b7b1831fc569acd813f00bc84d1
Core tap last commit: 4 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.7tIo8kYpU3/org.macosforge.xquartz:0
HOMEBREW_EDITOR: bbedit_wait
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: quad-core 64-bit kabylake
Clang: 12.0 build 1200
Git: 2.29.2 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
Java: 1.8.152.16, 1.8.0_152
macOS: 11.1-x86_64
CLT: N/A
Xcode: 12.3
XQuartz: 2.7.11 => /opt/X11

Brew doctor output:

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed kegs have no formulae!
This means they were either deleted or installed with `brew diy`.
You should find replacements for the following formulae:
  python@2
  pygtk
  vgo
  pygobject

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  protobuf@3.7

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  python@2
  hackrf
  numpy
  airspy

[carlocab]

Please still include the output of brew config and brew doctor as requested in the issue template. You can edit your original post to do this.

[tzs]

Please still include the output of brew config and brew doctor as requested in the issue template. You can edit your original post to do this.

Done.

[carlocab]

Looking into it some more:

Anaconda Python3.9 doesn't have this bug. Building Python3.9 from source using the release tarball with

./configure
make

doesn't have this bug either.

[carlocab]

This might be a bug in readline.

I did git bisect, which revealed that the bug was introduced here: 1af390c

I started with a known-good commit for python@3.8 at 245d93f, with the known-bad commit at the HEAD of the master branch.

[alebcay]

Maybe it has something to do with how we specifically want Homebrew readline?

homebrew-core/Formula/python@3.9.rb

Lines 142 to 146 in f257d87

	# We want our readline! This is just to outsmart the detection code,
	# superenv makes cc always find includes/libs!
	inreplace "setup.py",
	"do_readline = self.compiler.find_library_file(self.lib_dirs, 'readline')",
	"do_readline = '#{Formula["readline"].opt_lib}/libhistory.dylib'"

I wonder if without this modification, we just use system readline or something like that (or fall back to some system alternative like libedit).

[carlocab]

Checked python@3.9, it also seems to have broken at 1af390c.

Anaconda Python3.9 uses readline 8.0.

[carlocab]

Yep, almost definitely a readline 8.1 bug (or, well, an rlwrap bug which readline exposes):

https://lists.gnu.org/archive/html/bug-readline/2020-12/msg00002.html

Here's a workaround (suggested by the upstream bug report):

echo "set enable-bracketed-paste off" >> ~/.inputrc

Note: if you use bash, I think this will also disable bracketed paste there, which you may not want to do. Here's why: https://cirw.in/blog/bracketed-paste

[sharris40]

Do we want to change the default when configuring readline, or add that as a caveat to readline.rb?

h. Bracketed paste mode is enabled by default. There is a configure-time
option (--enable-bracketed-paste-default) to set the default to on or off.

[fxcoudert]

We don't want to change the default, and caveats are for things that are specific to homebrew. Let's see how the readline/rlwrap people fix that.

[dtrodrigues]

rlwrap is for now changing the default hanslub42/rlwrap#108

[fxcoudert]

an rlwrap bug which readline exposes

but python@3.9 doesn't use rlwrap, does it?

[carlocab]

My understanding is that python@3.9 uses readline, and readline uses rlwrap.

However, I'm not sure how the change in rlwrap defaults in

rlwrap is for now changing the default hanslub42/rlwrap#108

affects readline defaults.

[fxcoudert]

I think rlwrap uses readline:

class Rlwrap < Formula
  desc "Readline wrapper: adds readline support to tools that lack it"
  homepage "https://github.com/hanslub42/rlwrap"