Set-PSReadlineKeyHandler fails with certain key chords on Unix

Michael Klement (@mklement0)

From Michael Klement (@mklement0) on April 17, 2017 19:44

Steps to reproduce

Define Control and Alt-key-based chords for doubling characters ', ", and ( when typed with Control or Alt held down:

Set-PSReadlineKeyHandler -Chord "Ctrl+'" { [Microsoft.PowerShell.PSConsoleReadLine]::Insert("''") }
Set-PSReadlineKeyHandler -Chord 'Ctrl+"' { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('""') }
Set-PSReadlineKeyHandler -Chord 'Ctrl+(' { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('()') }
Set-PSReadlineKeyHandler -Chord "Alt+'" { [Microsoft.PowerShell.PSConsoleReadLine]::Insert("''") }
Set-PSReadlineKeyHandler -Chord 'Alt+"' { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('""') }
Set-PSReadlineKeyHandler -Chord 'Alt+(' { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('()') }

Expected behavior

The definitions should be accepted on all platforms and function as defined.
E.g. Ctrl+' should result in ''.

Actual behavior

On Windows, the actual behavior matches the expected behavior.

On Unix platforms (verified on macOS 10.12.4 and Ubuntu 16.04 with the respective default terminal applications), the definitions fail with (repetitions of) the following error messages (note that omitting modifier keys ctrl and alt from the definitions would work fine):

Set-PSReadlineKeyHandler : Unrecognized key 'ctrl'. Please use a character literal or a well-known key name from the System.ConsoleKey enumeration.
Set-PSReadlineKeyHandler : Unrecognized key 'alt'. Please use a character literal or a well-known key name from the System.ConsoleKey enumeration.

The error messages suggest that ctrl / alt are not recognized as modifier keys in this case.

Note that something like Ctrl+A does work, however; the problem appears to be the specific characters (', ", () that the modifiers are paired with.

Environment data

PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on Microsoft Windows 10 Pro (64-bit; v10.0.14393)
PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on Darwin Kernel Version 16.5.0: Fri Mar  3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASE_X86_64
PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on Darwin Kernel Version 16.5.0: Fri Mar  3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASE_X86_64

Copied from original issue: PowerShell/PowerShell#3584

I believe this is fixed in #561, but some bindings with Alt will not work correctly because the CLR does not set they KeyChar property properly - tracked by https://github.com/dotnet/corefx/issues/12336.

I'll publish a pre-release version to the PowerShell Gallery as soon as they support pre-releases.

If you'd like to try a fix before that, you can build the lastest yourself from the master branch, or
grab a build from AppVeyor like this one.

Note the version number is now 2.0, so be sure to copy the contents to \PSReadLine\2.0.

I'll have an official prerelease build out next week if you want to verify this is fixed, otherwise you can grab a build out of appveyor here.

Jason Shirk (@lzybkr)

Thank, Jason Shirk (@lzybkr), but it still doesn't seem to work with the 2.0.0-beta1 version from the gallery (published on December 06 2017).

Michael Klement (@mklement0)

Michael Klement (@mklement0) - this might have been fixed later - try the latest build from AppVeyor (linked from the build status badge from the repo readme.md.)

Jason Shirk (@lzybkr)

Thanks, Jason Shirk (@lzybkr) - just tried with the latest AppVeyor build on both macOS 10.13.4 and Ubuntu 16.04, but it still doesn't work, I'm afraid.

The current symptoms are:

The definitions are accepted and do show up in Get-PSReadLineKeyHandler's output.
On pressing something like Alt+', nothing happens.

It looks like Alt+' is affected by the CLR issue I mentioned above, and I hint at where a fix is probably necessary in this comment.

I see. Unfortunately, it doesn't work with Ctrl-based chords such as Ctrl+' either - or is this the same issue?

Not quite - on Linux, Ctrl-' is indistinguishable from '. The command Linux showkeys -a is useful to help me figure out what is possible, what might be a CLR bug, or my bug, in this case, it shows those keys generate the same char sequence which is what the CLR sees after the key press. The CLR then translates the char sequence into a ConsoleKeyInfo.

I thought there was an issue for rejecting key bindings that can't work, but I can't find it at the moment, and to be honest, I'm not exactly sure how I'd implement that on non-Windows platforms.

Jason Shirk (@lzybkr)

Thanks, Jason Shirk (@lzybkr) - showkey -a was illuminating.
sudo showkey -k - note the need for sudo - does show that the actual individual keypresses are all available at a lower level, however.

I'm not sure what's actionable about my Unix-related findings below, but hopefully it provides a few pointers:

`Alt`-based key chords (mapped from `Esc`-based key sequences):

showkey -a tells us that something like Alt-" is translated into a key sequence:

ESC (code point 0x1b)
" (code point 0x22)

And that is indeed how readline functions are often defined; e.g., to represent the above sequence in a readline definition in bash and map it onto inserting "<cursor-pos>", you'd run (I'm using the builtin bind here so that the command can be experimented with interactively, but such a definition is usually placed in ~/.inputrc on Unix platforms):

bind '"\e\"": "\"\"\e[D"'

To invoke this binding, you'd press Esc first and then ".

However, many terminal programs translate Alt-based key chords into these Esc-based key sequences, so that pressing Alt-" - simultaneously - has the same effect.

As for where this translation happens:

Windows:
- mapped by default in Bash on Ubuntu for Windows (WSL)
Linux:
- Mapped by default in the default terminals of many major distros:
  - Ubuntu
  - Debian
  - Fedora (presumably therefore also RHEL)
  - CentOS
- TBD:
  - OpenSUSE
  - RHEL (presumably mapped, assuming it uses the same terminal as Fedora)
  - Arch Linux
  - Kali Linux
  - Others?
- Alternative terminal applications:
  - Note: There may be terminals where the mapping may not even be available on an opt-in basis, through configuration.
  - NOT mapped:
    - XTerm and UXTerm - no obvious way to configure the behavior
macOS:
- Requires configuration, unfortunately:
  - Terminal.app (the default terminal): only if option Use Option as Meta key is turned on via Preferences > Profiles > {active-profile} > Keyboard
  - iTerm.app (iTerm 2): only if Esc+ is chosen for ⌥-key mapping in Preferences > Profiles > {active-profile} > Keys

So the question is: can PSReadLine conversely recognize Esc-based key sequences as Alt-based key chords?

`Control`-based key chords:

Control-based chords - expressed as \C-<char> - are tricky in the Unix world:

Fundamentally, the repertoire of available Control-based chords is limited to caret notation, which means the combinations are limited to the set of ASCII C0 control characters (plus (0x7F))
- In a nutshell (but see caveat below):
  - Control-A through Control-Z (whether Shift is involved doesn't matter).
  - plus a few punctuation-based chords:
    - \C-@ (0x0) also maps to: ⌃`
    - \C-[ (0x1b); i.e, ESC - DO NOT REDEFINE
    - \C-\ (0x1c) also maps to: ⌃4
    - \C-] (0x1d) also maps to: ⌃5
    - \C-^ (0x1e) also maps to: ⌃~
    - \C-_ (0x1f) also maps to: ⌃7 and ⌃/
    - \C-? (0x7f)(⇧-based!!) also maps to: ⌃8
- Notably, something like Control-' is not included.
Even this restricted repertoire above cannot be fully utilized, lest vital terminal functions be interfered with:
- Since the available Control-based chords generate control characters, they implement functions that are vital for an interactive terminal experience:
  - At the very least, the following chords should NOT be redefined:
    * Control-M ... this maps to CR and therefore ENTER - required to submit commands on the command line.
    * Control-I ... this maps to Tab - required for tab completion
    * Control-[ ... this maps to Esc
    * Control-H ... this maps to Backspace
  - In general, any Control-based chord you redefine takes away the ability to use the keyboard to generate the ASCII control character the chord was originally mapped to.

That said, the above is currently hypothetical, because none of the fundamentally available Control-based chords can currently be redefined with PSReadLine on Unix-like platforms:

Example: Redefinition of Control-J to insert the string jump:

# As of PSReadLine v2.0.0-beta.1: 
#   * OK on Windows
#   * BROKEN on Unix-like platforms
Set-PSReadLineKeyHandler -Chord 'Ctrl+j' -ScriptBlock { [Microsoft.PowerShell.PSConsoleReadLine]::Insert('jump') }

By contrast, the equivalent bash definition works fine:

bind '"\C-j": "jump"'

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr):

I thought there was an issue for rejecting key bindings that can't work, but I can't find it at the moment, and to be honest, I'm not exactly sure how I'd implement that on non-Windows platforms.

In light of the above (assuming that no attempt is made to intercept lower-level keyboard events , which may require even more support in CoreFX, if I understand correctly), at least fundamentally unavailable chords could be ruled out, namely:

Control-based chords that do not map onto C0 control characters / 0x7f via caret notation (as described above).

Note that among the Alt-based chords - assuming we get them to work - there may still be situationally unavailable chords, depending on the specific platform (e.g., there are Alt-based chords with system-wide meaning on Ubuntu), but trying to anticipate all of those is impractical.

Michael Klement (@mklement0)

Michael Klement (@mklement0) - A console application can't really depend on the raw keyboard input - a pty usually sits between the application and the hardware. That's why I mentioned showkey -a and not showkey -k.

The character sequences generated by a key press depend on the terminal type. A proper console application use the terminfo database (see man terminfo or man infocmp) to map the sequences to what a user thinks of as a key press. For example, F1 might generate very different sequences for some old obscure terminal or a more typical VT100 terminal.

In the .Net world, we rely on the Console api to do this for us, though it is not yet mature (hence the bug I mentioned previously, also your Ctrl+j example which produces a ConsoleKeyInfo that is identical to Enter.)

When investigating these issues, I start by comparing the info returned by showkey -a with what I get from Console.ReadKey() (which returns a ConsoleKeyInfo).

PSReadLine makes some attempt to normalize the ConsoleKeyInfo to deal with platform specific differences and differing keyboard layouts, but there are still some issues.

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr):

Good to know, thanks.

The character sequences generated by a key press depend on the terminal type.

And many terminals translate chords such as Alt-' into sequences such as Esc, ', so perhaps ConsoleKeyInfo only ever sees the latter?

If so, one conceivable approach is for PSReadLine to listen for the latter instead (while still allowing them to be declared as Alt chords) - is that doable?

Small correction: Ctrl+J maps onto \n (0xA, LF), not ENTER (Ctrl+M,0xD, CR), that's why it's OK to use.

Right - an application reading from stdin only sees the sequences. Keyboards haven't always had an Alt key, so the Escape key was an obvious way to add useful key bindings, and when keyboards evolved, it made sense to generate the same sequence.

PSReadLine could be implemented as you suggest, but it would take on more complexity than I'm willing to take on and there are issues, some unsolvable, if PSReadLine were to rely on sequences everywhere:

Sequences are somewhat limiting, e.g. some popular key bindings on Windows can't work on Linux, see SHIFT+ENTER set to AddLine - line gets executed instead of a new line being added 2.0.0-beta1 #638 and MenuComplete function not showing menu #639.
Sequences would not work on Win7, Win8, or earlier Win10 systems because the console does not provide a way to generate sequences.
Without complex terminfo support, sequences would only support the most common terminals.

PSReadLine actually has some support for sequences so that it can run under tmux on WSL (using the Windows version of PowerShell). The code is here, but it is doesn't (can't) use terminfo because Windows doesn't have terminfo support.

I'm not perfectly happy with how things are today, but I think I've made reasonable choices considering the alternatives.

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr):

Thanks for the background info, as usual.

On gaining a better understanding of the issue you've opened in the CoreFX repo, I do agree that CoreFX is the right place to address these issue, so I've posted a follow-up comment there.

Navigation Menu

Repository navigation

Set-PSReadlineKeyHandler fails with certain key chords on Unix #529

Steps to reproduce

Expected behavior

Actual behavior

Environment data

`Alt`-based key chords (mapped from `Esc`-based key sequences):

`Control`-based key chords:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Set-PSReadlineKeyHandler fails with certain key chords on Unix #529

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Activity

lzybkr commented on Oct 17, 2017

lzybkr commented on Nov 4, 2017

mklement0 commented on Apr 10, 2018

lzybkr commented on Apr 10, 2018

mklement0 commented on Apr 10, 2018

lzybkr commented on Apr 10, 2018

mklement0 commented on Apr 10, 2018

lzybkr commented on Apr 10, 2018

mklement0 commented on Apr 11, 2018

Alt-based key chords (mapped from Esc-based key sequences):

Control-based key chords:

mklement0 commented on Apr 11, 2018

lzybkr commented on Apr 11, 2018

mklement0 commented on Apr 11, 2018

lzybkr commented on Apr 11, 2018

mklement0 commented on Apr 15, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`Alt`-based key chords (mapped from `Esc`-based key sequences):

`Control`-based key chords: