Faied to input emoji in windows terminal (and vscode)

Environment

PSReadLine: 2.1.0
PowerShell: 7.1.3
OS: Microsoft Windows 10.0.21354
BufferWidth: 120
BufferHeight: 30

Last 36 Keys

 c d Spacebar D : | Backspace \ E x p Tab Enter
 c d Spacebar m i n i Tab Enter
 g i t Spacebar i n i t Spacebar - b Spacebar � Enter

(real command line: git init -b 💡 (Don't paste directly, input with windows emoji panel (win + .)), windows experimental unicode mode is NOT enabled)

Exception

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83D at index 12 to specified code page.
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass83_0.<WriteHistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Boolean overwritten)
   at Microsoft.PowerShell.PSConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)

Neko Hz (@codehz)

Neko Hz (@codehz) Did you use the hotkey windows+. to paste the emoji? If yes I can reliably recreate that.

Otherwise copying and pasting emoji works normally. The only problem is when I use that hotkey.

Unable to translate Unicode character \uD83D

those codepoints are a surrogate key. It looks like it's encoding as UTF-16BE, which seems weird, powershell defaults to UTF-16LE for WinPS and Utf8 for Ps

I tried each of the following, then re-tried win+. They all reproduce the error.

$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = [Text.Encoding]::GetEncoding('utf-16')
$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = [Text.Encoding]::GetEncoding('utf-16le')
$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = [Text.Encoding]::GetEncoding('utf-8')

Working: From clipboard

🐒> pwsh -nop -nol                              
PS C:\Users\cppmo_000\Documents\2020\powershell>
>> '🐒' * 3                                     
🐒🐒🐒

Crashes: Using `win+.` popup menu

### Environment                                                                                                                                                                                                     
PSReadLine: 2.2.0-beta2                                                                                                                                                                                             
PowerShell: 7.1.2                                                                                                                                                                                                   
OS: Microsoft Windows 10.0.19041                                                                                                                                                                                    
BufferWidth: 120                                                                                                                                                                                                    
BufferHeight: 44

Last 200 Keys

 Ctrl+v Ctrl+Backspace Ctrl+Backspace Ctrl+Spacebar p r e d RightArrow LeftArrow Enter                                                                                                                              
 Spacebar Ctrl+Spacebar RightArrow RightArrow RightArrow LeftArrow LeftArrow Enter                                                                                                                                  
 Enter                                                                                                                                                                                                              
 UpArrow Ctrl+Backspace Ctrl+Spacebar Enter                                                                                                                                                                         
 Enter                                                                                                                                                                                                              
 UpArrow UpArrow Shift+Home Ctrl+x g i t Spacebar s t a t u s Enter                                                                                                                                                 
 g i t Spacebar a d d d Spacebar Backspace Backspace Spacebar . Enter                                                                                                                                               
 g i t Spacebar c o m Ctrl+a g i t Spacebar d i f f Enter                                                                                                                                                           
 g i t Spacebar c o m m i t Spacebar - m Spacebar ' u p d a t e d Spacebar c o m m a n d s ' Enter                                                                                                                  
 g i t Spacebar p u s h Enter                                                                                                                                                                                       
 UpArrow UpArrow UpArrow UpArrow UpArrow UpArrow UpArrow UpArrow UpArrow DownArrow DownArrow LeftArrow DownArrow UpArrow UpArrow UpArrow UpArrow Ctrl+a Backspace Ctrl+v Home Backspace Ctrl+l Enter                
 Ctrl+v Enter                                                                                                                                                                                                       
 UpArrow Spacebar Backspace Backspace Shift+Spacebar | Shift+Spacebar s e l Tab Spacebar - f Tab Spacebar 1 Backspace 3 Enter                                                                                       
 UpArrow Ctrl+Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace ' Enter 
 Alt+. Escape Ctrl+Spacebar Ctrl+a Alt+, Escape Alt+, Backspace Backspace Backspace Alt+, Backspace Backspace � LeftArrow LeftArrow RightArrow RightArrow Enter

Exception

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83D at index 0 to specified code page.                                                                                              
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)                                                                                                                            
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)                                                                                                                
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)                                                       
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)                                                                                               
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)                                                                                 
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)                                                                                                                                       
   at System.IO.StreamWriter.Dispose(Boolean disposing)                                                                                                                                                             
   at System.IO.TextWriter.Dispose()                                                                                                                                                                                
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass98_0.<WriteHistoryRange>b__0()                                                                                                                        
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)                                                                                                                   
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Boolean overwritten)                                                                                                         
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)                                             
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()                                                                                                                                                            
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)

Jake Bolton (@ninmonkey)

Jake Bolton (@ninmonkey) yes, I use win+. to input emoji, it seems only works in wsl(with Windows Terminal)(cmd.exe is also failed to receive emoji)

This bug also seems to happen when pasting emojis.

I've seen the discussion. The essence of this problem is that PSReadline cannot correctly recognize surrogate pair characters.

Surrogate pair characters are famous for emoji, but they are also used to represent lesser-known kanji in Japanese (kanji).
Most of them are kanji that we Japanese don't even use. However, for example, the kanji 𩸽 (U+29E3D), which represents the fish"atka mackerel", causes exactly the same error as the emoji.

An example of a list of surrogate pair kanji can be found at https://www.magata.net/memo/index.php?%A5%B5%A5%ED%A5%B2%A1%BC%A5%C8%A5%DA%A5%A2. (Japanese)

I used Windows Terminal and typed echo 🤔 using three terminals: PowerShell, Windows PowerShell, and cmd.exe.
I attach the log and screenshot.
In this example, I tried with 🤔 , but even if 🤔 is 𩸽, the problem does not change.

PowerShell (7.1.3, PSReadLine 2.2.0-beta3)

C:\Users\User> echo �

Oops, something went wrong.
Please report this bug with ALL the details below, including both the 'Environment' and 'Exception' sections.
Please report on GitHub: https://github.com/PowerShell/PSReadLine/issues/new?template=Bug_Report.md
Thank you!

### Environment
PSReadLine: 2.2.0-beta3
PowerShell: 7.1.3
OS: Microsoft Windows 10.0.19042
BufferWidth: 88
BufferHeight: 22

Last 12 Keys
\```
 w e Backspace Backspace Backspace e c h o Spacebar � Enter

\```

### Exception
\```
System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83E at index 5 to specified code page.
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass98_0.<WriteHistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Boolean overwritten)
   at Microsoft.PowerShell.PSConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken, Nullable`1 lastRunStatus)
   
C:\Users\User> echo �

The error is exactly the same as the log attached by the other person.

However, if you enter echo 🤔 before PowerShell initialization is complete, you will see 🤔 successfully. Even if you execute it again from the command history, it is normal.
Even then, if you type or paste 🤔 yourself after initialization, you will get the same error.

Since only one � is displayed, it is presumed that only one of the two character codes of the surrogate pair can be recognized.

PowerShell (7.1.3, remove PSReadLine)

C:\Users\User> remove-module PSReadline
C:\Users\User> echo ��
🤔
C:\Users\User>

The characters are garbled in the input, but they are displayed normally in the output.
If you try to re-execute from the command history, the input will be displayed without garbled characters.

From this, it can be seen that the problem of garbled input is a problem of PowerShell itself, and it is a problem of PSReadline that an error occurs when reading surrogate pair characters.

Windows PowerShell (5.1, PSReadLine 2.0.0-beta2)

PS C:\Users\User> echo ??
🤔
PS C:\Users\User>

The input shows ??, but the output shows it normally.
If you try to re-execute from the command history, the input will be displayed as ?? as well.

If you run remove-module in advance, it will be in the same state as PowerShell (7.1.3, remove PSReadline).

cmd.exe

C:\Users\User>echo ��
🤔

C:\Users\User>

It is in the same state as PowerShell (7.1.3, remove PSReadline).

The characters are garbled in the input, but they are displayed normally in the output.
If you try to re-execute from the command history, the input will be displayed without garbled characters.

Since I am using translation, the text may be unnatural.
I strongly hope that this problem will be resolved as soon as possible.

Dongbo Wang (@daxian-dbw)

When I looked it up, I found something strange.

First of all, this issue does not occur at all in PSReadline 1.2. In addition, PSReadline 1.2 does not cause garbled characters when entering surrogate pair characters. This is the same for both Windows PowerShell 5.1 and PowerShell 7.

While using PSReadline 1.2, I ran Remove-Module PSReadLine to temporary remove PSReadline. Then, although the output display is normal, the input display is garbled.
From this, it is considered that PSReadline 1.2 implemented a measure to prevent the input display from being garbled, but PSReadline 2.x has unknowingly disabled it for some reason.

🔼 PSReadline 2.0.0-beta2 bundled with Windows 10 2004
🔽 PSReadline 2.0.0-beta2 distributed by PowerShellGallery

What's more, strangely, PSReadline 2.0.0-beta2 bundled with Windows 10 2004 seems to be different from PSReadline 2.0.0-beta2 distributed by PowerShellGallery.
The file structure and modification date are significantly different, and the dll file name is Microsoft.PowerShell.PSReadline.dll instead of Microsoft.PowerShell.PSReadLine2.dll.

And I found that only PSReadline 2.0.0-beta2 bundled with Windows 10 2004 did not give an error.
However, as mentioned above, garbled characters will occur when you enter surrogate pair characters.
Using PSReadline 2.0.0-beta2 distributed by PowerShellGallery causes exactly the same issues as PSReadline 2.2.0-beta3.

So there are currently two workarounds for this issue in PowerShell 7.
One is to uninstall PSReadline 2.x and install PSReadline 1.2. The other is to copy PSReadline 2.0.0-beta2 bundled with Windows 10 2004 from C:\Program Files\WindowsPowerShell\Modules\PSReadLine\2.0.0 to the Modules folder in PowerShell 7.

I'd like to avoid downgrading as much as possible, but I can't help it because the previous version solves the problem. If this problem is left unattended, it's a shame.

Dongbo Wang (@daxian-dbw)

I'm not sure if it's the same issue or a different one, (it seems similar)

some unicode characters are interpreted as 2-wide instead of 1-wide, but the resulting additional space(s) appear at the next change of color: (here with 𝔼, there is an additonal space between the red part and the orange one)

due to this same issue, the caret is also not at the right place when typing

### Environment
PSReadLine: 2.1.0
PowerShell: 7.1.3
OS: Microsoft Windows 10.0.19041
BufferWidth: 120
BufferHeight: 30

This bug also seems to happen when pasting emojis.

I met this too.

Environment

PSReadLine: 2.1.0
PowerShell: 7.2.0
OS: Microsoft Windows 10.0.22000
BufferWidth: 232
BufferHeight: 19

Last 195 Keys

 g i t Backspace t t Spacebar c Backspace Backspace Backspace Spacebar c h e c k o u i t Spacebar Backspace Backspace Backspace t Spacebar m a i n Enter
 g i t Spacebar p u l l Spacebar o r i g i n Spacebar m a i n Enter
 g i t Spacebar c h e c k o u t Spacebar d f Backspace Backspace f e Tab Enter
 e s l i n t Spacebar - - i n i t Enter
 g i t Spacebar a d d Spacebar . Enter
 g i t Spacebar c o m m i t Spacebar - m Spacebar " " LeftArrow 添 加 l i n t Enter
 g i t Spacebar a d d Spacebar . Enter
 g i t Spacebar c o m m i t Spacebar - m Spacebar " " LeftArrow � u p d a t e Spacebar r e a d m e Enter
 LeftArrow Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace � Spacebar u p d a t e Spacebar r e a d m e Enter

Exception

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83D at index 15 to specified code page.
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass83_0.<WriteHistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Boolean overwritten)
   at Microsoft.PowerShell.PSConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)

I experienced this too , entering smiley into a string variable in the console line of Visual Studio Code. Since this issue still is open, I don't need to submit a new bug report 😁

PS Version: 5.1.22000.832
PS HostName: Visual Studio Code Host
PSReadLine Version: 2.2.6
PSReadLine EditMode: Windows
OS: 10.0.22000.1 (WinBuild.160101.0800)
BufferWidth: 155
BufferHeight: 31

hello,
I had a problem with the emoji 🙂, it should be the same issue as this one... if not, i can open an issue.

To reproduce, open windows terminal, type echo , paste 🙂 from clipboard, press backspace, press enter.

> echo �

Environment

PSReadLine: 2.1.0
PowerShell: 7.2.6
OS: Microsoft Windows 10.0.22000
BufferWidth: 209
BufferHeight: 50

Last 9 Keys

 e c h o Spacebar � � Backspace Enter

Exception

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD808 at index 5 to specified code page.
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass83_0.<WriteHistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Boolean overwritten)
   at Microsoft.PowerShell.PSConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)

Hi again, at microsoft/terminal they pointed me to this repo for resolution, so this should be the right place to post...

regarding my previous comment, please see here for more details: microsoft/terminal#14084

Navigation Menu

Repository navigation

Faied to input emoji in windows terminal (and vscode) #2358

Environment

Exception

Working: From clipboard

Crashes: Using `win+.` popup menu

Exception

PowerShell (7.1.3, PSReadLine 2.2.0-beta3)

PowerShell (7.1.3, remove PSReadLine)

Windows PowerShell (5.1, PSReadLine 2.0.0-beta2)

cmd.exe

Environment

Exception

Environment

Exception

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Faied to input emoji in windows terminal (and vscode) #2358

Description

Environment

Exception

Activity

ninmonkey commented on Apr 11, 2021

Working: From clipboard

Crashes: Using win+. popup menu

Exception

codehz commented on Apr 11, 2021

Neptune650 commented on Jun 3, 2021

tsukumijima commented on Jul 9, 2021

PowerShell (7.1.3, PSReadLine 2.2.0-beta3)

PowerShell (7.1.3, remove PSReadLine)

Windows PowerShell (5.1, PSReadLine 2.0.0-beta2)

cmd.exe

tsukumijima commented on Jul 9, 2021

MarcMush commented on Jul 11, 2021

SnowWarri0r commented on Nov 19, 2021

Environment

Exception

sikkepitje commented on Aug 26, 2022

aetonsi commented on Sep 26, 2022

Environment

Exception

aetonsi commented on Sep 26, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Crashes: Using `win+.` popup menu