Core Text rendering issues with macOS 10.14 Mojave

I just did a fresh install of macOS 10.14 Mojave and found that MacVim has a number of rendering issues while scrolling. The test gets garbled and the window can sometimes go completely black.

Seems like the workaround was to disable Core Text rending in the advanced preferences.

This happens only when you compile MacVim with 10.14 SDK (previous SDK works fine). For me the MacVim GUI flashes dark/light grey with desert color scheme in rhythm with the cursor blinking, if "Core Text Renderer" is turned on.

This is because of a Mojave change that made all NSView automatically become layer-backed, which will break MacVim's rendering since it relies on behaviors from the legacy renderer. Fortunately it only happens when you link against the new 10.14 SDK so old builds would still work, but we would need a long term solution, presumably a Metal-based renderer similar to other fancy high-performance terminal emulator like Alacritty or Kitty.

Currently the CoreText renderer also supports a CGLayer mode but it has miscellaneous issues including slow scrolling so I don't think that's the way forward.

More info:

Unfortunately it appears that MacVim is completely unusable on Mojave. Is there any temporary fix?

@amadeus

@amadeus Are you building MacVim yourself using Xcode 10 on Majave? If so, you can build using the old SDK which will fix this issue. Easiest way is to download Xcode 9.4.1 from Apple Developer and build using that. If you are downloading builds from official releases I don't think it should be broken.

Otherwise you can do this in Terminal to turn on CGLayer mode: defaults write org.vim.MacVim MMUseCGLayerAlways -bool YES. I don't recommend this route though as it makes scrolling slow.

@amadeus

@amadeus Download the binary form https://github.com/macvim-dev/macvim/releases

@ychin

@ychin Could you give me a bit more detail on the problem? I'd like to hack a bit on it, although I predict I won't be able to patch it before the regular MacVim committers have fixed it. It seems to me like the issue is someplace in NSWindow not having a background color set? And you want to fix it by dropping Core Text entirely and switching to Metal/Metal2?

@joelfrederico

@joelfrederico I was actually planning to implement a first-pass for a fix for this this weekend since I have been looking at the renderer code when fixing some fullscreen rendering bugs, but if you do want to tackle the issue feel free to do that too.

The way Vim works is that it treats the GUI as a dumb terminal and just issue draw calls to the GUI like it does for a terminal. The way MacVim draws is that it batches up these draw commands and calls setNeedsDisplay to tell Cocoa that it needs to redraw, and then in the callback drawRect it plays back those draw calls Vim sent it. MacVim itself doesn't cache the full state of the texts, only Vim does.

The issue is that these draw commands Vim sent it are only incremental draw commands assuming the previous draws are still valid. When you call setNeedsDisplay technically you are telling the OS to invalidate the whole window even though we are only interested in invalidating part of the window to draw. Previously it somehow worked because for some reason Cocoa didn't clear the whole window for us even when setNeedsDisplay is called when MacVim is in windowed mode. The new APIs always clear the screen when you call that function so you need to know how to redraw the whole screen instead of just playing back the incremental draw commands. Note that this was already a problem in non-native fullscreen which is why we automatically turn on CGLayer mode in that. I think we may kind of fix this by using setNeedsDisplayInRect instead and make sure to set the proper dirty rect, but it has some other issues and doesn't work with scrollRect. Or we could just cache the text states in the renderer so MacVim always knows how to redraw itself.

Another issue is we currently use NSView's scrollRect function to scroll the window (which is deprecated) which I think is GPU accelerated given how smooth MacVim's scrolling is. I think this function no longer work under the new change to layered back view, so we would need to implement scrolling using other ways. If we cache the text states as described above we could just re-draw the texts using CoreText when we scroll but it will be slower than the previous scrollRect implementation since text rendering is slower than a simple Blit operation on the GPU.

The CGLayer implementation (you can manually turn that on by doing defaults write org.vim.MacVim MMUseCGLayerAlways -bool YES) from what I understand was implemented to address these issues when a previous macOS update broke non-native fullscreen mode, by caching the drawn content onto a CGLayer backing. This way you could incrementally draw to the CGLayer, and then just present the layer when you need to draw. Sadly CGLayer is not GPU accelerated and that's why scrolling in CGLayer mode is a lot slower than the normal one on a high resolution display.

All of this is kind of messy which is why I think we should just implement a hardware accelerated renderer (using Metal) to replace CGLayer. We will still use CoreText for the text rendering. We could either just use a GPU texture as backing layer and draw texts on it which ties into the current renderer easier, or we could cache the font glyphs which would allow us to draw texts much quicker but may be tricky to implement (since MacVim supports ligature etc). Another issue with cached glyphs is they won't work (easily) with subpixel font smoothing, which isn't an issue in Mojave since that feature got removed anyway :(, but in older OS we may still want to have subpixel font rendering to be consistent with other parts of the OS.

This is the gist of what I understand of the rendering bug, maybe we should make another issue dedicated to writing a new renderer.

but in older OS we may still want to have subpixel font rendering to be consistent with other parts of the OS.

I think supporting older macOS is mandatory, but I'm not sure how compatible that is with using Metal.

@ychin

@ychin Thanks for the detailed description!

@joelfrederico I was actually planning to implement a first-pass for a fix for this this weekend since I have been looking at the renderer code when fixing some fullscreen rendering bugs, but if you do want to tackle the issue feel free to do that too.

Please don't change your plans thinking I'm working on it! I'm just getting my feet wet, I don't expect to actually be able to do much of anything.

but in older OS we may still want to have subpixel font rendering to be consistent with other parts of the OS.

I think supporting older macOS is mandatory, but I'm not sure how compatible that is with using Metal.

Metal first appeared in macOS El Capitan, but not every graphics card supports it. So we could have current implementation as a fallback if Metal is failed to load or it set in settings before Mojave and Metal backend only on macOS Mojave+.

Download the binary form https://github.com/macvim-dev/macvim/releases

This didn't solve anything on my Mojave machine. MacVim (the whole window) blinks along with the cursor.

Download the binary form https://github.com/macvim-dev/macvim/releases

This didn't solve anything on my Mojave machine. MacVim (the whole window) blinks along with the cursor.

It works, for me. 151 on Mojave.

Any updates on this?

I have tried defaults write org.vim.MacVim MMUseCGLayerAlways -bool YES. It prevents the window from blinking, but scrolling makes the app crash...

I also downloaded 151 binary from https://github.com/macvim-dev/macvim/releases, it doesn't work for me.

@xiaohk

@xiaohk How does the release not work for you? Does it crash or just blink a lot when you type? Do you have an external monitor? You can try to remove the CGLayer setting first by doing defaults delete org.vim.MacVim MMUseCGLayerAlways, and also maybe try a fresh vimrc. I'm mostly asking because I'm running Mojave and the release does work for me.

@ychin

@ychin I think iTerm's Metal renderer uses pre-generated glyph images, it could serve as an example implementation.

My homebrew version (claiming to be 151) on MacOS 10.14.1 was still doing the flashing, until I disabled "Use Core Text Renderer" in the Preferences.

Fix rendering issues when built against the 10.14 SDK.

@oseiberts11

@oseiberts11 disable Use Core Text Renderer makes MacVim horrible for CJK IME user.

@alswl

@alswl yes my vim looks like burnt toast, it's horrible to code in. I'm just hanging in there until it's fixed, hopefully within a week or something :/

My alternative solution for CJK IME user is using vim in Terminal.
😂

@amadeus

@amadeus Are you building MacVim yourself using Xcode 10 on Majave? If so, you can build using the old SDK which will fix this issue. Easiest way is to download Xcode 9.4.1 from Apple Developer and build using that. If you are downloading builds from official releases I don't think it should be broken.

Otherwise you can do this in Terminal to turn on CGLayer mode: defaults write org.vim.MacVim MMUseCGLayerAlways -bool YES. I don't recommend this route though as it makes scrolling slow.

Can't use this version due homebrew error:

Error: Your Xcode (9.4.1) is too outdated.
Please update to Xcode 10.1 (or delete it).
Xcode can be updated from the App Store.

Fix rendering issues with the 10.14 SDK.

Just upgraded via "brew upgrade macvim" and mvim started to this issue. Luckily disabling CORE worked for me. I look forward to the update.

PR #757 should have fixed this issue. Sorry for the wait. Feel free to re-open if it didn't fix the issue. Note that there's a slight performance impact to scrolling but it should be acceptable, and it's a known issue.

Great! I will be watching brew for when the update is available.

@a7hybnj2

@a7hybnj2 I can confirm, if you install the Homebrew version with --HEAD, the fix is included.

How I updated:

brew uninstall macvim
brew install macvim --HEAD --with-override-system-vim --with-luajit
open /usr/local/Cellar/macvim

After that, I drag the MacVim app bundle/icon from the subdirectory of the now open Finder window into my Dock. Done and done.

if you try to install macvim via brew install macvim --HEAD but fail to compile please see #790 (comment)

Navigation Menu

Repository navigation

Core Text rendering issues with macOS 10.14 Mojave #751

29 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Core Text rendering issues with macOS 10.14 Mojave #751

Description

Activity

mario-grgic commented on Sep 24, 2018

ychin commented on Sep 25, 2018

amadeus commented on Sep 25, 2018

ychin commented on Sep 25, 2018

J-Liu commented on Sep 25, 2018

joelfrederico commented on Sep 26, 2018

ychin commented on Sep 27, 2018

chdiza commented on Sep 27, 2018

joelfrederico commented on Sep 27, 2018

eirnym commented on Sep 28, 2018

chdiza commented on Sep 30, 2018

J-Liu commented on Sep 30, 2018

xiaohk commented on Oct 2, 2018

ychin commented on Oct 3, 2018

felixbuenemann commented on Oct 3, 2018

29 remaining items

oseiberts11 commented on Nov 23, 2018

alswl commented on Nov 27, 2018

9mm commented on Nov 27, 2018

alswl commented on Nov 27, 2018

loadaverage commented on Nov 27, 2018

a7hybnj2 commented on Dec 3, 2018

ychin commented on Dec 4, 2018

a7hybnj2 commented on Dec 4, 2018

amcgregor commented on Dec 4, 2018

ApolloTang commented on Dec 5, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions