Repository navigation
[Discussion] Enable reading workspace contents on GitHub.dev #46421
Description
Activity
Jackson Kearl (@JacksonKearl) Johannes Rieken (@jrieken) On the VS Code side, do you currently have any plans for how to make a readonly view of the workspace contents available to other extensions?
Initializing TS with a single blob seems like it would be the easiest approach for us. If the majority of workspaces only have a few MB or so of code, for the first iteration we could even skip using a
SharedArrayBuffertoo and instead send typescript a simple JSON tree structure that contains all the files and their contents/cc Joyce Er (@joyceerhl) who is owning the remote hub file system implementation.
For anycode this is currently opaque and it simply consumes things via
vscode.workspace.fs. That makes it very nice and simple for me but I actually never really know when I am benefitting from blob-caching or causing individual fetch-requests.For the
SharedArrayBufferapproach there is Safari which doesn't support it... A long, long time back we used indexed-db to share data between various workers since all can access data without postMessage-calls. Tho, there is still some deserialization happening.The implementation side of RemoteHub's indexing is just a
.tar.gzkept in IndexedDB, reading from IDB can't be sync, but once it's read and decompressed, all the data should be there in that one ArrayBuffer. I don't have much experience with SharedArrayBuffers, but I imagine we could pull the tarball into one of them and let many threads have read-only access to it?From talking with Eric Amodio (@eamodio) my understanding is that the tar approach we have is essentially a workaround for the limitations of the GitHub text search APIs and should go away when the GitHub search APIs improve. It's not clear when the search APIs will be improved, but I slightly worry about whether we should really be exposing that and having TypeScript depend on that implementation.
Joyce Er (@joyceerhl) I agree we should consider if exposing the whole contents of the workspace to extension makes sense. With any code also needing this data though, I think it would be nice to see if we can come up with a generic solution instead of having each extension/language re-implement this functionality
I'll keep talking with the TS team about our requirements for this. For experimentation purposes, it sounds like I may be able to hack something together using the remote hub file system provider today though?
Jackson Kearl (@JacksonKearl) and Johannes Rieken (@jrieken) Good suggestion to look into
IndexedDB. @joyceerhl/ Jackson Kearl (@JacksonKearl) Out of curiosity, is the tar stored as a big binary blob in indexeddb, or does Remote Hub generate a more useful data structure to work with it?One quick note from discussions today: the GitHub.dev case is kind of just a special case of working with a virtual file system.
When working with arbitrary virtual file systems, we might not be able to easily or efficiently construct workspace contents blob that can be shared with TSServer. Idea 2 would likely work better in those cases then
The TARs were originally just to deal with the lacking GitHub search apis, but they also became useful to avoid thrashing the GitHub api (and causing lots of latency) when an extension uses the
workspace.fsapis (like anycode). IMO, it would be great to leverage for this scenario as well -- though I'm not sure if a TAR is the "right" way to go or not. As I'm not sure if Azure Repos or other virtual fs' could support it.Matt Bierner (@mjbvz) the on-indexeddb representation is always a
.tar.gz. We deal with it on the RemoteHub side here: https://github.com/microsoft/vscode-remotehub/blob/main/src/search/tar-pit.ts, the package we use for parsing tar's is here: https://github.com/JacksonKearl/TarFileExplorer. We do currently do more post-processing on the remotehub side than is maybe ideal, for instance:- the tar entries come from GH with the repo name prefixed onto the path, we remove that on RH side of things)
- the decompressed file is stored in memory via a sort of debounced
WeakRef
It could be helpful for the vscode extension API to provide some sort of:
type Contents = {path: string, uri, etc. ; contents: UInt8Array}; // maybe a Map instead requestFullRepositoryContents(reason: string): Promise<Contents | null>; /* requestHighBandwidthMode? */
Where io-heavy operations could be gated on VS Code presenting some sort of UI to prompt for entering "high bandwidth" mode, perhaps with a list of the operations that have pending requests for the full contents and why (sorta like workspace trust). They could then either use
workspace.fs(ideal, no code change for them), or read from the returnedContentsdirectly if its a ts-like situation where they need synchronous file reads..As we're thinking about bringing Azure Repos support to parity with GitHub Repos support and generalizing the VirtualProvider concept, I prefer idea 2 for the flexibility it offers as well. (FWIW, the AzDO API supports returning blobs as zip files)
In RemoteHub we have an internal
FileTreestructure which represents the repository contents. (We're currently only using this for GitHub repos, but we should implement it for Azure Repos as well.) We fill out the tree using the tar contents and could expose a method like this on RemoteHub's extension API for the TypeScript extension to access the repo contents:getFileTree(workspaceUri: vscode.Uri): FileTree | undefined;
It seems like the current direction to power this up is #47600. I'll close this issue and if we need to in the future, we can come back to this.
Problem
On serverless web, the TypeScript server currently only knows about the files that VS Code opens as editors. This means that features such as cross-file navigation/IntelliSense will not work unless you open editor for each of the files involved
We've faced this problem with two other features for VS Code: search and our any code implementation (which uses tree sitter to provide limited IntelliSense for languages that don't yet have web enabled extensions). For those cases, we now download a copy of the entire repository to the user's browser so that we can index/process all files in the workspace
Since we already have a copy of the workspace contents, we should see if TypeScript is also able to use this data to provided better intellisense on github.dev. This issue lays out a few potential paths for implementing this.
Goals
Out of scope
node_modules. For this discussions, files undernode_modulesare not considered part of the workspace contentsIdea 1) Initialize TS with the workspace contents
VS Code constructs a data structure representing the workspace contents and transfers this to TypeScript. TypeScript consults this data structure to read any unopened files.
To avoid creating multiple copies of the workspace contents, the data structure could be stored inside a
SharedArrayBuffer. However we'd need to figure out how to represent the workspace data structure since TS would fundamentally just be getting a big blob of binary data that it needs to interpretAdvantages
Idea 2) VS Code as file reader
TypeScript delegates all file operations back through VS Code's TypeScript extension.
The first challenge with this approach that calls from TS back to VS Code would need to be synchronous. This could be implemented using a
SharedArrayBufferto synchronize message passingAdvantages