Repository navigation
Support upload of large databases (> 2GB) #1383
Description
Currently the codeql-action crashes when trying to upload a CodeQL database bundle that is larger than 2GB. This is because the codeql-action tries to load the entire zip file into a Buffer which is limited to 2GB in Javascript.
We should probably rewrite:
codeql-action/src/database-upload.ts
Lines 43 to 45 in 4fddc51
and supply the payload to the POST request as a streaming input.
Creating zip file at C:\a\_temp\codeql_databases\csharp.zip.
Error: File size (8048141762) is greater than 2 GB
RangeError [ERR_FS_FILE_TOO_LARGE]: File size (8048141762) is greater than 2 GB
at new NodeError (node:internal/errors:371:5)
at tryCreateBuffer (node:fs:419:13)
at Object.readFileSync (node:fs:464:14)
--->at uploadDatabases (C:\a\_actions\github\codeql-action\v2\lib\database-upload.js:51:28)<---
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async run (C:\a\_actions\github\codeql-action\v2\lib\analyze-action.js:184:9)
at async runWrapper (C:\a\_actions\github\codeql-action\v2\lib\analyze-action.js:235:9) {
code: 'ERR_FS_FILE_TOO_LARGE'
Activity
aeisenberg commentedon Nov 21, 2022
What is the max db size that turboscan can handle?
rneatherway commentedon Nov 22, 2022
We believe that the limit is 5GB due to the need to use multipart uploads above that size. However, we haven't tested around those sizes yet.
robertbrignull commentedon Jan 10, 2023
This particular issue should be fixed now. Though note that uploads of multi-gigabyte databases is still not fully supported, so it will currently fail for other reasons. We'll be working on improving this over time so larger databases can be uploaded.