Repository navigation
Hit "Inconsistent kallsyms data" for riscv64 debug config #502
Description
gcc: 9.0
rustc: 1.54.0
Hit a "Inconsistent kallsyms data" in the checks:
https://github.com/Rust-for-Linux/linux/runs/3677382723?check_suite_focus=true
I added a simple debug output to show the diff between two ".map":
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index 36ef7b37fc5d..5eb7531aea04 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -440,6 +440,7 @@ if [ -n "${CONFIG_KALLSYMS}" ]; then
if ! cmp -s System.map .tmp_System.map; then
echo >&2 Inconsistent kallsyms data
echo >&2 Try "make KALLSYMS_EXTRA_PASS=1" as a workaround
+ diff >&2 .tmp_System.map System.map
exit 1
fi
fiand result was
Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround
--- .tmp_System.map 2021-09-22 16:00:21.593766590 +0000
+++ System.map 2021-09-22 16:00:21.541766045 +0000
@@ -42127,8 +42127,8 @@
ffffffff8086fe08 R kallsyms_relative_base
ffffffff8086fe10 R kallsyms_num_syms
ffffffff8086fe18 R kallsyms_names
-ffffffff8098d310 R kallsyms_markers
-ffffffff8098d688 R kallsyms_token_table
+ffffffff8098d308 R kallsyms_markers
+ffffffff8098d680 R kallsyms_token_table
ffffffff8098dbe0 R kallsyms_token_index
ffffffff809b25e0 R __begin_sched_classes
ffffffff809b25e0 R idle_sched_class
Some analysis:
link-vmlinux.sh will first run kallsyms (script/kallsyms.c) twice to generate the tables for symbols of the kernel (for kernel itself to lookup), the program kallsyms will try to compress the tables with a simple algorithm, and the tables may be different between different runs (link-vmlinux.sh call it steps). Program kallsyms reads the symbol table of the kernel image, and generate a .S file, which eventually be compiled into an object file. The object file will be linked into the kernel image.
Currently the logic is that if the size of the object file stays the same in step 1 & 2, the compilation continues, otherwise, do an extra run for kallsyms. And in kallsyms data consistency checking, the script compares the address of every symbol (including kallsyms symbols) between the final kernel image and the result of the previous kallsyms step.
What we are hitting here is that the sizes of the object file of step 1 & 2 are the same (so no step 3), but the contents are different. Moreover, kallsyms_names is the table to store kernel symbols, and kallsyms_token_table is the table to store data for compression, so according to the .map diff above, the case we are hitting is the kernel symbols (compressed data) space shrinks 8bytes while the compress utility data space increases 8bytes.
The issue is very hard to reproduce since a small change in symbol table can result in different compressed results. Actually when I tried to randomize (by hand) some function names introduced in my PR, the problem was gone. FWIW, my original PR that hit the issue is as follow: https://github.com/fbq/linux-rust/tree/debug/kallsym-issue
Activity
ojeda commentedon Sep 30, 2021
Thanks a lot for taking a look at this -- I also hit this yesterday with the sync PR (after the CI servers stopped timing out due to networking issues) when trying to reproduce locally the hang on QEMU that happened in the CI.
ojeda commentedon Sep 30, 2021
In my case, the sizes 1 & 2 are different, and so there is a step 3, but the extra step does not help. There is a big difference in size between step 1 and 2 (856 bytes), which then gets reduced to a few bytes between 2 and 3 (16 bytes). Doing a 4th pass finds the fixed point, though.
So we have 2 issues: this one and the one you report above (that the check is too lax and does not notice that step 3 is required). I assume that a forced step 3 (i.e.
KALLSYMS_EXTRA_PASS=1) helped you?ojeda commentedon Sep 30, 2021
One more case: the sizes increase quite a lot (rather than decrease, which is what I saw so far), requiring 5 passes.
ojeda commentedon Sep 30, 2021
One more: there is no fixed point, it oscillates between two sizes. Great!
fbq commentedon Sep 30, 2021
;-) While I was looking at this issue, I also found that I cannot prove that the process has a fix point.
fbq commentedon Sep 30, 2021
Right, I remembered that I did comment out the
ifstatement in scripts/linux-vmlinux.sh, which in fact forced step 3, and the CI passed after that.nbdd0121 commentedon Sep 30, 2021
I suppose RISC-V linker relaxation has something to do with this. What's the diff between the two?
ojeda commentedon Oct 1, 2021
The first few got shifted by 8, everything else after that by 0x20 (alignment, I guess):
I will see tomorrow what are the contents.
CI: disable (riscv64, debug)