Preserve command-line history after interpreter crash

When the CPython interpreter crashes - recent history entries are lost. An easy reproducer (on any Linux):

$ ulimit -v $((1024*256))
$ pip install -q gmpy2
$ python  # started with empty history
Python 3.14.0a2+ (heads/main:e2713409cf, Dec  2 2024, 07:50:36) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from gmpy2 import mpz
>>> mpz(2222222222222211111111122222222222222)**33322222
GNU MP: Cannot allocate memory (size=503998640)
Aborted
$ python  # history is empty, again :(
...

That "works" both in the old repl and in the new repl. Sometimes you even can't recover all session from the terminal screen.

IMO, if it's a feature - it's a misfeature. I would expect, that all entered input will be preserved in the history. Or such behavior can be optionally turned on. BTW, this feature "doesn't work" in the IPython shell.

Linked PRs

gh-127495: Append to history file after every statement in PyREPL #132294

I'm wondering if this has something to do with bash. I'm not sure what processing the python-shell does when it is interrupted, but the handling of history is in bash.

No, it's not about bash at all, it's history is fine in the given example. It's all about python repl (>>> ) history.

The shell session shown here just to demonstrate a close-to-real-life example (extension doesn't handle well out of memory condition and abort the python interpreter process). But you could just do kill -SIGABRT <python interactive interpreter> (or it's analog in Windows).

I seem to understand that you may need a file similar to .history to store the history of the previous pyrepl. Even if the interpreter crashes, it can be restored through the file.
Or similar methods to recover the loss of records caused by crashes.

you may need a file similar to .history to store the history of the previous pyrepl.

Perhaps. I was planning to work on the issue, but I've no review-ready patch yet. Let me know if you would like to work on this.

you may need a file similar to .history to store the history of the previous pyrepl.

Perhaps. I was planning to work on the issue, but I've no review-ready patch yet. Let me know if you would like to work on this.

I'm not going to deal with that, feel free to do so. But I will take this opportunity to learn pyrepl :)

I seem to understand that you may need a file similar to .history to store the history of the previous pyrepl.

There's .python_history already, which normally stores the history. The issue is that it's written at interpreter exit, which doesn't happen in case of a crash (or os._exit()):

cpython/Lib/site.py

Lines 578 to 586 in 53908bd

    
           def write_history(): 
        
               try: 
        
                   readline_module.write_history_file(history) 
        
               except (FileNotFoundError, PermissionError): 
        
                   # home directory does not exist or is not writable 
        
                   # https://bugs.python.org/issue19891 
        
                   pass 
        
           atexit.register(write_history)

@pablogsal

This patch works for me:

diff --git a/Lib/_pyrepl/simple_interact.py b/Lib/_pyrepl/simple_interact.py
index a08546a931..b6e6023f09 100644
--- a/Lib/_pyrepl/simple_interact.py
+++ b/Lib/_pyrepl/simple_interact.py
@@ -31,7 +31,7 @@
 import sys
 import code

-from .readline import _get_reader, multiline_input
+from .readline import _get_reader, multiline_input, write_history_file


 _error: tuple[type[Exception], ...] | type[Exception]
@@ -143,6 +143,10 @@ def maybe_run_command(statement: str) -> bool:

             input_name = f"<python-input-{input_n}>"
             more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single")  # type: ignore[call-arg]
+            try:
+                write_history_file()
+            except (FileNotFoundError, PermissionError):
+                pass
             assert not more
             input_n += 1
         except KeyboardInterrupt:

@pablogsal, I'm on the right way? BTW, as noted in the description: I think it's rather a bug than the feature request. The IPython shell e.g. keeps history well.

This patch works for me:

Can you do a quick check of whether it impacts performance by pasting a long block of e.g. a = "a" * 100 lines? If it's writing to the history file once per line (not sure it is), it might slow things down (which probably isn't a big issue for the REPL use case).

If there's a significant impact, it might be better to write history when a new history entry is created, as IIUC it would only write once for the whole long block.

it might be better to write history when a new history entry is created

It already on that way: writing is not per-line, but per history entry.

On another hand, using something like readline.append_history_file() would be better, of course.

Edit:

Version with append_history_file() to play with

diff --git a/Lib/_pyrepl/readline.py b/Lib/_pyrepl/readline.py
index be229488e5..deb2bfb618 100644
--- a/Lib/_pyrepl/readline.py
+++ b/Lib/_pyrepl/readline.py
@@ -89,6 +89,7 @@
     # "set_pre_input_hook",
     "set_startup_hook",
     "write_history_file",
+    "append_history_file",
     # ---- multiline extensions ----
     "multiline_input",
 ]
@@ -446,6 +447,7 @@ def read_history_file(self, filename: str = gethistoryfile()) -> None:
                         del buffer[:]
                     if line:
                         history.append(line)
+        self.set_history_length(self.get_current_history_length())
 
     def write_history_file(self, filename: str = gethistoryfile()) -> None:
         maxlength = self.saved_history_length
@@ -457,6 +459,19 @@ def write_history_file(self, filename: str = gethistoryfile()) -> None:
                 entry = entry.replace("\n", "\r\n")  # multiline history support
                 f.write(entry + "\n")
 
+    def append_history_file(self, filename: str = gethistoryfile()) -> None:
+        reader = self.get_reader()
+        saved_length = self.get_history_length()
+        length = self.get_current_history_length() - saved_length
+        history = self.get_reader().get_trimmed_history(length)
+        f = open(os.path.expanduser(filename), "a",
+                 encoding="utf-8", newline="\n")
+        with f:
+            for entry in history:
+                entry = entry.replace("\n", "\r\n")  # multiline history support
+                f.write(entry + "\n")
+        self.set_history_length(saved_length + len(history))
+
     def clear_history(self) -> None:
         del self.get_reader().history[:]
 
@@ -526,6 +541,7 @@ def insert_text(self, text: str) -> None:
 get_current_history_length = _wrapper.get_current_history_length
 read_history_file = _wrapper.read_history_file
 write_history_file = _wrapper.write_history_file
+append_history_file = _wrapper.append_history_file
 clear_history = _wrapper.clear_history
 get_history_item = _wrapper.get_history_item
 remove_history_item = _wrapper.remove_history_item
diff --git a/Lib/_pyrepl/simple_interact.py b/Lib/_pyrepl/simple_interact.py
index a08546a931..c168549bec 100644
--- a/Lib/_pyrepl/simple_interact.py
+++ b/Lib/_pyrepl/simple_interact.py
@@ -31,7 +31,7 @@
 import sys
 import code
 
-from .readline import _get_reader, multiline_input
+from .readline import _get_reader, multiline_input, append_history_file
 
 
 _error: tuple[type[Exception], ...] | type[Exception]
@@ -143,6 +143,10 @@ def maybe_run_command(statement: str) -> bool:
 
             input_name = f"<python-input-{input_n}>"
             more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single")  # type: ignore[call-arg]
+            try:
+                append_history_file()
+            except (FileNotFoundError, PermissionError):
+                pass
             assert not more
             input_n += 1
         except KeyboardInterrupt:

pythongh-127495: append to history file after every successful statem…

PR is ready to review: #132294

Related d.p.o thread: https://discuss.python.org/t/better-pyrepl-history-file-format/87918

gh-127495: Append to history file after every statement in PyREPL (GH…

Navigation Menu

Repository navigation

Preserve command-line history after interpreter crash #127495

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def write_history():
	try:
	readline_module.write_history_file(history)
	except (FileNotFoundError, PermissionError):
	# home directory does not exist or is not writable
	# https://bugs.python.org/issue19891
	pass

	atexit.register(write_history)

Preserve command-line history after interpreter crash #127495

Description

Linked PRs

Activity

rruuaanng commented on Dec 2, 2024

skirpichev commented on Dec 2, 2024

rruuaanng commented on Dec 2, 2024

skirpichev commented on Dec 2, 2024

rruuaanng commented on Dec 2, 2024

devdanzin commented on Apr 8, 2025

skirpichev commented on Apr 8, 2025

devdanzin commented on Apr 8, 2025

skirpichev commented on Apr 8, 2025

skirpichev commented on Apr 9, 2025

skirpichev commented on Apr 11, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions