You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Regarding spdlog-rs's ThreadPool, every AsyncPoolSink all use the (only) one internal thread to perform tasks (write and flush) by default, but when the user creates a bit more async sinks, one thread may be a bit overstretched. So we provided an abstraction -- ThreadPool, that allows the user to set up a separate thread for an async sink instead of sharing the internal thread.
Currently, one ThreadPool runs only one thread; it neither gives the user the ability to make a Pool run multiple threads, nor does it do so internally. Because for now, I haven't found any indication that there would be any performance benefit to running multiple threads for a single async sink. But for ease of understanding and extensibility, I've named it as such and reserved the possibility of supporting multiple threads in the future.
@SpriteOvO "Privately" speaking, I like spdlog-rs macro and record design (over log's, or yet, lol), while logforth's Append/Layout/Diagnostic/Filter fits my scenarios better :D
Maybe an ErrorHandle would be introduced later also, but so far, since log macros never return Err, I don't think it's quite necessary rather than leave an eprinln!.
Anyway, log is the official lib and I'd try to improve it directly first. You may read this blog (in Chinese) about the story of logforth if you're interested in.
Currently, one ThreadPool runs only one thread; it neither gives the user the ability to make a Pool run multiple threads, nor does it do so internally. Because for now, I haven't found any indication that there would be any performance benefit to running multiple threads for a single async sink. But for ease of understanding and extensibility, I've named it as such and reserved the possibility of supporting multiple threads in the future.
Good to know. I think typically one thread nonblocking impl is enough. When you let users configure the thread numbers, they don't actually know how to do it properly, or even there is no proper way. Options in open-source are sometimes evil, and cheat users into playing with them with no practical reasons >~<
I have read that blog two weeks ago, my friend found it then told me about it 😆. I honestly admire your motivation to improve log crate.
And I'll also take a look at logforth's implementation when I have time, some things I still haven't figured out how to appropriately design, such as filters, and the log-crate-proxy for spdlog-rs still feels worth refining.
@Phoenix500526 FYI I'm looking into this issue these days. But it should not block you from spending your time, since we can combine both thoughts together and the credits can be shared with Co-authored-by attr.
@Phoenix500526 FYI I'm looking into this issue these days. But it should not block you from spending your time, since we can combine both thoughts together and the credits can be shared with Co-authored-by attr.
Ok, thanks! I’m tied up with some personal things for the next couple of days. I’ll probably check it on Monday or Tuesday.
Seems spdlog-rs uses a global logger setup while the guard is held internally. This may cause some issues when the global static variable getting dropped on program tearing down.
@SpriteOvO I'd appreciate it if you can give some context here how things around Drop works upon program tearing down and where the issues may occur.
I haven't gotten a chance to read the logforth implementation yet, so I'm not sure how logforth does it. I'll explain as much as I can about spdlog-rs, feel free to ask if you have further questions.
First of all, global variables in Rust are different from C++. In Rust, global variables cannot be initialized (also known as calling constructors, except for const) before the main function, and they cannot be cleaned up (calling destructors or drop) after the main function. For some perspective, this is a good thing in Rust, because in C++, the order of initialization of global variables in different translation units is unspecified, Rust avoids this mess.
In spdlog-rs, we have a default logger, which is a global variable that is initialized using OnceCell. We need drop to work on the default logger to ensure that any buffered log records (in BufWriter or asynchronous tasks) are not lost, so we took the C++ principles and implemented it for the default logger -- using libc function atexit to register a callback that will be called before the program terminates normally. In this atexit callback, flushing the default logger once, it can make sure that buffered log records are get flushed.
For asynchronous sinks, additional considerations are needed:
Thread-local stuff is not supported in the atexit callback, in spdlog-rs, AsyncPoolSink uses crossbeam channel to send tasks, which includes the use of thread-local. The solution is that if the program is tearing down (global variable IS_TEARING_DOWN: AtomicBool is set to true in our atexit callback), AsyncPoolSink will flush directly on the current thread instead of sending the flush task to the ThreadPool via crossbeam, thus the use of thread-local is avoided.
In addition to flushing the log records, wait for all of the asynchronous tasks in the ThreadPool to complete to ensure that there are no pending tasks before exiting.
All of these mechanisms only make sense for the global variable default logger, since local variables always get dropped before the main function exits, which performs a flush, and the last sink that owns the Arc<ThreadPool> makes sure that all pending tasks have completed.
UPDATE: More accurately, for asynchronous sinks, the order is: first wait for all tasks to complete, then stop the thread pool from receiving new tasks, and then perform the final flush.
Activity
Phoenix500526 commentedon Sep 5, 2025
I'm interesting in that. Can you assign it to me?
tisonkun commentedon Sep 5, 2025
Just go ahead and submit a patch for review :D
You don't need get assigned to start. And you can follow rust-lang/log#696 for more background.
I suppose this can introduce more design and impls over an owned record.
SpriteOvO commentedon Sep 5, 2025
Nice to see spdlog-rs's idea inspiring elsewhere.
Regarding spdlog-rs's ThreadPool, every AsyncPoolSink all use the (only) one internal thread to perform tasks (write and flush) by default, but when the user creates a bit more async sinks, one thread may be a bit overstretched. So we provided an abstraction -- ThreadPool, that allows the user to set up a separate thread for an async sink instead of sharing the internal thread.
Currently, one ThreadPool runs only one thread; it neither gives the user the ability to make a Pool run multiple threads, nor does it do so internally. Because for now, I haven't found any indication that there would be any performance benefit to running multiple threads for a single async sink. But for ease of understanding and extensibility, I've named it as such and reserved the possibility of supporting multiple threads in the future.
I hope this helps.
tisonkun commentedon Sep 5, 2025
@SpriteOvO "Privately" speaking, I like spdlog-rs macro and record design (over
log's, or yet, lol), while logforth's Append/Layout/Diagnostic/Filter fits my scenarios better :DMaybe an ErrorHandle would be introduced later also, but so far, since log macros never return Err, I don't think it's quite necessary rather than leave an
eprinln!.Anyway,
logis the official lib and I'd try to improve it directly first. You may read this blog (in Chinese) about the story of logforth if you're interested in.tisonkun commentedon Sep 5, 2025
Good to know. I think typically one thread nonblocking impl is enough. When you let users configure the thread numbers, they don't actually know how to do it properly, or even there is no proper way. Options in open-source are sometimes evil, and cheat users into playing with them with no practical reasons >~<
SpriteOvO commentedon Sep 5, 2025
I have read that blog two weeks ago, my friend found it then told me about it 😆. I honestly admire your motivation to improve
logcrate.And I'll also take a look at logforth's implementation when I have time, some things I still haven't figured out how to appropriately design, such as filters, and the log-crate-proxy for spdlog-rs still feels worth refining.
tisonkun commentedon Sep 6, 2025
@Phoenix500526 FYI I'm looking into this issue these days. But it should not block you from spending your time, since we can combine both thoughts together and the credits can be shared with
Co-authored-byattr.Phoenix500526 commentedon Sep 6, 2025
Ok, thanks! I’m tied up with some personal things for the next couple of days. I’ll probably check it on Monday or Tuesday.
tisonkun commentedon Sep 7, 2025
I'm going to remove non-blocking and let syslog/rolling_file appender with a global mutex first.
Then I'll try to add an Async appender like spdlog-rs' one. However, I noticed:
AsyncPoolSinkmay lose the last log records and panic on program exit SpriteOvO/spdlog-rs#64Currently, nonblokcing uses a
WorkerGuardand users should hold it explicitly:Seems spdlog-rs uses a global logger setup while the guard is held internally. This may cause some issues when the global static variable getting dropped on program tearing down.
@SpriteOvO I'd appreciate it if you can give some context here how things around Drop works upon program tearing down and where the issues may occur.
SpriteOvO commentedon Sep 8, 2025
I haven't gotten a chance to read the logforth implementation yet, so I'm not sure how logforth does it. I'll explain as much as I can about spdlog-rs, feel free to ask if you have further questions.
First of all, global variables in Rust are different from C++. In Rust, global variables cannot be initialized (also known as calling constructors, except for
const) before themainfunction, and they cannot be cleaned up (calling destructors ordrop) after themainfunction. For some perspective, this is a good thing in Rust, because in C++, the order of initialization of global variables in different translation units is unspecified, Rust avoids this mess.In spdlog-rs, we have a default logger, which is a global variable that is initialized using
OnceCell. We needdropto work on the default logger to ensure that any buffered log records (inBufWriteror asynchronous tasks) are not lost, so we took the C++ principles and implemented it for the default logger -- using libc functionatexitto register a callback that will be called before the program terminates normally. In thisatexitcallback, flushing the default logger once, it can make sure that buffered log records are get flushed.For asynchronous sinks, additional considerations are needed:
Thread-local stuff is not supported in the
atexitcallback, in spdlog-rs,AsyncPoolSinkuses crossbeam channel to send tasks, which includes the use of thread-local. The solution is that if the program is tearing down (global variableIS_TEARING_DOWN: AtomicBoolis set to true in ouratexitcallback),AsyncPoolSinkwill flush directly on the current thread instead of sending the flush task to theThreadPoolvia crossbeam, thus the use of thread-local is avoided.In addition to flushing the log records, wait for all of the asynchronous tasks in the
ThreadPoolto complete to ensure that there are no pending tasks before exiting.All of these mechanisms only make sense for the global variable default logger, since local variables always get dropped before the
mainfunction exits, which performs a flush, and the last sink that owns theArc<ThreadPool>makes sure that all pending tasks have completed.UPDATE: More accurately, for asynchronous sinks, the order is: first wait for all tasks to complete, then stop the thread pool from receiving new tasks, and then perform the final flush.