Skip to content

Stack overflow on FreeBSD #100

Open
Open
Stack overflow on FreeBSD#100

Description

@oscarcarlsson
oscarcarlsson
opened on Jun 30, 2021

Hi,

I'm using dma on a FreeBSD 13 machine with a smarthost (smtp.migadu.com), but dma crashes with a stack overflow after consuming 100% cpu for a few minutes. I've seen this on two machines, on older releases of FreeBSD as well.

I've tried both with the dma in the base system, and installing the latest dma from pkg. I'm not too familiar with how this is really used, but I can reproduce the issue like this (or by just trying to use mail):

echo "test!" |/usr/local/libexec/dma -D -r 'test@prutt.party' 'oscar@prutt.party'

If I try to press ctrl-t in FreeBSD when running the above, I get some minor useful data:

# Removed a bunch of these: load: 1.82  cmd: dma 91917 [running] 230.39r 58.55u 111.70s 100% 5884k
mac_socket_check_receive+0x7a soo_read+0x19 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
amd64_syscall+0x10c fast_syscall_common+0xf8 
soreceive_generic+0xe7 soreceive+0x44 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
soreceive_generic+0xe7 soreceive+0x44 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
soo_read+0x19 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
soreceive_generic+0xe7 soreceive+0x44 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
soreceive_generic+0xe7 soreceive+0x44 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
sys_read+0x99 amd64_syscall+0x10c fast_syscall_common+0xf8 
soreceive_generic+0xe7 soreceive+0x44 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8

Any ideas?

This is a typical case, from /var/log/maillog:

Jun 30 13:13:50 hilda dma[181e2][60585]: new mail from user=root uid=26 envelope_from=<root@hilda>
Jun 30 13:13:50 hilda dma[181e2][60585]: mail to=<oscar@prutt.party> queued as 181e2.800e48050
Jun 30 13:13:50 hilda dma[181e2.800e48050][61054]: <oscar@prutt.party> trying delivery
Jun 30 13:13:50 hilda dma[181e2.800e48050][61054]: using smarthost (smtp.migadu.com:465)
Jun 30 13:13:51 hilda dma[181e2.800e48050][61054]: trying remote delivery to smtp.migadu.com [2001:41d0:2:863f::] pref 0
Jun 30 13:13:51 hilda dma[181e2.800e48050][61054]: connect to smtp.migadu.com [2001:41d0:2:863f::] failed: No route to host
Jun 30 13:13:51 hilda dma[181e2.800e48050][61054]: trying remote delivery to smtp.migadu.com [2001:41d0:2:aacc::] pref 0
Jun 30 13:13:51 hilda dma[181e2.800e48050][61054]: connect to smtp.migadu.com [2001:41d0:2:aacc::] failed: No route to host
Jun 30 13:13:51 hilda dma[181e2.800e48050][61054]: trying remote delivery to smtp.migadu.com [2001:41d0:2:267::] pref 0
Jun 30 13:13:51 hilda dma[181e2.800e48050][61054]: connect to smtp.migadu.com [2001:41d0:2:267::] failed: No route to host
Jun 30 13:13:51 hilda dma[181e2.800e48050][61054]: trying remote delivery to smtp.migadu.com [94.23.1.103] pref 0
# snip
Jun 30 13:18:51 hilda dma[181e2.800e48050][61054]: stack overflow detected; terminated

Here's my /etc/dma/dma.conf:

SMARTHOST smtp.migadu.com
PORT 465
AUTHPATH /etc/dma/auth.conf
SECURETRANSFER
STARTTLS

My /etc/dma/auth.conf looks valid, user@domain.tld|smtp.migadu.com:password.

My /usr/local/etc/mail/mailer.conf looks like this, currently configured for the dma from pkg:

sendmail      /usr/local/libexec/dma
mailq         /usr/local/libexec/dma
newaliases    /usr/local/libexec/dma
rmail         /usr/local/libexec/dma

Activity

  1. oscarcarlsson commented on Jun 30, 2021

    @oscarcarlsson
    oscarcarlsson
    on Jun 30, 2021
    Author

    I've tried removing STARTTLS from my configuration (as recommended by others) but it doesn't fix the issue. It's still crashing from a stack overflow.

    Could it be something as silly as permission denied? Nope.

    # ls -la
total 14
drwxr-xr-x   2 root  wheel    4 Jun 30 15:02 .
drwxr-xr-x  27 root  wheel  111 Jun 24 23:16 ..
-rw-r-----   1 root  wheel   85 Jun 24 22:48 auth.conf
-r--r-----   1 root  wheel   96 Jun 30 15:02 dma.conf

  2. corecode commented on Jun 30, 2021

    @corecode
    corecode
    on Jun 30, 2021 via email
    Owner

  3. oscarcarlsson commented on Jun 30, 2021

    @oscarcarlsson
    oscarcarlsson
    on Jun 30, 2021
    Author

    I got it working after some help - I changed port (to 587/tcp, from the provider recommended port 465/tcp) and added masquerade. But I guess that the stack overflow should still be looked into.

    SMARTHOST smtp.migadu.com
PORT 587
AUTHPATH /etc/dma/auth.conf
MASQUERADE user@prutt.party
SECURETRANSFER
STARTTLS

    This is a full crash from my /var/log/maillog:

    Jun 30 17:20:32 hilda dma[180ec][48243]: new mail from user=root uid=26 envelope_from=<root@hilda>
Jun 30 17:20:32 hilda dma[180ec][48243]: mail to=<oscar@prutt.party> queued as 180ec.800e48050
Jun 30 17:20:32 hilda dma[180ec.800e48050][48399]: <oscar@prutt.party> trying delivery
Jun 30 17:20:32 hilda dma[180ec.800e48050][48399]: using smarthost (smtp.migadu.com:465)
Jun 30 17:20:32 hilda dma[180ec.800e48050][48399]: trying remote delivery to smtp.migadu.com [2001:41d0:2:267::] pref 0
Jun 30 17:20:32 hilda dma[180ec.800e48050][48399]: connect to smtp.migadu.com [2001:41d0:2:267::] failed: No route to host
Jun 30 17:20:32 hilda dma[180ec.800e48050][48399]: trying remote delivery to smtp.migadu.com [2001:41d0:2:aacc::] pref 0
Jun 30 17:20:32 hilda dma[180ec.800e48050][48399]: connect to smtp.migadu.com [2001:41d0:2:aacc::] failed: No route to host
Jun 30 17:20:32 hilda dma[180ec.800e48050][48399]: trying remote delivery to smtp.migadu.com [2001:41d0:2:863f::] pref 0
Jun 30 17:20:32 hilda dma[180ec.800e48050][48399]: connect to smtp.migadu.com [2001:41d0:2:863f::] failed: No route to host
Jun 30 17:20:32 hilda dma[180ec.800e48050][48399]: trying remote delivery to smtp.migadu.com [188.165.223.204] pref 0
Jun 30 17:25:32 hilda dma[180ec.800e48050][48399]: stack overflow detected; terminated

    That's it.

  4. tuffnatty commented on Oct 2, 2021

    @tuffnatty
    tuffnatty
    on Oct 2, 2021

    Also getting stack overflow in port 465; seemingly fixed by switching the port to 587.

  5. corecode commented on Oct 2, 2021

    @corecode
    corecode
    on Oct 2, 2021
    Owner

    you need to remove STARTTLS if you use port 465. 587 + STARTTLS should be preferred.

    The stack overflow is definitely a bug though.

  6. corecode commented on Oct 3, 2021

    @corecode
    corecode
    on Oct 3, 2021
    Owner

    could you attach with gdb and show a backtrace? It's not clear what leads to the stack overflow.

  7. emaste commented on Feb 3, 2022

    @emaste
    emaste
    on Feb 3, 2022
    Collaborator

    You might need to set sysctl kern.sugid_coredump=1 to get a core file that you can open in gdb or lldb

  8. egwynn commented on Jun 19, 2026

    @egwynn
    egwynn
    on Jun 19, 2026

    I'm running into this too. @oscarcarlsson's minimal reproducible example triggers it for my config. GDB does give good insights. I wasn't able to start it up with gdb, but was able to attach post-init. Here's the stack:

    (gdb) c
Continuing.

Program received signal SIGABRT, Aborted.
Sent by kill() from pid 8819 and user 1001.
kill () at kill.S:4
warning: 4      kill.S: No such file or directory
(gdb) bt
#0  kill () at kill.S:4
#1  0x0000239ad4b99a30 in __fail (msg=0x239ad4aa0524 "stack overflow detected; terminated")
    at /usr/src/lib/libc/secure/stack_protector.c:119
#2  0x0000239ad4b999a0 in __stack_chk_fail () at /usr/src/lib/libc/secure/stack_protector.c:126
#3  0x00002392b2f0ac46 in do_timeout (timeout=<optimized out>, dojmp=<optimized out>) at /usr/src/contrib/dma/util.c:296
#4  0x00002392b2f08666 in read_remote (fd=fd@entry=7, extbufsize=extbufsize@entry=0, extbuf=extbuf@entry=0x0)
    at /usr/src/contrib/dma/net.c:171
#5  0x00002392b2f08e1a in deliver_to_host (it=0x28a79c248000, host=0x28a79c25e000) at /usr/src/contrib/dma/net.c:516
#6  deliver_remote (it=it@entry=0x28a79c248000) at /usr/src/contrib/dma/net.c:671
#7  0x00002392b2f05c46 in deliver (it=0x28a79c248000) at /usr/src/contrib/dma/dma.c:328
#8  run_queue (queue=queue@entry=0x239ad30b3730) at /usr/src/contrib/dma/dma.c:384
#9  0x00002392b2f06499 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/contrib/dma/dma.c:631


(gdb) f 3
#3  0x00002392b2f0ac46 in do_timeout (timeout=<optimized out>, dojmp=<optimized out>) at /usr/src/contrib/dma/util.c:296
296             return (ret);
(gdb) l
291                     if (sigaction(SIGALRM, &act, NULL) != 0)
292                             syslog(LOG_WARNING, "can not remove signal handler: %m");
293                     sigbuf_valid = 0;
294             }
295
296             return (ret);
297     }
298
299     int
300     open_locked(const char *fname, int flags, ...)

    This corresponds with the return statement from do_timeout (although I'm running a slightly older version, I don't think this code has changed). Here we see that it's the stack protector which is intervening to kill the process during function return. The sequence appears to be:

    1. Try to connect
    2. Set the timeout
    3. Timeout triggers (probably a real failure to connect to my mail host)
    4. Triggering the timeout fails the stack protector's check
    5. Process dies

    I don't generally write C code, but my reading of the related source tells me that we could be seeing conflicts between the sigsetjmp/siglongjmp usage and the stack protector. I suspect that do_timeout might be clobbering the stack canary.

  9. egwynn commented on Jun 19, 2026

    @egwynn
    egwynn
    on Jun 19, 2026

    Confirmed. I extracted do_timeout and its supporting code and wrote a tiny function that sets the timeout and then sleeps. When compiled with the stack protector, I get the same error:

    // timeout_test.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <setjmp.h>
#include <syslog.h>
#include <sys/wait.h>

static sigjmp_buf sigbuf;
static int sigbuf_valid;

static void
sigalrm_handler(int signo)
{
    printf("Handler triggered!\n");
    (void)signo;
    if (sigbuf_valid) {
        siglongjmp(sigbuf, 1);
    }
}


int
do_timeout(int timeout, int dojmp)
{
    struct sigaction act;
    int ret = 0;

    sigemptyset(&act.sa_mask);
    act.sa_flags = 0;

    if (timeout) {
        act.sa_handler = sigalrm_handler;
        if (sigaction(SIGALRM, &act, NULL) != 0)
            syslog(LOG_WARNING, "can not set signal handler: %m");
        if (dojmp) {
            ret = sigsetjmp(sigbuf, 1);
            if (ret)
                goto disable;
            /* else just programmed */
            sigbuf_valid = 1;
        }

        alarm(timeout);
    } else {
disable:
        alarm(0);

        act.sa_handler = SIG_IGN;
        if (sigaction(SIGALRM, &act, NULL) != 0)
            syslog(LOG_WARNING, "can not remove signal handler: %m");
        sigbuf_valid = 0;
    }

    return (ret);
}

int
main() {
    // Clear any previous state
    sigbuf_valid = 0;

    printf("Starting Stack Protector Test...\n");

    // Start with a small alarm to ensure signal fires
    printf("Calling do_timeout with 1 second timeout...\n");
    int res = do_timeout(1, 1); 

    sleep(5);

    printf("Program ended successfully.\n");

    // Optional: Wait for any child processes to clean up
    // exit(0); 
}

    Here it is with / without stack protection.

    $ cc -o timeout_test timeout_test.c
$ ./timeout_test
Starting Stack Protector Test...
Calling do_timeout with 1 second timeout...
Handler triggered!
Program ended successfully.
$ echo $?
0


$ cc -fstack-protector-strong -o timeout_test_protected timeout_test.c
$ ./timeout_test_protected
Starting Stack Protector Test...
Calling do_timeout with 1 second timeout...
Handler triggered!
Abort trap
$ echo $?
134

  10. corecode commented on Jun 19, 2026

    @corecode
    corecode
    on Jun 19, 2026 via email
    Owner

  11. caseyandgina commented on Oct 7, 2026

    @caseyandgina
    caseyandgina
    on Oct 7, 2026

    Another data point, with a different trigger from the port-465 misconfiguration earlier in this thread: this still occurs with the dma in FreeBSD 15.1-RELEASE-p4 base (contrib/dma there and on main still have the sigsetjmp inside do_timeout()).

    Our trigger was a correctly configured smarthost (port 25, STARTTLS) whose per-IP connection rate limit stalls connections over the limit instead of refusing them. A host's daily periodic mail plus a dma -q retry opened more connections in one second than the limit allowed. Each stalled delivery sat in read_remote() until CON_TIMEOUT, then aborted with stack overflow detected; terminated exactly 300 s after trying remote delivery.

    The practical consequence is worse than a crash: the timeout path never runs, so the message is never deferred or bounced, and it stays queued. The next queue run resends the whole queue at once, so the same messages hit the same limit and abort again in lockstep every day. We had messages stuck for three days with nothing logged anywhere except the sending host's maillog.

    We worked around it on the server side by not throttling our own hosts. Any peer that stalls a read for five minutes will still trigger it. Thanks for confirming the cause above. Keeping the sigsetjmp in a frame that outlives the read (or using poll()/SO_RCVTIMEO with no longjmp at all) would fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions