Repository navigation
Handle lines longer than 1000 bytes: error "bad mail input format" #18
Description
Currently dma chokes on mail input which contains lines longer than 1000 bytes, because of the buffer being used.
When it does not find a newline at the end, it adds one, but then assumes/requires this to be the last line.
The error is "bad mail input format".
This might happen when using dma to handle web application errors during development / local setup, and the included debug information contains huge environment etc settings (e.g. "LS_COLORS").
I think that it would be saner to not require RFC2822 compliance, but accept the mail nonetheless.
The code in question: https://github.com/corecode/dma/blob/master/mail.c#L376
Activity
corecode commentedon Jan 11, 2014
Do you have a suggestion how to deal with longer lines? Something that
does not complicate the code?
On 01/11/2014 07:00 AM, Daniel Hahler wrote:
blueyed commentedon Jan 11, 2014
The easiest approach would probably be to increase the buffer length, from 1000 bytes to maybe 64kB (65536).
It would/could then still fail in the same way, but it's much more unlikely.
Or, just do not add a newline (and effectively aborting / cutting the input) and keep reading in 1000 byte chunks.
corecode commentedon Jan 12, 2014
The RFC is absolutely clear on this issue:
and
However, there are so many implementations that (in compliance with
the transport requirements of [RFC5321
http://tools.ietf.org/search/rfc5321]) do not accept messages
containing more than 1000 characters including the CR and LF per line,
it is important for implementations not to create such messages.
It is clear that we can not forward any message with more than 1000
characters per line. What do you suggest as alternative? Insert a
linebreak after 998 characters?
On 01/11/2014 11:15 PM, Daniel Hahler wrote:
blueyed commentedon Jan 12, 2014
Well, I think that dma is not creating the message, but only transports it.
The bug is really with the app that hands it over to dma.
Inserting newlines can be very problematic, if the message is meant to contain (encoded) data, and the result may fail to parse afterwards.
Imagine an automated message, with "key=value" pairs on separate lines, where it may result in "key=verylongvalue\r\nstillvalue".
However, breaking lines after 998 bytes is still far better than rejecting the mail altogether.
I can imagine having a configuration option (in dma.conf) for this: RFC2822_SPLITLINES.
If set, CRLF gets inserted after 998 bytes, otherwise the lines would get bypassed unchanged.
corecode commentedon Jan 12, 2014
It is not optional to forward messages with more than 998 characters per
line: it is forbidden. Do you know how other MTAs handle this? I only
see two options: break line, or reject (bounce).
On 01/12/2014 06:01 PM, Daniel Hahler wrote:
blueyed commentedon Jan 12, 2014
Then breaking the line is better than rejecting the mail.
Postfix has a line_length_limit option, with a default of 2048 (http://www.postfix.org/postconf.5.html):
It sounds like this will effectively pass through long lines?!
bcoca commentedon Mar 26, 2014
@blueyed, IIRC that limit is for internal processing across Postfix apps and filters, for actual smtp this is the setting:
http://www.postfix.org/postconf.5.html#smtp_line_length_limit
corecode commentedon Mar 26, 2014
Good catch. Any suggestions how we should handle this?
On 03/26/2014 08:24 PM, Brian Coca wrote:
bcoca commentedon Mar 26, 2014
I think rejection is OK, but maybe add more info to the message, 'bad mail input format' (I just ran into this) is a bit too succinct, knowing that its a line length error helps pinpoint the issue much faster. Also, quoting RFC helps convince programmers change their code vs demand that the MTA just accept their emails.
corecode commentedon Mar 26, 2014
alternatively we could force a linebreak when accepting the mail, like
postfix does when sending?
On 03/26/2014 08:47 PM, Brian Coca wrote:
blueyed commentedon Mar 27, 2014
The Postfix approach seems sensible.
To me everything is better than rejecting the mail.
I agree. There could be a warning instead of the current error.
Given the use case of dma (for personal or lightweight use in containers) I do not think it should be too strict (i.e. rejecting the mail).
bcoca commentedon Mar 27, 2014
I disagree, I use dma because it it simple and lightweight and doesn't try to do complex stuff like line splitting with character or armor encodings. If it accepts mail and it then gets rejected by another mailer, it creates more issues than it solves.
I am a fan of fail early and fail loudly, if you start accepting emails and push the problem downstream you never have an incentive to fix the actual issue, the program composing non compliant smtp messages.
Add more information to "bad mail input format" error
Add more information to "bad mail input format" error
emaste commentedon Mar 24, 2016
This is one issue that prevented the FreeBSD clusteradm team switching to DMA internally: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208261
28 remaining items
corecode commentedon Feb 3, 2022
I think we can keep the
feof()loop condition, but just do something about error return fromgetline.emaste commentedon Feb 3, 2022
I think it's fine to use getline()'s -1 return
I have a FreeBSD review in https://reviews.freebsd.org/D34159
ant5 commentedon Jun 21, 2023
Hello!
After switching to
dmafor local delivery and few monthes of things going well I've got an error"bad mail input format"with one email message.Does I understand right that there is a patch of cure and this patch is only in FreeBSD 13+ source tree?
How can I cure this?
Does I need to extract somehow this patch(es) from FreeBSD source and create custom port for
mail/dmathen add those patches to the custom port and then make custom package?Can someone guide me to eliminate future accidents?
P.S. My system is
FreeBSD 12-STABLE. Dma is mostly invoked by sieve scripts for coping/forwarding.corecode commentedon Jun 21, 2023
Are you submitting an incorrectly formatted message? The correct fix would be to only submit correctly formatted messages.
ant5 commentedon Jun 21, 2023
One user (who is in company management) sended email to other users. Email message seems very casual.
But then he got email back:
I'm sorry to have to inform you that your message could not be delivered to one or more recipients. ... <[X@Z](mailto:X@Z)>: host 127.0.6.1[127.0.6.1] said: 451 4.2.0 <[X@Z](mailto:X@Z)> Execution of Sieve filters was aborted due to temporary failure (in reply to end of DATA command) ...for those users who have configured copying/redirecting via sieve scripts.
The log say:
Jun 20 21:49:37 mail dma[e98a03][8169]: bad mail input format: from pop (uid 68) (envelope-from X@Z)corecode commentedon Jun 21, 2023
I would need to see the message to understand what happened. With the log messages it is not clear what happened. How are pop, sieve, dma, and the user related?
ant5 commentedon Jun 21, 2023
Technicaly speaking, user work on workstation with some email client. Email client sent message to postfix via SMTP. Postfix relay message to dovecot via LMTP. Dovecote execute sieve script. Sieve script invoke mailwrapper (sendmail). Mailwrapper execute dma.
I have this message extracted from bounce email but I don't know is it in virgin form and will it raise an error.
corecode commentedon Jun 21, 2023
That is a complicated setup. Why are you using dma and not postfix for dovecot for delivery? Without the original mail delivered to dma we don't know what is going on - it seems that it passed through several MTAs, so the assumption should be that the mail was formatted correctly. This pretty much rules out a 1000+ byte header line.
ant5 commentedon Jun 21, 2023
They all in separate jails. Dma in dovecot jail will relay to postfix in other jail (host).
ant5 commentedon Jun 21, 2023
I'll try to invoke dma and feed it problematic message...
But it's not so simple for me.
ant5 commentedon Jun 21, 2023
Yes. The message gives an error.
Zipped and attached:
< deleted confidentional data >
corecode commentedon Jun 21, 2023
Thank you. The problem is that one line is exactly 1000 bytes long including the
\r\n.haarp commentedon Jan 4, 2024
Wouldn't a config option setting the max buffer size/line length be the easiest solution here? Yes, it would break the spec, but it would be up to the admin to decide that. And for local-only delivery, there are no other servers further down the line that might choke on it.
Splitting is also a decent solution, but not always desireable. Some content, like code or logs shouldn't be split, to retain proper syntax.
Relegating this to the mail producer is also not always an option. They can be unpredictable. Maybe there was an error during a cronjob and it prints an unpredictably long line. The admin would get no mail, and very likely not realize.
corecode commentedon Jan 6, 2024
we definitely don't want to break the spec. if there is a line >1000 chars, we either have to split it or reject it outright. we can't pass on an invalid message.
kgcgs commentedon Sep 11, 2024
The only correct thing to do here is mimic the behavior of the fully featured MTAs. Sendmail has always split long lines in order to enforce compliance in messages that it creates but happily passes messages with long lines through SMTP without molesting them. Failure to address this in dma is just going to lead to messages being lost unexpectedly which is how most of us probably got here in the first place. Seems like an easy fix? (Even if it's a part of the RFC that's likely completely unnecessary to respect.)