Repository navigation
extract_msg.Message.export() fails to export updated attributes #388
Description
Hi,
Bug Metadata
- Version of extract_msg: 0.45.0
- Your python version: Python 3.9.12
- How did you launch extract_msg?
- I used the extract_msg package
Describe the bug
I am using extract_msg to open an Outlook email template, replace some values in the body of the email and export the updated email as a .msg file. I managed to update the body of the Message object and save it as a txt file using extract_msg.Message.save(). However, extract_msg.Message.export() always exports the original body instead of the updated one.
See the some sample code below to reproduce the error.
import extract_msg
REPLACEMENT_STRING="test"
f = "template.msg"
msg = extract_msg.Message(f)
msg.body = msg.body.replace("[STRING_TO_REPLACE]",REPLACEMENT_STRING)
print('Body: {}'.format(msg.body)) # [STRING_TO_REPLACE] has been replaced
msg.htmlBody = msg.htmlBody.replace(b"[STRING_TO_REPLACE]",bytes(REPLACEMENT_STRING, encoding='utf-8'))
print('HTML Body: {}'.format(msg.htmlBody)) # [STRING_TO_REPLACE] has been replaced
msg_name = "updated_" + f
msg.save() # [STRING_TO_REPLACE] has been replaced in .txt file
msg.export(msg_name) # [STRING_TO_REPLACE] is still in updated_template.msg
msg.close()
Github doesn't accept .msg files as attachments, but the "template.msg" in question is empty apart from the two following fields:
subject: Test subject
body: [STRING_TO_REPLACE]
Traceback
No error is printed out but the body of the exported "updated_template.msg" file is the same as "template.msg".
Am I doing it wrong or is this indeed not the expected behaviour?
Activity
TheElementalOfDestruction commentedon Oct 11, 2023
This is the intended behavior, as the
MSGFileclass and all derivatives are read only for all intents and purposes. Writing to them is not supported, and in fact writing entirely new or modified MSG files is something that the module does not yet support in any way. While the module does not support it, it can be used to do so, provided you have a more advanced understanding of MSG files.The property you are modifying is a cached property, which does not have a setter method. This is important because of how the export function works, since it takes the underlying reader and uses it to create a copy of the streams that would make up the new versions of the file. The export method will not read any of the properties on an MSG file, and would require much more extensive code to even be able to do so.
Instead, the way to create new, or edit existing, msg files is to create a new instance of
extract_msg.ole_writer.OleWriter, use the appropriate function to copy all of the streams to it, then modify those streams directly. Because of the low level nature of that writer, it is not designed to understand any of the properties of an MSG file, so you would have to lookup the mapping for any streams you wish to change. Looking at the source code for many properties can give you a good idea of what stream would be required to perform the changes you desire.The reason the writer is so low level was for simplicity and versatility, allowing it to write other files that use the same underlying format (the Compind Binary File Format, if I remember the name right, though I usually just call is "OLE File Format") of an MSG file, but that may not be MSG files themselves.
Future versions are intended to better support the modification of MSG files, but a lot of significant work is required. If you look to the work being done for version 0.46.0, you'll notice the change in a number of the classes and the change in writing for the newly added classes, namely that I am working to make as many as possible have the ability to be modified and exported back to bytes, something that would be required for making those high level structures usable for a writer. Changes like this will be required for the
MSGWriterclass, a class planned for the future which will be a higher level writer, which would help facilitate what you desire without having to know the mapping between what we know properties as and what the format knows them as.Additionally, the reason you are able to modify the properties was due to a change made after consideration of the code base, where it was decided that using built in features that may allow the user to do a limited under of undesired things (like modifying properties they weren't intended to) was worth having a vastly simplified code base. In fact, much of the module actually can respond, in some ways, to these changes, and this was considered and understood by me. Export, for the reasons stated, is an exception.
TL;DR:
MSGFileand it's subclasses are readers, and do not expect to be modified like that, even if it is possible in places.MSGFile.exportknows this and doesn't even look at those readonly properties.If you have any ideas for where the documentation could be better adjusted to reflect this discussion, please let me know.
jill-leflour-db commentedon Oct 12, 2023
Thank you for the detailed response, it makes a lot of sense. I mistakenly assumed that the use of
@functools.cached_propertyover@propertymeant that the attributes were intended to be modifiable. Ultimately, my use case isn't critical enough for me to figure out how to perform those changes at a lower level.I'm not sure where the best place to point this out would be. Maybe in
exportdirectly? In case others try to use it for the same purpose as me.TheElementalOfDestruction commentedon Oct 12, 2023
Yes, that does sound like it might be a good place to put a note regarding this. I think I'll also put a note in the documentation for MSGFile to note that the class is specifically for reading and doesn't have writing capabilities to hopefully avoid confusion.
I do think that it's unlikely to be worth the effort to manually attempt all of the low level code that would be required for replacement of the bodies as well, and there are a number of reasons for that. While certain properties may be easy to do, the format is a lot more complicated than people might expect.
The body specifically can end up being rather complicated to acknowledge modification for a number of reasons. The first is that that cached property does not always draw directly from the body stream, and may use the RTF body and, in the future, the HTML body to generate a plain text representation. Anything trying to export it the file may suddenly find a new body stream where one wasn't there before, even without user modification.
The other main reason that the body stream is complex is due to the type of stream it is, namely a string stream. String streams can be stored in one of two ways: Unicode (in which case the suffix to the stream name is "001F") or any other encoding (in which the suffix to the stream name is "001E"). These two encoding methods are mutually exclusive and MUST match the details specified in the properties.
MSGFile.areStringsUnicodecan be used to figure out which suffix to use for the file name, andMSGFile.stringEncodingcan be used (if I got the name right) to determine how to convert the plain text body into bytes. Fortunately,the body stream, excluding the suffix, is simple and is simple__substg1.0_1000, so checking those bits of data to find the suffix could allow rather simple code for that property.As for the HTML body, that also has the same initial problem as the plain text body, since it is often generated from the RTF body or the plain text body if it can't be found. If that is the case, you may notice further oddities when editing the bodies, since the existence of the RTF body may cause that to be used over your preferred bodies, making your changes invisible. The HTML body stream is a simple name that only has one possible suffix, and the property is already bytes, so writing that one is easy. The RTF body, on the other hand, must be compressed with a certain format, but some of my tests have shown the compression module we use to be rather hit or miss when it comes to Outlook accepting it. I do not know why Outlook will sometimes not accept it, and further testing is something on the todo list for MSGWriter.
TheElementalOfDestruction commentedon Oct 25, 2023
Sorry for taking a bit. Here is what I am considering for the new documentation for the export function. Let me know what you think and if you have any suggestions about how it might be better worded.
jill-leflour-db commentedon Oct 26, 2023
No worries. I think what you're considering adding to the docstring is clear.
TheElementalOfDestruction commentedon Nov 8, 2023
Documentation in version 0.46.0 now reflects the behavior that export uses.