Old Problems never die …


The last few days I’ve spent quite some while debugging and wondering and scratching my head with a weird problem with SMTP mails sent from a VA Smalltalk image, just to find out it is one of the oldest problems in computing history….

But it took me quite a while to find that out, because it looked so completely like something else.

So what was going on?

A Seaside application I am working on needs to send out mails using SMTP, and the simplest thing for now to get things going was to add some method to a class that returns an HTML-String as constant. This String is then handed over to SstLocalEndpoint<<#send:to: which will then handle all the SMTP stuff. I had to tweak Server Smalltalk a little to support ESMTP with User-ID and Password for authorization at the SMTP Server, but everything else worked like a charm.

Until one day I realized that mails received from my Seaside/Smalltalk server looked weird. I sent out nicely styled HTML mails, and Thunderbird showed funny letters at the bottom of the message. Something like this:

Something's wrong here Strange. This were headers that obviously were added by my web hoster. But why on earth were they added to the end of the message instead of to the headers section???

And why did I never realize this before?

It took a while for the coin to fall, but then it turned out it was a bog coin. I develop on windows and deploy to a Linux server. So I went back to my development machine / image and tried again: The headers were added to the perfect place. On Linux they were at the wrong place.

So back to the Debugger and see what’s going on.

A few hours and cups of coffee later I suddenly saw the light: I am fighting the famous “line-end-convention-differences-between-Windows-and-the-rest-of-the-world” problem.

Hugh?

Well, the method that produces the message text simply returns some text. I had edited the class on a Windows machine, so the line-ends consisted of <Cr><Lf>.

SstSmtpAssembler uses a PositionableStream instance to read the text and send it to the SMTP server line by line (you probably already hear the bells ringing). For this it uses the method nextLine.

PositionableStream is a clever class, because it asks the current Platform for its LineDelimiter. In Unix, the Line Delimiter is <Lf>. And here is where the trouble begins: If the Message text contains a line with <Cr><Lf>, it will be cut off correctly on Windows, because there the line delimiter is (you name it) <Cr><Lf>. On Linux. however, the line will end with a lonesome <Cr>.

When Sst sends each line to the SMTP server, it adds a <Cr><Lf> to each line. In my case this leads lines ending in <Cr><Cr><Lf>, which doesn’t cause much trouble in HTML messages.

It does, however, cause trouble in the communication with the SMTP server at the end of the message, because the SMTP standard says the end of a message has to be indicated with <Cr><Lf>.<Cr><Lf> (a single line containing only a dot), AND NOTHING ELSE. It doesn’t specify the as exactly how the headers have to be separated from the mail’s body, but it seems additional <Cr>’s are not valid there.

My SMTP server tries to be clever and gets along with a lot of additional <Cr>’s but not completely, because it seems it cannot tell where exactly the headers end and the body starts.

When I changed PositionableStream>>#nextLine to also cut off a trailing <Cr>, all worked like a charm again. But I have a sore feeling about this “fix”, because nextLine is a method that gets used in so many contexts…

So what now?

To be honest, I don’t know. I know the problem, and I know a possible solution. But I am not sure if changing PositionableStream is a good idea. I could of course also give the PositionableStream the Rfc822LineDelimiter, but this would break the code for Strings edited on Linux and maybe other Unix machines, because there the line end convention is Lf only. It seems cutting off a remaining trailing Cr after removing the local LineDelimiter is not wrong at all, because nextLine is expected to return a line without trailing line-ends, and a trailing Cr clearly sounds like it is one. Except for maybe on Linux… Hmm- puzzled…

Maybe Instantiations support can help me get it sorted out…

3 thoughts on “Old Problems never die …

  1. Hi Joachim,
    To solve that line-end problems once forever, just switch to binary streams instead. And add cr-lf manually for any internet protocol you’ll send and receive over.

    Janko

    1. Hi Janko,

      I solved it now with an extra filter that scans the input String for non-CrLf lineDelimiters and replaces them. I am not sure how Binary Streams could help in my situation. The source String comes out of my source code repository (envy)…

Comments are closed.