To shreds, you say?

jrnl: Command line braindump

A year or so ago, I thought it would be great if there was a tool that would allow me to easily record notes and thoughts via command line. At the time, nothing really impressive existed (and I was too lazy to create something myself). Luckily, wandering the the net this morning, I came across jrnl¹. Coupled with intelligent date parsing, this is the ideal tool for me to quickly record anything that needs noting with minimal effort. Export options, including markdown² and JSON make it doubly appealing.

The trouble with SOAPHandlers and MTOM

Introduction

At a high level, MTOM¹ is a SOAP optimisation feature for binary data. As SOAP uses XML as the standard message format, this presents an issue when binary data must be included in a SOAP message. To work around this limitation, binary content can be in-lined in a SOAP message as a Base64-encoded string (specifically, the base64Binary primitive data type²). Encoding to Base64 text does not come without a price; a 33% size increase of the binary content³. Enter MTOM to the rescue (sort of).

MTOM allows you to work around the Base64 performance hit by replacing the in-line Base64 text with a reference to a separate MIME part containing the binary data. For example:

SOAP Message Content without MTOM

Note: the binary content in line - Li4uIGJpbmFyeSBjb250ZW50IC4uLg==.

<?xml version="1.0" ?>
<S:Envelope xmlns:S="http://www.w3.org/2003/05/soap-envelope"install>
    <S:Header>
        .
        .
        .
    </S:Header>
    <S:Body xml:id="id">
        <Request xmlns="http://somenamespace/for/Request">
            Li4uIGJpbmFyeSBjb250ZW50IC4uLg==
        </Request>
    </S:Body>
</S:Envelope>

SOAP Message content with MTOM

Note: the binary content now replaced with an XOP include reference to the Content-Id of the additional MIME part — ContentId: <64449861-d66c-4540-86bc-4486bfa796cc>.

--uuid:952d2ab0-043e-4233-bc73-40e3f2c01694
Content-Id: <rootpart*952d2ab0-043e-4233-bc73-40e3f2c01694>
Content-Type: application/xop+xml;charset=utf-8;type="application/soap+xml"
Content-Transfer-Encoding: binary

<?xml version="1.0" ?>
<S:Envelope xmlns:S="http://www.w3.org/2003/05/soap-envelope"install>
    <S:Header>
        .
        .
        .
    </S:Header>
    <S:Body xml:id="id">
        <Request xmlns="http://somenamespace/for/Request">
            <xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include"
             href="cid:64449861-d66c-4540-86bc-4486bfa796cc"></xop:Include>
        </Request>
    </S:Body>
</S:Envelope>

--uuid:952d2ab0-043e-4233-bc73-40e3f2c01694
Content-Id: <64449861-d66c-4540-86bc-4486bfa796cc>
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

... binary content ...
--uuid:952d2ab0-043e-4233-bc73-40e3f2c01694

Problem

So this brings me to back to the title of this post and what problems arise when you start to alter your SOAP message. While this post is specifically targeted at JAX-WS⁴ and Java, I have seen similar issues on the .Net side of the fence when examining or altering the SOAP message and its MTOM multi-part structure. In my experience, SOAPHandlers⁵ are typically used to access the state of your request or response before it heads out into the ether. At times, this might be to log the content or to alter it directly. Irrespective of your objective (and I’ve tried both independently), adding a SOAPHandler to your port proxy’s handler chain⁶ breaks MTOM in that your outgoing SOAP request/response reverts to the following:

SOAP Message Content with MTOM and a SOAPHandler

Note: The Base64 string is back!

--uuid:952d2ab0-043e-4233-bc73-40e3f2c01694
Content-Id: <rootpart*952d2ab0-043e-4233-bc73-40e3f2c01694>
Content-Type: application/xop+xml;charset=utf-8;type="application/soap+xml"
Content-Transfer-Encoding: binary

<?xml version="1.0" ?>
<S:Envelope xmlns:S="http://www.w3.org/2003/05/soap-envelope"install>
    <S:Header>
        .
        .
        .
    </S:Header>
    <S:Body xml:id="id">
        <Request xmlns="http://somenamespace/for/Request">
            Li4uIGJpbmFyeSBjb250ZW50IC4uLg==
        </Request>
    </S:Body>
</S:Envelope>
--uuid:952d2ab0-043e-4233-bc73-40e3f2c01694

So you’ve effectively got MTOM working (it’s still got an MTOM MIME type encapsulating the SOAP envelope), but with none of the bloat-reducing benefits.

Cause

Of all the difficulties I’ve had, it was determining the root cause of this issue. Unfortunately, I still don’t have a solid answer. However, this old JIRA issue is the best reference for an explanation I could find:

When I use a logging handler that implements javax.xml.ws.handler.soap.SOAPHandler, and use MTOM the message that is returned by the service is still MTOM but the file attachment is no longer returned as an xop reference; instead it is returned as inline base64 text. If I omit the handler by commenting out the @HandlerChain annotation in AddNumbersImpl.java and rebuild the service, the file attachment is returned as an xop reference.

To recreate the issue deploy the attached war file in Tomcat 6.0 and then run the csharp client from the csharp-client directory by invoking the run.bat file without arguments. Make sure you also capture the input and output messages on the Tomcat console by enabling the following JVM option:

-Dcom.sun.xml.ws.transport.http.HttpAdapter.dump=true

When @HandlerChain is commented out you will see the attachment returned as an xop reference. When it is used, you will see the attachment returned as inline base64 text.

Joe Roberts - JAX-WS JIRA Ticket WSIT - 1320⁷

The key information to note here is that the resolution of the ticket is Will Not Fix, so to my knowledge, this remains unfixed⁸.

Solution

So what to do? I found there were a number of suggestions out there including use of tubes⁹ and codecs¹⁰. These options proved difficult to implement; more a low-level hack than a simpler solution. Fortunately, my attention was drawn in more detail to the SOAPMessageContext¹¹. In my frantic efforts to solve this problem, I had overlooked the basic functionality at the SOAPMessage¹² level that provides the ability to add additional AttachmentParts¹³ to the SOAP message. To cut a long story short and get to the workaround or solution, this allowed me to do the following:

1. Extract the Base64-encoded text from the SOAP body.
2. Create a new AttachmentPart to hold the extracted content in byte form.
3. Replace the original Base64-encoded text from the SOAP body with a reference element that points at the part in Step 2.

While this is at a relatively high level, that’s essentially all that is required. The following snippets give you a rough idea of the code required to carry out these steps:

/**
 * Note: no null checking or anything nice like that done as I only want to illustrate the
 * process.
 */

SOAPEnvelope envelope = context.getMessage.getMessage().getSOAPPart().getEnvelope();
SOAPBody body = envelope.getBody();

// Create a new reference for use with the attachment part.
UUID ref = UUID.randomUUID();

// You then get the element content using something like the following.
// This will be dependent on the type of body you're dealing with.
NodeList nodeList = body.getElementsByTagNameNS("namespace", "localname");
Element element = element = (Element) nodeList.item(0);
String elementContent = element.getFirstChild().getNodeValue();    
InputStream is = IOUtils.toInputStream(elementContent);

// Create a new part with a reference ID.
AttachmentPart attachment = context.getMessage().createAttachmentPart();
attachment.setBase64Content(is, "content/type");
attachment.setContentId(ref.toString());
context.getMessage().addAttachmentPart(attachment);

// Clear the old content.
SOAPBodyElement bodyElement = (SOAPBodyElement) element;
bodyElement.removeContents();

// Add the new include element, using the reference ID.
bodyElement
    .addChildelement("Include", "xop", "http://www.w3.org/2004/08/xop/include")
    .setAttribute("href", "cid:" + ref.toString());

This all should take place within a new SOAPHandler that you add to your handler chain. Depending on your use case, you can do this on request and/or response.

Conclusion

At best, this solution is a workaround but it will get you past a fairly significant hurdle, should you be constrained to using MTOM in your service calls. As always, if anyone has any comments or recommendations on how better to solve this problem, please leave them in the comments section.

Instamobi: Ruby web automation

I’ve been using Instapaper¹ for a while now. While it’s an excellent service, more often than not, I’ve needed a way of getting through some of the more lengthy articles on my Kindle². Fortunately, there’s a feature that allows you to download a Kindle/mobi format file. To give you and idea of what’s produced, see the image below:

I’ve been meaning to learn more Ruby³ so I figured this would be a good chance to try it out. Using the Mechanize⁴ gem, I’m able to:

1. Login to Instapaper
2. Check if there is are unread items
3. Download a compiled mobi format book
4. Save mobi book to Dropbox
5. Archive the unread items

Having done the above, it’s then relatively easy to access the books via the web browser on the Kindle. The script isn’t particularly robust or useful outside of this use case but feel free to take a look at it to give you an idea on:

• logging in to pages programmatically
• page scrapping
• Mechanize

If you are interested, feel free to check out the code and fork me on Github⁵.

How to use QuickTime and Automator for screencasts

I recently found the need to record a screen cast for a demo of an application I was working on. If you’re using a Mac, there is an extremely simple option available to you using the in-built QuickTime¹ application.

1. To get started, you need to launch Automator:
   

2. You then need to select the option to create a new Workflow:
   

3. Selecting Actions, do a search for Screen Capture or something similar. You should see the following appear:
   

4. Drag that New Screen Capture action into the right-hand workflow pane and you’re almost done:
   

5. If you click the Run icon, you should be presented with the following after QuickTime has launched:
   

Previously, I needed to rely on third-party apps that cost money so it’s nice to have this feature so tightly integrated with the OS. Having created and saved the workflow, the only thing left to do is make it easy to launch. I’m a big fan of Alfred², so I can use the features provided by the Alfred Powerpack to associate workflows with the launcher:

It shows up like this when you attempt to launch the new workflow from Alfred:

Alternatively, you can create a keyboard shortcut using these³ instructions.

How to add an OSX Mountain Lion Notification Centre shortcut

1. Open System Preferences.
2. From the Hardware section, pick Keyboard.
3. From the leftmost panel, pick the Mission Control icon.
4. On the rightmost panel, you should now see a Show Notification Center checkbox. Ensure it’s checked and then add a shortcut. I picked control + alt/option + command + N. That one doesn’t seem to conflict with anything obvious so far.