A Coder's Log

This Greek Tragedy

2011-05-18T15:47:00.000-07:00

Grigoris was a man who lived a good life, made a solid name for himself, and was well-respected by his community. But now, beset by the consequences of some unfortunate, perhaps even foolish, spending habits, he found himself in a great deal of debt in his middle years.

He might have thought to change his ways and begin the ascent from debt, but as we often find, old habits are hard to break. Fortunately for Grigoris, he came from a very large family. Counting himself, there were twenty-seven siblings, amazingly, all boys and just one girl, Francesca. Grigoris appealed to his family, but they were reluctant to help, since many of them were struggling to pay their debts as well.

After a long time of appealing to his siblings and offering promises that he would change his ways, Grigoris’ brother Gregor reluctantly agreed to give Grigoris enough money to settle his debts and make a new start. Gregor hoped that someday he would be repaid, but didn’t expect it.

To no one’s surprise, Grigoris accepted the money but did not change his ways. Within a couple years, he was back in debt and once again appealing to his family for help.

Although Grigoris spent much of his money on foolishness and temporary pleasures, he managed somehow to retain a large store of family heirlooms that had great value. The family lived in constant concern that he would sell these so some unsavory neighbor, but even in the depths of his flurry of spending, Grigoris held these back.

Once again, he appealed to his family. They were no idiots, and chose not to repeat the mistake of loaning him the money. He sustained his appeal, even while the debts continued to grow. But none of his family would part with their money.

One cold spring morning a thin beam of sunlight drilled through the curtains of the room where Grigoris slept, and settled directly on his left eye, awaking him with a start, a dream still fresh in his mind. In this dream was the spark, the answer that he had desired.

He immediately came to his senses and decided what he must do. He must choose one of the heirlooms and sell it to pay off his debts. “This is what all sensible people do,” he said aloud as the light of the morning enveloped him. “Selling some precious item from my collection will be a constant reminder that I must not fall again into debt, that I must change my ways and never again shame my family in this fashion.”

And so he set out to choose one precious item, and there were many to consider. Finally, he settled on one, thinking, it must be the most precious of all. Many generations back, he knew not exactly how many, a forefather had constructed a lavish gazebo on a small piece of land on a hill from the finest marble drawn from quarries at great cost. This gazebo was a delight to the family, and many would often return home just for the purpose of seeing it once more, and spending time there. Even visitors from various places would pass by just to behold its beauty.

He must sell this.

Though wracked and distressed, he set out with determination to find a buyer. He quickly found one, a foreign man who wanted to make a tourist destination of it, a veritable circus. Should Grigoris sell to this man? Would that not compound his grief? He thought that surely none of his twenty-six siblings would ever speak to him again. Perhaps they would even disown him, sending him out from the family in disgrace.

In the end, he made the only sensible decision - he would approach his family, and ask them to buy it. To his relief, Gregor and Francesca were willing to pool their resources and purchase the prized gazebo from Grigoris. Though they were hard-pressed to afford it, they recouped their cost through reasonable and respectful marketing as a modest tourist destination.

Grigoris reformed his old foolish habits, and to his dying day, never again found himself in debt. After a long and satisfying life, he passed out of this world having won the respect of his family and all who knew him.

Read this at PenWag

Using JUnit To Generate Mockito Custom ArgumentMatchers Descriptions

2011-04-25T16:40:00.000-07:00

I'm a huge fan of Mockito. It has made our unit testing lives a veritable cakewalk for years now. There is something I'd like to be different, and that's what it provides for failure descriptions in custom argument matchers. It seems like a lot of work (well, relatively speaking) to provide a description then return false, assuming a test fails. It works out to about four lines of code for each value to be tested, if you use it something like this:

class IsPopulatedDataHolder extends ArgumentMatcher<DataHolder> {

private final int expectedValue;
private String failure;

public IsPopulatedDataHolder(int expectedValue) {
this.expectedValue = expectedValue;
}

@Override
public boolean matches(Object argument) {
DataHolder holder = (DataHolder) argument;
if(holder.getValue() != expectedValue) {
failure = "Held value does not match. Expected " + expectedValue + " but was " + holder.getValue();
return false;
}
return true;
}

@Override
public void describeTo(Description description) {
description.appendText(":" + failure);
}
}

Look at the description it builds. It looks suspiciously like a JUnit assertion message. We should let JUnit build the message for us. You can just throw the AssertionError out of matches(), but then you lose the nice Mockito-provided stack trace that hooks in with your IDE.

Have a look at this extension of the ArgumentMatcher. We'll catch the AssertionError, and stuff the message into Mockito's failure description:

public abstract class AssertConvertingArgumentMatcher<T> extends ArgumentMatcher<T> {

private String failure = null;

@Override
public void describeTo(Description description) {
description.appendText(failure);
}

@SuppressWarnings("unchecked")
@Override
public boolean matches(Object argument) {
try {
verify((T) argument);
} catch (AssertionError e) {
failure = e.getMessage();
return false;
}
return true;
}

protected abstract void verify(T argument);
}

Given this class, we can now use a JUnit Assert to build the string for us with a one-liner:

class IsPopulatedDataHolder extends AssertConvertingArgumentMatcher<DataHolder> {

private final int expectedValue;

public IsPopulatedDataHolder(int expectedValue) {
this.expectedValue = expectedValue;
}

@Override
protected void verify(DataHolder argument) {
Assert.assertEquals("Held value", expectedValue, argument.getValue());
}
}

Normally if we use an Assert in matches(), we lose the failure stack that Mockito so nicely provides for us. AssertConvertingArgumentMatcher:

Ties together the descriptive JUnit error with the Mockito custom matcher
Reduces lines to test the target code
Keeps intact the standard mockito stack trace (double-click drill-into-test, double-click drill-into-target)
Uses JUnit to build "Line width expected: <2.0> but was <1.0>," and this means no hand-coding of descriptions. JUnit builds the description for us.
Assert is one line instead of if(this) etc,etc

I've put some code samples here:

git://github.com/DonBranson/MockitoMatcherExamples.git

Favorite Project Series - The Data Federation Service

2010-06-29T06:01:00.001-07:00

The Data Federation Service (DFS) is a project that we did (in 2006-2007?) for a client as part of a much larger project. The goal of DFS was to provide a framework for distributing documents to users around the globe, many of whom would be on very slow and/or very unreliable networks. A document in this context is any kind of file that contains published information - it might be a text document, a spreadsheet, an image, a diagram, or anything else. To the DFS, they're just files.

The system had to run on hardware that was already largely in place. There was a primary node with failover at the highest layer, an intermediate layer, and layer of regional nodes distributed around the world. The regional nodes acted as connection points for mobile devices.

The goals of the system were to 1) provide a web-based façade for document authors to publish documents that might be of global or regional interest, 2) distribute global documents to all the regional servers and regional documents to a regional server plus an alternate server that would act as that server's backup, 3) provide redundancy alternate paths for document delivery in case of server failures in the middle layer, 4) a mechanism to deliver annotations to documents back to the author for vetting and addition to the document, 5) provide pluggable reachback to external data sources, 6) allow users of mobile devices to flag documents that they wanted pushed all the way to their mobile devices whenever updates are available, 7) store metadata about the documents that describe the document and the source, and can be used for filtering, 8) provide a complete list of available documents to all of the regional servers, so that mobile users could search for and retrieve documents that are targeted to other regions, and 9) provide command-line and service interfaces for programmatic interaction with the DFS.

The system required 99.9% availability for document publication. The mobile devices existed in an environment where network connections might be sporadically unavailable, or even unavailable for weeks before connectivity was restored. They needed to get whatever updates were available when the reconnected. Furthermore, the actual NICs as exposed on the mobile devices by the O/S to the application could come and go, and the application needed to be able to tell when a NIC became available. The application had to detect network availability and retrieve new documents.

The team varied over time, but included five (and more, at brief intervals) developers over the course of the project. Working on a distributed system like this creates some complexities that a lot of developers wouldn't necessarily consider. For example, we wanted to be able to delete documents. No problem, right? Well, what about annotations that might come in weeks or months after a document is deleted? The answer was to simply mark documents for deletion, and not physically delete them. This allows then, that an author can be notified of an annotations to a deleted document, and see if the document needs to be re-activated, or if the annotation provides information needs to be attached to another document. Fortunately, the folks on the team were able to spend the time that it takes to dig into it and understand the distributed ramifications of code changes and design decisions.

Some of the key technologies involved:

JmDNS - Used to detect server presence. This was where we learned that Java's ConcurrentHashMap can sometimes throw a ConcurrentModificationException, requiring us to handle it and recover from it, so that server availability was reliably detected.

Servlets - WebServices were considered and rejected because we needed to stream data through a series of nodes so that the latency hits would overlap and reduce overall delivery time from the primaries or external sources all the way to the regional servers and mobile devices. The required servlets were mostly hosted on Tomcat engines. I originally scoffed at the idea of running them under Jetty - but I had a wrong view of what Jetty could do for us, and we could have replaced all the Tomcats with Jetty.

JMS - document publication triggers multiple notification messages to the intermediate layer. A regional document for, say, the North America region, would trigger two notifications to the North America regional node and two to its backup node, which might be the Europe node. Two (or more) messages are triggered, and each goes through a different node in the intermediate layer, providing redundancy for the message delivery. If a node in the intermediate layer is down, delaying delivery of one of the messages, the alternate message can still go through.

One of the key attributes of the DFS was the ability to live in a highly unreliable network environment. Connections could come and go, which means that we needed a JMS provider that would reliably reconnect. We started initially with ActiveMQ. It's easy to configure via Spring, and has some nice automatic features that reduced the amount of manual configuration needed. During endurance testing where I ran a series of integration tests over a period of about two days, I found ActiveMQ would reconnect after a server failure/recovery between 60-70% of the time. We tried a number of different solutions, but nothing worked to raise that percentage. I tried Joram as an alternative and found that it was well-suited to behave reliably on an unreliable network, and would reconnect 100% of the time. We did have problems with Joram's distributed JNDI at the time (I think that's since been resolved), so I used local JNDI and wrote a bridge to move messages between nodes, meaning that the rest of the DFS only had to perform local JNDI queries. Joram stood up well in short tests of a couple days and in longer endurance tests of up to two or three weeks.

The architecture relies on the guaranteed delivery that JMS provides to ensure that each node will receive the message if connectivity is available. If a node is down, it will receive the message when it comes up. Each message provides the document's MD5 and server location for the identification and retrieval of the document. When a node receives a message, it turns around and attempts to fetch the document from the originating server. If retrieval fails, the message remains on the JMS queue and an attempt is made later. When the alternate message comes through, the regional node can determine from the MD5 that it's already pulled the document, and can discard the second message. This is the basic reason for using notification messages instead of document delivery messages - to avoid the unnecessary duplication of the document on the wire.

Each layer is like the next. That is, there's nothing special about how one layer retrieves documents from the next layer up. So, additional layers can be added if desired. Mobile users have a complete directory, which means they can request documents that are not on there regional server, and therefore have to be fetched from the primary server or from another regional server. Fetching from other regional servers is tried first to distribute the load.

A portlet provided the front-end for authors. It was, unfortunately, kind of clunky, but served as a starting point where a better wrapper could be written over the underlying framework. The only thing really interesting here is that we discovered that the way we used the file upload utility caused three copies of documents to be held in memory during the upload process. Yuck.

As it turned out, this solution ended up sitting on a shelf. Why? Because the three or so applications that were targeted to use this framework for document distribution were delayed and eventually cancelled.

After that, we began to look for opportunities to apply this architecture elsewhere. Given that it's a lot easier to sell something that has a GUI than it is for something that's just a bunch of wiring with a clunky portlet, I took an OpenMap earthquake application that I wrote at home and modified it to fetch its earthquake section data from the DFS. The application displays a world map, pulls NEIC data that we published to the DFS, and displays the earthquake events on the map.

Why is this a favorite project, despite that fact that it's now collecting dust? Because of the extra challenges presented when designing applications that are distributed. It's an extra dimension of complexity above and beyond what a basic web app or gui app requires, and that makes a project like this quite a bit of fun. Another reason is the people on the team. We had a small, competent group that was dedicated to code quality and more importantly, to product quality.

Weblogic JMS, The PointBase 30MB Limit, And Switching To MySQL

2010-01-30T02:37:00.000-08:00

Overview

I've been running into this problem lately that occurs when dumping messages into Weblogic JMS queues on a developer's workstation. The underlying problem is that the PointBase database provided with Weblogic has a hard-and-fast size limit of 30MB. The last thing the company I work for wants to spend money on is database licenses on developers' boxes, especially when there are plenty of free options available.

I'm pretty comfortable with MySQL, and always have it installed on any developer box I'm using. But, Weblogic supports a long list of alternative DB types, so pick one you like: Adabase, Cloudscape, DB2, Derby, EnterpriseDB, FirstSQl, MS/DB, Informix, Ingres, MS SQL, MaxDB, Oracle, PostgresSQL, Progress, and Sybase.

My original goal was to rip out PointBase entirely, and use MySQL exclusively. There may be a way to do that, but it seems that it's a matter of going through and replacing each configured PointBase datasource with a MySQL datasource, then switching over. In the meantime, I just replaced the one datasource I needed to store JMS messages. Assuming you already have Weblogic and your database of choice installed, it boils down to about four steps: Creating the database, creating the datasource, creating the JDBC store, and creating a JMS JDBC store. Most of the info is from links I hunted down and pulled together to make this list, so links back to the original docs are included.

Creating The Database

Okay, this one's really hard. ;) Here's the MySQL command:

mysql -uroot -e 'create database wls'

I just picked the DB name. We'll use it later. The name doesn't matter, just be consistent.

Creating The JDBC Data Source

Navigate Services->JDBC->DataSources->New. The name and JDNI name don't matter, again, be consistent.

Name = mysql-wls
JNDI name = jdbc/mysql/wls
Database type = mysql
Driver = com.mysql.jdbc.Driver

Click Next. I took the defaults:

Supports Global checked
One-phase commit

Click Next.

Database name = wls
Host = localhost
User = root
Password = <your root's pswd>

Services->JDBC->DataSources->mysql-wls->Targets

Servers=examplesServer

Creating The JDBC Store

http://download.oracle.com/docs/cd/E12840_01/wls/docs103/ConsoleHelp/taskhelp/stores/CreateJDBCStores.html

Services->Persistent Stores->New->Create JDBC Store

Name=mysql-jms
Target=examplesServer
Datasource=mysql-wls
Prefix=jms_

Creating The JMS JDBC Store

http://download.oracle.com/docs/cd/E12840_01/wls/docs103/config_wls/store.html#wp1142690

Navigate to Services->Messaging->JMS Servers->examplesJMSServer

Persistent Store->mysql-jms

As a final thought, you may want to make your Queues persistent. ;) There's not much point setting this up if you're keeping your messages in memory:

Navigate to Services->Messaging->JMS Modules-><module>-><queue> ->Overrides,Delivery Mode Override=Persistent

Adsense Ads and GWT - Making it work.

2009-12-19T11:26:00.000-08:00

It seems like a lot of people have had this same problem, but I haven't found anywhere on the net where someone has found a solution. Here's how I got it to work. If you find this post helpful, I would ask the favor that you check out http://penwag.com, and ask your friends to do the same.

I struggled for a long time trying to get an Adsense ad to appear in a <div>, but that seems to be the wrong approach. Divs are nice for styling reasons, but it seems that Adsense knows when it's in a div, and won't display.

I avoided IFrames (which is what the GWT Frame object compiles to) because sizing isn't automatic. Eventually, though, it became apparent that IFrames were the way to go, since Adsense ads will load in them. I create an IFrame and point it at a static page that contains the necessary Adsense script. That just works. The content loads correctly, and ads will display.

But here's the rub. IFrames need to be sized with custom javascript. I use this javascript: https://penwag.com/home/iframe.js. This works easily for all browsers except - you guessed it - IE. Below is the GWT code that I use to bring it all together, including a work-around for IE.


    public static native String getUserAgent() /*-{
        return navigator.userAgent.toLowerCase();
    }-*/;

    private Widget buildMainPanel() {
        Widget mainPanel;
        if(getUserAgent().contains("msie")) {
            mainPanel = buildIEPanel();
        } else {
            mainPanel = buildNonIEPanel();
        }

        mainPanel.getElement().setId(getPanelId());
        mainPanel.addStyleName(Styles.StaticPanel);

        return mainPanel;
    }

    private Widget buildNonIEPanel() {
        Frame mainPanel = new Frame();
        mainPanel.getElement().setAttribute("onLoad", "resizeCaller();");
        mainPanel.setUrl(getRootPage());

        return mainPanel;
    }

    private Panel buildIEPanel() {
        Panel mainPanel = new VerticalPanel();

        HTML adBar = new HTML("Loading...");
        mainPanel.add(adBar);

        try {
            RequestBuilder builder = new RequestBuilder(RequestBuilder.GET, GWT.getModuleBaseURL() + "ie_ads/index.html");
            builder.sendRequest(null, new RequestHandler(adBar, null));
        } catch (RequestException e) {
            adBar.setHTML("");
        }

        HTML content = new HTML("Loading...");
        mainPanel.add(content);

        String errorMessage = "Failed to load content, please try again later.";
        try {
            RequestBuilder builder = new RequestBuilder(RequestBuilder.GET, GWT.getModuleBaseURL() + getRootPage());
            builder.sendRequest(null, new RequestHandler(content, errorMessage));
        } catch (RequestException e) {
            content.setHTML(errorMessage);
        }

        return mainPanel;
    }

A New Star

2009-10-24T07:53:00.001-07:00

I'd like to introduce you all to an up-and-coming star in the universe of photography. Also, he's my son, Taylor. I may be biased.

Please take a moment to visit his web site PhotoImageOgraphy. I think you'll be glad you did.

Ozark Trail Volunteer Work

2009-09-13T04:59:00.000-07:00

Well, after a couple of years using the Ozark Trail, I finally got around to helping to build the Ozark Trail on the Courtois Section. (It's pronounced Code-Away.) What a tremendous experience. The people were great to work with, and great to hang out with after the work was done for the day. And the food, thanks to Jeff The Chef, really hit the spot. Nothing like a couple burgers and a few brats to make the day complete.

The group included some tremendously hard workers on the rock-wall team, guys I couldn't even begin to keep up with energy-wise, including Scotty from the USFS, Russ, Gabe, Charles and others. Good work, guys.

No injuries, either, which is good. Couple close calls, though - the hill was about a 25% grade, and a few of the boulders got away from us.

What a great bunch of folks to hang out with. I'm looking forward to the next event.

The Value Of Pictures In Software Design

2009-07-14T04:27:00.001-07:00

There are some very good reasons why software engineers use visual communication to quickly and effectively transfer knowledge from one person to another.

While people have many different learning styles, and while everyone employs all of the styles to a greater or lesser degree, most people, or at least, enough people to matter, are predominantly visual learners. Various sources claim that around 60% of us are visual learners. Therefore, it's worthwhile to use visual techniques for this reason.

Visual communication transfers information at a very high rate compared with aural and textual communication. You can tell with a glance a system's structure, or lack thereof. A verbal description takes longer.

Visual communication helps the sender, too. That is, the person creating the graphical representation has to understand the system well enough to draw it. This applies to verbal and written communication as well, so visual does not necessarily hold an advantage over other forms, but it's certainly a valid approach.

Visual communication helps a newcomer to the team come up to speed and become productive more rapidly.

Furthermore, graphical representations of software systems can reveal flaws, voids, and redundancies that are not immediately obvious in verbal or written communication. How many times have you drawn a system diagram, only to see that there's something that can be cleaned up? If you have not done this, try it - it's a worthwhile exercise.

To illustrate the value of pictures, let me point you at one that someone else drew, one that helped me to quickly understand a software framework's design and intended usage. I'm talking about the design of Mina, Apache's ongoing effort to wrap the basic Java NIO components. I think the graphics they provide are a great example of what we should be doing on our own projects.

Why is it important to understand the need for visual communication in software development? I've noticed something unsettling on the last couple projects I've been on: no graphical representations of the systems. In each case, there was no perceived need or gain to having images. For those of us who have been on non-trivial projects and witnessed the indispensable benefits of this form of communication, this is a red flag.

This red flag is a reliable indicator of a project in trouble. The particular dysfunctions might take one of several forms, but most likely is a combination of them. I'll try to name a few here, without trying to make an exhaustive list.

First, it means that the system probably has no clear direction. The team doesn't know where it's headed, or at least doesn't have a common vision of the goal.A shiny new a idea comes along, it's legitimately cool, and we go down that track. And that's great - provided it complements the existing framework. We can't follow all of those cool ideas. Some we'll have to back-burner for another day or another project. A solid understanding of a system as a whole, bolstered by few good drawings, can help us stay disciplined and on the road toward our goal. The images help to remind us of the goal, and to keep us from working at cross-purposes.

A lack of graphical communication might indicate that the team is unable to create them. The system has grown disorganized and chaotic over time (that is separation of concerns is largely gone), and the team, however good they are, cannot walk up to a whiteboard or show on paper a cohesive, overall design. The discipline of striving to achieve the goal of always having something drawn (simple, not 200 pages), drives us to keep the system cohesive and well-organized.

A lack of drawing might indicate something far worse - that the team refuses to draw them. This might be from fear that the drawings will become stale (a legitimate, but addressable concern), or because it takes time from coding, or from a misapplied development philosophy, from plain and simple laziness, or because of a lack of experience building complex systems. Even with current development paradigms that eschew grotesquely large architectural documents, some documentation is essential.

It's this last statement that seems be key element on the last couple projects I've been on. Agile development philosophies encourage us to limit the amount of useless documentation that gets created. This is a worthy and noble goal. Sadly, some have twisted the intent of these goals, eliminating strong, time-tested tools from their arsenal, to the detriment of the projects and teams they represent.

John Roth, founder of the Ozark Trail Association, has died

2009-07-09T05:14:00.000-07:00

This is sad news, indeed. I only ever "met" John via email and discussion forums, but never had the privilege of meeting him in person. As I read the notes from the forum linked below, it's clear what a void his passing leaves.

Ozark Trail home page
Forum notes

A Worser Mousetrap

2009-07-04T14:30:00.000-07:00

<tirade>

I have this, uh, friend, that has a rodent problem. Yes...a friend has this rodent problem.

So anyway, in order to help this, uh, friend, with his rodent problem, I used to buy this Victor snap trap at Lowes. Now, they don't sell those any more, they sell this Tomcat snap trap instead. They're very similar, except that the Tomcat is $0.78 instead of $1.10. There is one other minor difference - the Victors work. They actually catch mice. The Tomcats don't. They suck, actually. The mice come, take the bait, and leave. The traps don't trip. I even jammed the bait into the little holes so that they'd really have to apply a lot of force to get it - end the traps still don't trip. I would've been happy to spring for the extra 32 cents even, for a trap that just ^*(&^*&^%$%# works.

I wouldn't even be griping here if I hadn't just come back from Lowe's, having returned a branch cutter that pulled apart during normal operation: I was cutting a redbud branch about the size of my thumb. Forty bucks and it lasted about an hour. Jeez, what a piece of crap.

Next time you and the kids want to have fun, go play this game. It's called Made in America. Go into your local Lowes, and try to find a product that's made in America. Set a time limit, though. If you haven't found something after an hour, it's time to give up.

</tirade>

Seeking Input For My Next Opensource Project

2009-06-13T13:31:00.000-07:00

Friends,

I'm seeking your input to help me think about my next opensource project. I have two ideas, either one of which I'd like to do, probably using Java as the primary language, just as a matter of preference. I'd particularly like to know whether either is already being done, so that I don't duplicate work, and whether or not you think it might be something useful in work that you've done.

The first is a SOA Directory Service alternative to UDDI. When I worked to help implement a middleware/SOA framework in the mid-90s, one of the pieces we built was a directory service. While it didn't offer the metadata storage capability that UDDI offers today, it had some advantages over UDDI. It was simple: easy to register services, do lookups, etc. It was very fast. Services registered using a lease mechanism, so you could get a list of matching service instances, knowing that the instances were probably still up. Next, it was replicated. Certainly, many UDDI implementations support replication. What I don't want to do is create another UDDI implementation, but rather to build an alternative Directory Service that is more like what we did in the 90s, consistent with today's framework needs, but more lightweight. To my mind, UDDI is far more heavyweight a solutions than most enterprises need, and a simpler solution might offer some appeal, provided that it integrated well with whatever framework they're already using. That is, that it would be easy to choose it as an alternative to UDDI.

The second possibility is to do an opensource implementation of a data federation system. We built one for a client that was never used, but there were some good ideas in there. I'd like to do it again as an opensource project, because it offers some useful capabilities. It essentially allows users to publish documents to a master node, then replicate documents to regional servers, that is, to push the data close to where it would be used within the organization. For example, if a document were flagged as pertinent to an organization's European region, it would be be pushed to that region's server, and to its backup server in a neighboring region. Users in the region can then make annotations to the documents as needed, and push those back to the original author for consideration. A federation system such as this offers some availability and performance benefits relative to having a monolithic document server. When a user wants a document that not stored in his or her region, the system goes back to the master or another regional server to fetch it. As an added capability, the original system supported plugins that could fetch data from external sources, and that might be useful to include.

We Don't Have Time To Skip That Step

2009-06-07T16:22:00.001-07:00

In the mid-90s I was privileged to be on a team building a service-oriented middleware architecture (Datagate), which I have mentioned before. We used an underlying library that implemented XDR, for which we had no unit tests. Since much of our software was built on this and a couple other core technologies, it was important that they be as solid as possible. Make the foundation solid, and the rest will follow. I decided to build a suite of unit tests against this software, suspecting that there were some bugs in there. Using a coverage tool, I wrote tests that covered the entire XDR library, and we found that the suite would not run on one of our twenty or so supported platforms. We fixed that bug in the library.

After that, we found that one of our nagging bugs went away in our Directory Service. It turns out that the two were related. I took some time to build the test suite, but not that much. It saved us time in supporting the Directory Service. It probably saved application and service developers time, too, but we didn't research that.

After that, when it came to the value of testing, I told everyone who would listen, "We don't have time to skip that step." And it's still true today. Can you afford the extra time it takes to skip testing?

OT Courtois Section Trip Report - 5/30 and 5/31

2009-06-06T13:22:00.000-07:00

I hiked the part of this section from Hazel Creek to Bass' River Resort last weekend, camping after it crosses FR2265 the first time, before the trail cuts west there. What a great weekend to hike - a little on the warm side, but tolerable by watching my pace. Finished up Sunday morning about 11 before the heat really got going, and sat in the shade by the creek, enjoying a couple free beers from some new-found friends and skipping a few stones. Good times.

If you hike this direction, you need to know that when it goes into the field near Harmon Spring, you need to head forward across the field, taking the trail that goes slightly to the right. There's a trail to the left and to the right, and I went left, getting about a half-mile up the road before convincing myself that that wasn't the right way.

Also, if you're going this way, make sure you get plenty of water at the Beecher Artesian Well spring, especially in warm weather. The next decent water isn't until you're almost at Bass', and a long trek on the gravel road section will surely dry you out. There is some water before there, but I decided it wasn't for me, even purified, and I'm not too picky.

South of Highway 8, there are about 4 or 5 deadfalls. North of 8, maybe a couple.

The trail's pretty horsey in sections - plow through, and it gets better.

The ticks were bad, too. Had a couple in new places from this trip. Both standard ticks and seed ticks were on the prowl.

Not a lot of people out - just saw three guys in the Berryman section, but there were at least three people ahead of me, but making better time.

I really like this section. At least on the part I hiked, the hills are pretty docile, and there's some pretty trails through some really impressive pine groves.

Manage Your Eclipse Install With A Local Git Repository

2009-06-04T15:10:00.000-07:00

I had something of an epiphany this morning. If it wasn't the real thing, it sure felt like it. With Eclipse, I'm often trying out new plugins that purport to do one other or another, and there are usually a few that do the same thing, but with different features. So the quandary is, what do I do when I pick one of the several, try it, and don't like it? Perhaps it's hard to use, perhaps my IDE crashes more often than it used to, or perhaps one of the other similar plugins suddenly seems more appealing.

Traditionally, I've solved the problem using rsync. I rsync -a eclipse/ eclipse.beforeWonderfulPlugin, then install the plugin. If the plugin turns out to be a flop, I go back to the old version, which takes a little while. First, delete eclipse, then rsync -a eclipse.beforeWonderfulPlugin/ eclipse. It's slow, but it easier and faster than say, cp or tar. Also, I now have (at least) two entire copies of eclipse laying around.

There are further issues, too, though they're perhaps less important. It's hard to remember what plugins I have installed, for example, above and beyond what's already included.

Lately I've making the move to git to manage source for various projects. Then it hit me - I should make my Eclipse install a git repository. So that's what I did. There's a master branch - that's where the downloads from Eclipse go - ganymede->ganymedeSR1->ganymedeSR2, and so forth. Then, there's the working-branch, which is where I'll normally run from. When I'm trying a new plugin, I'll branch off of working-branch, try the plugin for a while. If it's good, I'll merge it back into working-branch.

Presumably, there will be an Eclipse Ganymede SR3 at some point, and I'll rebase there. When Galileo comes out, I'll put that on master, and probably start a new working-branch.

If I merge a plugin into working-branch, and find out a week later that there's a problem, it's easy to re-branch from a point prior to the merge, and get rid of the plugin. Sometimes it's the case that there's some quirky behavior I didn't notice at first. It's a matter of a few minutes to go back to a previous version, try it out, and see if the quirky behavior was there all along, or started with some new plugin.

Sure, Eclipse lets you uninstall plugins, most of the time, but there are often problems with that.

Could this be done with CVS, Subversion, ClearCase or other revision control systems? Very possibly - but not in practical terms, simply because of the performance issues.

If there are alternative plugins, say, Subclipse and Subversive, and I want to try them both, I can have them both on separate branches off of working-branch, and explore them both for a while before picking one. As an added bonus, gitk lets me see where I've been, and when I started using which plugins.

Too Much Testing?

2009-05-24T05:43:00.000-07:00

I recently heard this question:

"Is too much focus on testing benefits a bad thing overall?"

TDD done well can improve readability. TDD done poorly, that is without consideration of other important principles, can reduce readability.

A guy I worked with in the mid-90s would say "You can always make a system more flexible by adding a layer of indirection. You can always make a system simpler by removing a layer of indirection." Both flexibility and simplicity are important qualities of a system. The two principles can often live together in harmony, but often they work against each other. If you go too far towards one extreme or the other, you move away from the ideal that exists where these two principles are balanced.

TDD is partly about testing, partly about design. TDD done poorly can tend too much towards either flexibility or simplicity. It can push towards too much flexibility. The objects become more testable, and often simpler, but the inherent complexity of the domain problem then is pushed out of the objects into the interaction of the objects. We gained flexibility, and to the naïve eye, it can look as though we've gained simplicity because our objects are simpler. The complexity, however, is still there. It's moved out of the objects, and into the object interaction, where it's harder to control. There are code smells that can act as red flags here - a system with hundreds of small objects and no larger objects is one, lots of objects with only one-line methods is another.

TDD done poorly can move in the other direction as well, that is, towards too much simplicity. So, we do TDD by writing the test first, but it has little impact on our design. We still have long methods and huge objects, and those are code smells that can red-flag this problem.

Now TDD will not by its nature knock you off-balance in either direction, provided it's well-applied. Use other practices to keep you on track. For example, draw pictures of what you're doing before you do it. Obviously, not all the time. Some things are far too simple for that. Some pictures are worth saving, some are just sketches that help us to visualize the problem, and we are, by varying degrees, mostly visual learners. If you can't draw a picture of the problem, you don't understand it.

How will this help with TDD? It will help to keep a system from going too far on the flexibility side, away from the simplicity side. If you draw a picture and it's ugly, that's a red flag. Sometimes it's necessary, but often when you draw the picture, your mind will quickly see things that can be simplified. The solution becomes more elegant and simplified, easier to maintain, and more enjoyable to work on. If you can't or won't draw pictures of your system, you're losing this opportunity to make your software more solid, more elegant, more beautiful to see and easier to maintain.

Applying this comes with experience, and some coders will never understand the value that a good balance provides. There's no metric that you can run that tells you you're in the right place. If someone gives you a prescribed method to arrive at that harmonious point, he's lying to you. More importantly, he's probably lying to himself without realizing it.

So, my answer to his question is 'yes': test everything without forgetting the other good principles.

Any good practice will throw you off-course if it's not balanced with other good practices.

Creating A Remote Branch With Git

2009-05-23T05:35:00.000-07:00

If you're still doing this the way I suggested. Stop. Stop right now. Back away from the keyboard, and nobody gets hurt. Also, read the comments below, and do it the way Graham shows.

Original post:

----------------------------------------------------------------------

So, I've seen this documented in a few places, and it seems like they all give you various hard ways to do it. Here's an easy way. But, given git, it's surely not the only easy way. I create remote branches as a way to self-collaborate on a new branch from the various boxes in my daily life. If you're not a computer geek, you probably don't know that by "box" I mean "machine." And by "machine" I mean "computer." But, if you're not a computer geek, you're probably not using git, in which case you're not reading this. In that case, stop now.

Assume that you're in a clone of a remote repository. We'll create a remote branch named "mybranch." First create the branch locally:

>git branch mybranch

Switch to that branch:

>git checkout mybranch
Switched to branch "mybranch"

At this point, list all your branches:

>git branch -a
 master
* mybranch
 origin/HEAD
 origin/master

Now, here's the command that creates the remote branch:

>git push --all
Total 0 (delta 0), reused 0 (delta 0)
To /tmp/remote.git
* [new branch]      mybranch -> mybranch

It's just that easy. Now, in another clone on another machine, I can self-collaborate by creating a local branch that mirrors the remote branch. First, list all the branches:

>git branch -a
* master
 origin/HEAD
 origin/master
 origin/mybranch

And create a local branch that tracks with the remote branch:

>git checkout -tb mybranch origin/mybranch
Branch mybranch set up to track remote branch refs/remotes/origin/mybranch.
Switched to a new branch "mybranch"

List the branches again to see the change:

>git branch -a
 master
* mybranch
 origin/HEAD
 origin/master
 origin/mybranch

Run gitk to see visually that your local branch is tracking the remote branch. Use git push and pull as you change machines to move the changes around. When the branch is done, you can merge it.

I like git. I tried bazaar and mercurial, but git "just works" and is easier to get started with than bazaar. Plus, there's github. Visit me there at https://github.com/DonBranson.

DonsProxy on github

2009-05-20T18:17:00.000-07:00

I've been thinking about getting DonsProxy out on github for a bit now, but needed to break out the good stuff from the slough. So, it's finally out there. My next DonsProxy task is to get the SSL part working better - maybe by injecting Mina. Still trying some things out. You can clone DonsProxy from github at http://github.com/DonBranson/DonsProxy/

Favorite Project Series - Datagate (14 Years of SOA)

2009-05-07T17:17:00.000-07:00

SOA is one of the big buzzwords these days, and a lot of people have started doing SOA for the first time in the past few years. I have some good memories of the "good old days of SOA," and now it seems like everybody's using the groundwork that was laid in the 80s and 90s. Now, people can just go download Weblogic and Systinet or JBoss, and and the various pieces, build a system using those frameworks, and they're "doing SOA." There are so many powerful tools; the IDEs such as Eclipse and Intellij do a lot of the tedious work for you.

My first introduction to SOA was in 1995 on a project called Datagate at Southwestern Bell. I came on the project when it was about six months old. All of the things that people take for granted now, that they use and it "just works" (or not), we had to build our own. There was no app engine, no UDDI, no SOAP. The things that people download and use today, we built and used back then. If you're a computer geek, this is the project of a lifetime. It was all message-based (that's a MOM for all you acronymophiles). Now I can look around and see SOA everywhere, but we were, in our own way, pioneers of the technology, because we did things no one had ever done before. Sure, not everything we did was new - Tuxedo had been around, so MOMs weren't exactly new.

What we did in the mid-90s that was new was to make the technology more accessible. It was hard to write Tuxedo services back then. We made it accessible to every C, VB, and PowerBuilder coder out there. There are some ways in which today's technology is better. But, what we built was so smokin' fast, and there's nothing that SOA folks are doing today that even comes close to the performance we delivered on those old, slow machines. We had a great, usable middleware product that was the framework for developers to construct clients and reusable services on a variety of platforms - about 12 unixes, MVS running on IBM 390s, Windows 3.0, Windows NT, and Tandem. It's one of those experiences that you looked back on, and think, we did it before most anybody else. What people today use, we built.

I wasn't one of the visionaries for that group. I was brought in to develop an interface so that VB and PB programmers could write clients on top of our C framework. In the end, I help design and architect that, the Directory Service Replicator, and the Dashboard for this system, and we did good work.

There was no SOAP back then. The Datagate team developed its own protocol. It was message-based, so was much more performant, scalable, and simpler than the RPC-style calls are today. Yes, even the message-driven services are RPC under the covers, not the true MOMs like what we built. A true MOM lets you fire off multiple messages to services without waiting for the ack that the message was received, then process the responses as they come back in. All this with one thread, not multiple threads that RPC pushes you towards for this kind of processing.

There was no UDDI then for service lookup back then. There was no LDAP, either. We built our own distributed, replicated X.500 directory service. Infrastruture and business services registered and reserved leases dynamically. If a service went down, its service entry went away, and clients would call one of the other instances that was still registered. And of course, it was fast - 10s of millis to do a service lookup, not 100s or 1000s like you see with UDDI. Smokin' fast.

I architected and led the development of the Directory Replication Service along with my friend Dave C. This cell-based, scalable solution allowed us to run multiple Directory Services, and replicate data between them in a scalable, available fashion.

We used XDR for marshalling. That's one piece we did download. I lead the charge to unit test the whole thing - and found only one bug. There was an endian issue on that affected one of the platforms, and we found it and fixed it.

There were no application engines. What we built was a resource manager that would control service lifecycle and heartbeat services to quickly detect outages.

There was no PKI, or even SSL. We bought licenses for the encryption libraries, then designed and built our own secure message protocol (with mutual authentication where needed), plus the infrastructure to support it. Certificate Revocation Service, Certificate Authentication Service, and al the APIs for services to use to load their certs, encrypt their messages. We built the services and the GUIs so that administrators could manage the PKI we built.

There was monitoring software available - Tivoli - but it was far more than our budget would allow. We built our own Dashboard that allowed us to closely monitor infrastructure and business services running over much of the United States, including Missouri, Texas, California, and more.

We did training to teach developers how to build reusable business in C, and how to build clients in C, in VB, and in PowerBuilder. I taught the VB and PowerBuilder classes.

Our team consisted of three subteams - Infrastructure, which was the team I was on, the Service Writers team, and System Adminstration, which handled the care and feeding of production systems.

At A.G. Edwards, we leveraged SOA frameworks for 6-1/2 years as part of a broker workstation, to develop the pretty scalable agedwards.com (which, last I heard some years back, had about 300,000 users signed up). I was a member of the Lead Architect Team that designed and developed agedwards.com. We used SOA for some other pretty cool stuff. One of my favorites was the BLServer, that used a modified version of what is now called the "Competing Consumers" pattern in a highly-available cluster of message-based services that self-allocated using leases to process roughly 1.5 million messages a day for about 4-1/2 years before it was mothballed in February of this year.

The industry has come a long way since then. Like I said, what we built, you can now go download, and the SOA business has experience explosive growth. After 2-1/2 years at Bell on the Datagate project, I did 6-1/2 years of SOA work at A.G. Edwards, continuing to architect systems using new SOA technologies as they emerged - Tengah, (now Weblogic) Dynamo, Novell's LDAP server, SiteMinder, WSDs, Big/IP, and various JMS providers. Some of this stuff I had more exposure to than others, but that gives you an idea of how things have changed over the past 14 years. Now there's CXF and JAXB and other exciting new tools coming along.

Every Tom, Dick and Harry does SOA now. Years after that work at Southwestern Bell, a small company hired a "SOA expert" (after I was already there!). This poor fellow didn't even know what LDAP is. When I described how it could be used for service lookup, he said you can't use it for that - it's for authentication. I explained to him that it's just a database that's optimized for reads. You can use it for anything that falls into that category. Now, he's an expert, downloading what other people have built and putting it all together. One boss at that company tried to get me to join a project where I would have been writing architecture documents all day for six months, and building nothing. His angle? That the SOA experience would look good on my resume. I kid you not.

On Datagate, we built SOA sooner, faster, more scalable, and at least in some ways better than what exists today. The primary advantage I see in today's world is standards. Almost everything is standards-based, and that means vendors can provide tools to do many of the things we used vi for. It's a change for the better, in most every way, because SOA is much more accessible today that it was in 1995. Plus, now it's an acronym. ;)

Migration from Subversion to Git

2009-05-05T18:42:00.000-07:00

I'm migrating from Subversion to Git. It's about time, of course - decentralized SCMs have some significant advantages over the old-style SCMS such as CVS, Clearcase, Subversion and the like. I looked at Bazaar, which touts "Bazaar is a distributed version control system that Just Works." Sadly, I couldn't get it to work. It needed a lot of python modules that I, not being a python guy, don't really know where to get. Found some, not others. Funny thing is, Git "just works." Sure, had some hiccups with converting from Subversion before realizing that I had made an incorrect assumption about the repository structure in one case. Looks cool, though, especially when you throw github in the mix. Good stuff.

Recommended reading if you're thinking about converting from your current SCM to something new, whatever that may be: http://whygitisbetterthanx.com/

Upgrade from GWT 1.5 to GWT 1.6

2009-05-05T18:40:00.000-07:00

Just a refer:

http://penwag.blogspot.com/2009/05/upgrade-from-gwt-15-to-gwt-16.html

Tell Your Story!

2009-03-28T06:53:00.000-07:00

There's a web site that I'm building, penwag.com, that lets me enjoy two of my favorite interests: software development and story telling.

It lets me enjoy software development in that I'm using two technologies that I will gain experience with: the Google Web Toolkit (GWT) and Hibernate. The GWT is a great tool from Google that provides a framework for building AJAX-style applications. The framework includes the necessary pieces for client-side and server-side development, and manages the communication between the layers. Add in Hibernate and a data layer, and you have everything you need for easily building AJAX web sites.

Then there's the story-telling side. When I was very little, we visited my great-uncle Henry Shoop, who was my father's mother's brother. He was born near Cherryville, Missouri, but his family moved to California when he was six. Uncle Henry was a friend, a beekeeper, and one of the great storytellers, and I hung on his every word as he instilled in me a love for a well-told story. He was one of the best.

Another favorite story-teller of mine was Earl Halbert, a family friend. You can see his picture here from National Geographic Magazine. He's the one on the left in the overalls. Earl was born in 1910 in rural Missouri, and lived through the Great Depression, Prohibition and numerous other hard times. He knew my great-grandfather Thomas Jefferson Branson, and had a few good stories to share about him as well.

Both of these men, shall we say, infected me with a love of a story told friend-to-friend, perhaps while sipping some sassafras tea, or around a campfire, or just hanging out in the living room. I miss both of them not just for their story-telling and their wit, but for their character as human beings.

Certainly the art of a well-told story has not been lost, but is changing over time. How many people nowadays have had the chance to hear stories of hard times in the Great Depression, friends lost, good times, and funny times, from "way back when"? I'd hate to lose those stories. So join me, won't you, and let's tell each other those stories, like we're sitting over a beer or around a campfire, and let's enjoy a good laugh together, savor tales of clever dealings with shysters, and the occasional twist ending.

Just head out to penwag.com. You're getting in early at a site that will someday, hopefully, have many stories that we have shared with each other. But it's your stories that will make the site great. If you have even one story to share, come on out and put it up there.

This site is in its early stages, but everything you need to add a story and read others' stories is already there. Since it's in its early stages, there's still a lot of work that I need to do add features. Some are pretty obvious - you can create a story, but can't edit the story to make corrections yet. I have a long list of features that I will add as the site grows. But the feature you want may not be on my list! So, click on the Contact Support link, or just send an email to support@penwag.com, and give me a shout. Tell me what feature you want, and I'll add it to my list.

Let me reiterate - this site's success depends on people like you who have a story to tell. As you add your stories, others will visit the site just to read a great story. The stories are short. Don't feel like you have to write a novel. Just write it how you would share it with your friends over a beer.

Now, to get started, you have to register. For simplicity's sake, your email address is your user id, and you need to make up a password. Registration is quick and easy, and your email address will not be shared. If you still have questions or concerns, or if the site doesn't work right for you, use the Contact Support link and let me know!

Thanks, and hope to see you out there. Come tell your story!

OT Trip Report - Bell Mountain to Council Bluff Lake

2009-03-25T05:00:00.001-07:00

My friend James and I hiked Council Bluff Lake to Bell Mountain (reversed) March 22 and 23, in about 24 hours. It was our first hike of the year, and left me a little sore.

The view from Bell Mountain is a must-see, and is easy to get to on foot. It's pretty level all the way from the North Bell Trailhead. It'd make a fine afternoon hike, out and back.

Once you get past that, the rest of the trail has some pretty good hills, by our Midwest standards. There are long level parts along ridge-tops and in creek bottoms, but you'll have some climbs in the 200-400 foot range. There are some beautiful creeks along the way, too, so finding water was not a problem for us. Joes Creek is big enough that in wetter weather, it is probably hard to cross.

We heard from other hikers that there are feral pigs in the area, so we hung our food out of there reach overnight. We really saw very little animal life - a couple squirrels and birds along the whole trip. We saw lots of deer scat from last fall - many were full of persimmon seeds.

AIG Bonus Scandal

2009-03-24T14:12:00.000-07:00

Okay, I haven't been political yet, but here goes...

I've been thinking about the whole AIG bonus thing. Correct me if I'm wrong, but here's the way it looks to me. Congress rushed through this legislation. It's fairly complicated, and they were in a hurry to get this thing done. It seems like you can pick two of: complicated, fast, and right. They picked all three, and they screwed it up. The issue of bonuses was in the legislation in black and white, and they missed it because they voted on something without taking enough time to read it. To me, it really seems that simple.

Now as far as AIG's role, maybe the folks deserved the bonuses, maybe not. I've heard the argument that things would have been a lot worse without the work of the executives that got the bonuses. That may or may not be true. It's a real possibility, but without knowing the inside info, I can't say. But it seems clear that AIG was contractually obligated to pay those bonuses, with Congress' backing per the legislation. If they didn't pay the bonuses, they could have been sued, and lost the bonus money plus punitive damages and court costs. And they wouldn't have had a leg to stand on in court, so they probably would have lost.

AIG was between a rock and a hard place. Congress put the rock there, and now they're whining about it. Congress should man up (if there are any there) and say "We screwed up. We know what we did wrong this time, and we won't make that mistake next time." Since they're not owning the problem, they'll probably screw it up next time, too.

Don't misunderstand me - I am not defending AIG. But Congress is pretty quick to shift the blame here, because they know if we stop and think about it, we'll realize what idiots we've elected.

Now, as complicated as the bailout is, it pales in comparison to what's coming down the pike with health care. If they can't even get bailouts right, can they be trusted to get health care right? At this point, I don't think they collectively have the skills to lance a blister, much less to do their jobs well.

Testing Webapps That Send Email

2009-02-15T12:19:00.000-08:00

I've been working on a GWT-based web application (https://penwag.com) that relies on email for some of its functionality. For example, it sends email when you register, and email when you ask for your password. Standard stuff.

I've been working towards having full-blown integration tests that hit the site to make sure everything works from end-to-end. There are really two tricky areas with this. First, we want the app to send the emails, and think it's sending them out, but without actually sending them to a real email address. Second, we do need to actually check those emails to make sure that the correct content went into them.

This weekend, it finally all came together, and the solution was very straightforward. It depends on two opensource components, and adds about 100 lines to code to glue it all together.

The first opensource component is Dumbster, a fake SMTP server that's controllable through java code. It listens on port 25, and acts like a normal SMTP server as far as the connecting app is concerned. The key difference is that it doesn't send out the emails, but collects them and makes them available via a Java API.

The second component is the well-known Jetty, a Java web server that is easily embedded into other applications, which is the key for us.

Here's how we pull it all together. When the test harness starts up, it completely initializes the database, readying it for the Selenium tests. It then starts Dumbster, and finally starts Jetty. The harness creates a handler that responds to a very limited set of HTTP requests:


/ - list all emails
/stop - calls System.exit(0)
/reset - restarts Dumpster, which throws away emails and restarts with an empty list
/<number> - returns the email at this index in the list.

Combined, these features allow Selenium tests to hit a URL and fetch a given email, validate the content of the email, and, in the case of my verification email, click on the link embedded in the email.

I thought you might want the code. It's not tested, no guarantees, yada yada, but it works for me. Grab it here: http://moneybender.com/IntegrationTestHarness.java

First Chaco Sale

2009-01-11T14:30:00.000-08:00

So, I think I just sold my first pair of shoes. ;)

My wife and I were at the mall the other night, wandering around, waiting for Gran Torino to start. I noticed a couple of the shoe stores had sales on, so I headed over to Tradehome shoe store, the one where I bought my Chaco Redrocks, to see if they have them marked down, seeing as how I'm very happy with mine. They did not, sadly. So, my wife's there with me, tries on a couple pairs of shoes. A couple Merrels, and the women's Chaco Redrocks. She definitely favored the Redrocks, and now she has a pair, too. My first shoe sale!

My grandpa would be pleased, I think. He worked for Brown Shoe for 34 years, and while on the board of directors pushed to have a shoe plant built in Steelville, Mo., and it lasted for quite some time before they shut it down.

Eclipse Breakpoints

2009-01-09T09:25:00.000-08:00

Hey, here's a tip for Eclipse users. Breakpoints can be created that either suspend just the one thread your code is on, or suspend the whole VM. You can set them individually using the breakpoint properties page. Also, you can set how they're created by default in Window..Preferences..Java..Debug..Default suspend policy for new breakpoints. Pick "Suspend VM" or "Suspend thread."

Enjoy.

Leveraging Misunderstanding To Generate Great Ideas

2008-12-19T17:48:00.000-08:00

Today we had a little meeting to talk about the priorities for our next iteration. We had a really odd, fun experience.

We're trying to figure out how to implement a feature that none of our competitors' products have really addressed, but the client needs. But, it's a bit tricky, so we were talking about the problem.

The team lead explained a possible solution to the problem. It sounded like a great idea, and I re-stated the problem back to him, but I had misunderstood what he said. He in turn, heard my response incorrectly and thought that I had had a great idea. But the thing he thought I was saying actually was a great idea, and solved the problem beautifully, elegantly. We all had a good laugh, since we had arrived at a terrific solution as we went back and forth misunderstanding each other.

So, whose great idea was it? Nobody's, really, and everybody's. So cool.

Now, of course, we have something of a puzzler - how do we effectively leverage our confusion to produce more groundbreaking ideas?

Windows Vista And My Network

2008-12-07T20:16:00.000-08:00

Well, if you've been paying attention to Apple's commercials and your friends with new Windows computers, you may have heard that Windows Vista has issues. Here's one that got me.

When I hooked up a new Windows Vista computer to my wireless hub, it asked me for a PIN number. Hm. That's interesting, never seen that before. It told me to find the PIN on the underside of the WRT54G2 wireless hub, so I got it and entered it. It showed me a network SSID, which I carelessly clicked through. Suddenly, all the other devices hooked to the wireless network were dropped. Hmmm, that's weird.

Here's what happened. Apparently, using the PIN gives your computer complete control over the network hub. It changed the network SSID, randomly generated a new WPA shared key, and reset it. At this point everything else was dropped. What a pain! Shame on me for missing the fact that it was going to change the SSID, but surely I don't want the shared key changed, blocking out all other network devices. Yuck!

Bridging WAP54G and WRT54G2

2008-12-07T18:39:00.000-08:00

Let's start off by stating the goal: I have a network in a room off the kitchen where my DSL comes into the house. Downstairs is an XBox 360 connected to the family TV. Rather than run a cable down to the basement, I decided to set up a wireless bridge between the basement and the network upstairs.

Despite the warnings from Linksys to the contrary, you can use WAP54G to bridge to devices besides other WAP54Gs. In my case, I have a WAP54G in the basement bridged to the WRT54G2 upstairs. It's not too hard if you follow these instructions.

First, be very very careful not to follow the instructions provided by Linksys. It's easy to accidentally read them, thinking they'll tell you what you need to do.

Next, hook a cable directly from your PC to the WAP54G. Configure your PC to have a static IP address of 192.168.1.200 (most other addresses on 192.168.1.x will work, too). Open a browser and go to 192.168.1.245, which is the default address of the WAP54G. You will be asked for a user ID and password, and the Linksys instructions are wrong. Here's what actually works: Use a USER ID of blank, and a password of "admin."

In the Setup tab, click on "AP Mode." Select the radio button next to "Wireless Bridge Remote Wireless Bridge's LAN MAC Addresses." The WAP54G needs the MAC address of the WRT54G2, which is on the underside of the WRT54G2. Enter the MAC address in the first box and click "Save Settings."

At this point, I reconnected my PC to the network, and moved the WAP54G downstairs. I plugged in the WAP54G, connected the XBox 360 to the WAP54G, and everything's good to go. For me, the XBox 360 won't dynamically pick up an address, so it's statically configured to 10.5.128.10. Your mileage may vary.

Favorite Projects Series, Installment 2

2008-12-05T02:56:00.000-08:00

The previous project in this series I chose as a favorite due to the impact it had for the user. I consider this second project to be a favorite for the interesting technical challenges that it presented. While it certainly had impact for a lot of users, that was far less visible to me.

I lead a team of about three developers on this project at A.G. Edwards. The project was named BLServer based on a service by the same name that we used from an external provider. This provider's service was implemented using a COBOL program that would handle requests over a limited number of network connections. It had a number of limitations that we needed to overcome, which we did by wrapping it with Message-Driven Web Service running in a Weblogic cluster.

The function of the BLServer service was to receive messages for account creation and modification, including changes in holdings of various securities. The provider's BLServer had a nightly maintenance window (I think it was about four hours), and used a proprietary message format. It was secured by using a dedicated leased line and a single common password.

Our wrapping BLServer service was required 1) to expose a standards-based (SOAP) interface, 2) to provide 24x7 uptime, 3) to preserve message ordering, 4) to secure access to the service without exposing the single common password, and 5) to queue requests during maintenance windows delivering them when the provider's BLServer became available. There were also scalability and performance issues which, in combination with message ordering requirements, drove us to an interesting solution. I'm not sure about the exact scalability requirements, since that was four years ago. If I remember correctly, we initially had to be able to handle about 300,000 requests during a normal 8-hour business day, with the ability to handle peak loads of around 1.5 million per day.

The first benefit that our service provided was to expose a standards-based (SOAP) interface, and interact with the provider BLServer which took requests and delivered responses using a proprietary protocol and message format. Our service was then used by application Web Services to provide customer value.

In order to meet the scalability and availability requirements, we proposed standing up a small cluster of WebLogic engines to host a BLServer WebService. This WebService would receive requests and (using JMS) queue them for processing. Responses would then be queued for later retrieval by the calling services. By queuing requests in this way, we could use the transactionality of the JMS provider to guarantee that each message was processed once and only once. Furthermore, we could queue up a backlog of messages and feed them through the finite number of connections made available by the provider BLServer.

By using a cluster, we would be able to handle the necessary load of incoming requests, queue them, and run them through the provider BLServer, keeping it as fully loaded as possible over the finite number of available connections.

Aye, but here's the rub. We had a cluster of WebLogic engines pulling messages from the queue. How do you go about maintaining message order while at the same time leveraging the parallelization of the cluster to handle the load? Consider what happens if you have two messages in the queue in order. The first is a stock sell that results in an increase in available cash. The second is a buy that uses the that cash to buy some other stock. You can see that these must be processed in order. If one server in the cluster grabs the first from the queue, and another grabs the second, there's no guarantee that the sell will be handled first by the provider BLServer. Therefore, we have to guarantee the order in our BLServer service.

How to do that? The solution became more obvious once we realized that total message ordering was not required. What's really required is that messages within certain groups be correctly ordered. These groups are identified by key, and all messages for a given key must be ordered. Depending on the type of request, that key might be a CUSIP, might be an account number, or some other identifier.

Now message ordering with scalability becomes simpler. If all messages for a certain key are handled by a given engine, then we can guarantee ordering by pulling a single message at a time from the queue, and processing it to completion before beginning the next message. Other engines in the queue will be doing the same thing at the same time for other keys. Thus, we gain some scalability.

Oooh, but we've just introduced Single Points Of Failure (SPOFs) for each key. If a given server that handles keys that start with '17' for example, and that server crashes, then messages for those keys won't be processed, and we have failed to meet our availability requirements. That's where the second bit of creativity came into play. We employed a lease mechanism. Leases were stored in a highly-available database. Upon startup, a given engine would go to the database and grab a lease record. Each lease represented a group of keys. For example, a lease record might exist for all records starting with the range '00' to '03'. An engine starts up, finds that this lease is the next available, and grabs it. In order to 'grab' a lease, an engine will update the lease with a time in the not-to-distant future, say, five minutes. As long as the engine is up, it will continue to update the lease every two minutes or so with a new time. If the engine crashes, the time expires, and some other engine grabs the lease.

As long as an engine has a lease for a given range, it can use a selector to receive messages from the queue for that given range. We now have scalability, message ordering and high availability. Everybody say, "Woah, that's so cool!"

At this point, we've solved a significant technical issue that should be captured as an architectural pattern. We never did that. It may be that this solution is documented somewhere as a pattern, but I'm not aware of it.

At the end of the project I moved on to other things, and left BLServer in the capable hands of my friend Brian S. I heard some months down the road that the service was in active use in production, and had seen only one minor bug. I've always been proud of the product quality that our team delivered. We went through at least four or five variations of possible solutions before arriving at the one described above. In each case, we'd get into the details of the solution only to ask, "yeah, but what happens if..." and realize that we were close, but had some problem because of the distributed nature of the environment, or whatever. It was very satisfying to finally arrive at the elegant solution that we delivered.

On Software Development Metrics

2008-11-20T05:49:00.000-08:00

As software developers, we need to be careful with metrics. I think there is an understanding that it's possible to cause more harm than help with an ill-chosen approach to metrics. One of the concerns is metrics that are susceptible to gaming. To me, a concern at least as great as gaming is measuring the wrong things.

The primary opportunity for measuring the wrong thing is by measuring mechanisms instead of results. For example, measuring pairing is a measuring a mechanism. Measuring the degree of siloing is measuring a result. Measuring testing is measuring a mechanism. Measuring code quality, but better yet product quality is measuring a result. It's the results that we care about more than the mechanisms. The mechanisms are a means to an end, not the end in themselves.

It's critical to measure the result rather than the mechanism. The first reason for this is that it's less susceptible to gaming. Consider measuring the number of tests versus the number of support calls received. Certainly, both can be gamed. But it's far easier to artificially jack up the number of tests. The real desire is to produce a system of great quality, which is subjective. It's harder to measure these subjective things, but it's worth it.

The second reason is that if we measure mechanisms, we'll miss important components of producing a quality system. So, for example, tests are a mechanism that help us deliver quality systems, but not the only mechanism. What we really care about is the quality of the delivered product. What happens if we measure the desired results instead of the means to achieve that result? First, it's harder to measure, and the outcome is more subjective. But, by measuring that, we also indirectly measure all those little things that we do as developers to make sure we don't get those 2AM calls, such as perusing the code a bit before check-in, or being well-read on pitfalls and benefits of various patterns.

The third reason for measuring the result instead of the mechanism is that measuring mechanism creates a box to think in. To make a trivial example, if we take as a metric the number of JUnit tests, we'll never be free to consider alternatives. We'll always create JUnit tests, because that's what measured. When the next great thing comes along, we'll be slower to adopt it, since it's not what we're measuring. We're thinking in a box. If we're measuring results, we will be more inclined to adopt new techniques as they come along to the extent that they seem to provide a real contribution to product quality.

It's easier to measure mechanisms than results. The main reason for this is that mechanisms tend to be more quantifiable than subjective results. The ease of measuring mechanisms is why most companies do it this way, and remain mediocre. The rule of thumb is this: You'll get more of what you measure. If you want more of a certain technique, measure it, and you'll get more of it. If you want more product quality, measure that instead - whatever it takes - and you'll get more of that. When it comes down to brass tacks, you don't want more of certain mechanisms, you want better results.

OT Middle Fork Trip Report - 11/1 to 11/2

2008-11-09T07:27:00.001-08:00

This was my second hike on the OT, and my first solo hike of any length. I walked the Middle Fork section from the DD trailhead to Brushy Creek lodge, and it was beautiful weather. Surprised not to see more people out - you all missed a great weekend.

I left the DD trailhead at about 11 AM. Couldn't get down there earlier, unfortunately. Near the beginning of the walk I could hear an F15 overhead, and caught a few glimpses of it. He was doing some loops and rolls, as if he were training for a show or practicing evasive maneuvers. Not what I went in the woods to see, but pretty cool, nonetheless.

The trail is pleasant all the way, quite a few nice little creeks. Along this section it can be a ways between signs. There were a couple times where I might have wondered if I were still on the trail, except for how well-maintained it is. Most of the trail is shaded by woods, too, which is nice. There were a couple groups ahead of me, but never caught up to them enough to see them, just saw their shoe prints. Just before crossing the bridge at MF7, there was a little persimmon tree. A shake knocked a few off (if they drop from a shake, they're ripe), so I got to have a couple persimmons as a sweet treat on the trail. There were quite a few deer droppings along the trail, and they almost always had some persimmon seeds in them. Met Dan and Richard at the primitive camp there at MF7, and we chatted a bit. They saw a couple other groups on the trail.

I was planning to camp somewhere between MF8 and MF9, but there was about 2 hours of light left, so I pushed on, and ended up camping at the bottom of the hill by MF12. It was a little chilly down there, but I was warm enough to get some good sleep. This was my first night out after completing the net-tent part of my Ray-Way tarp. It's slippery sleeping on the net-tent floor, and I had just a slight incline, which meant a couple adjustments in the night.

Also new on this trip was my Cat Stove [url]http://coders-log.blogspot.com/2008/10/cat-stove.html[/url], which worked pretty well. I had a simple menu. For each meal, I had some multi-grain pasta, some pre-cooked Bob Evans breakfast sausage, and some cheddar cheese. Fuel up the stove, pour in a cup of water (that's up to my first knuckle). Get the water boiling, then add the pasta, put the meat and cheese on top, cover and cook. Tasty and provides some good energy for the trail.

Second day I started out at first light, and headed up the hill at first light, warming up quickly. Continued to see tracks from people ahead of me on the trail, but the only other people I met was a group of four on horseback going the other way. They had seen someone out who was on his ninth day on the trail.

I like Middle Fork Section. I did a hike with a friend on the Highway 21 to Devil's Tollgate section in August, and that was pretty dry and rocky, with some pretty aggressive climbing. A nice hike, don't get me wrong, but a lot more work. :) By contrast, Middle Fork is gravelly but not rocky, has plenty of water, and gentle grades throughout. The last climb before descending to Brushy Creek takes you up about 300 feet, but it's gentle enough that it's not a killer. I cooked and ate lunch at the bottom after crossing the creek, and that gave me enough energy to complete the hike.

Remember not to drink the water at Strother Creek. Check the map and fill up with water before getting there. It's not a terribly long stretch without water, but just in case.

This was also my first hike after trading in my New Balance trail running shoes for my Chaco Redrock shoes. I definitely like the Chacos. They're heavier, but don't show any deterioration after 25 miles on the trail, like the NBs did.

The hike ended at 2PM at Brushy Creek, which looks like a nice place. Friendly folks, and all that. Rested there and waited for my ride to pick me up. All in all, a very, very nice hike. I highly recommend this section as a starter hike, too. You can start at DD, and there's a trailhead at 12 miles, 20 miles, and 25, so you can bug out early if you get in over your head. There are also numerous gravel road crossings, if it comes down to that.

Also note that cell phone coverage is very sparse out there, so it's a tenuous life-line, if that's what you're counting on.

if you're on Facebook, see the pictures here:

http://www.facebook.com/photos.php?id=1295841432#/album.php?aid=10650&id=1295841432

Installing OpenCV on Fedora 8

2008-10-28T11:57:00.001-07:00

I've just finished installing and documenting this process on our company blog: OpenCV on Fedora 8

Favorite Projects Series, Installment 1

2008-10-26T12:57:00.000-07:00

I've been spending quite a bit of time these days thinking about what exactly makes a software project a good project to be on. Ever since I became a computer geek in 1980 with the purchase of an Ohio Scientific C1P from Commsci Corporation in Manchester, MO,

I've loved working with new technology, and using new, cool stuff. There are a number of projects that have been great experiences from that perspective, and I'll get to those later. But there's one that I always talk about when I'm asked what are some of my favorite projects, and we should look at that one first, and what it was that made it one of those most memorable projects.

In 1994 I was still working at Washington University, and more specifically, was doing some work for the School of Arts and Sciences. The plan was that I would sit and work in the department instead of being at a desk in the IT department. I had been up there for a little while, and we would identify different things that needed addressing.

So it turned out that there was this task that Cindy N. was responsible for that had to be done every year. It had never been automated, so she was spending two-and-a-half weeks every year manually completing the task. She dreaded it for weeks ahead of time every year, and made an otherwise happy job for her become miserable for several weeks.

Every year, Cindy had to review prospective students' records online, evaluate for what student aid they were eligible, then type up a letter inviting them to apply and detailing this information. As you can imagine with all of the manual work involved, there were going to be mistakes, and that's part of what she agonized over.

So, applying the technology at the time, we wanted to assemble the information available from an IBM mainframe to produce all of the letters and mailing labels needed. With today's technology, that's quite easy, given how everything's networked together. Even then, it was NOT rocket science. I had learned C, and wanted apply it to the problem of massaging the data into CSV format. We had a mainframe running the CP/CMS timesharing system (an early implementation of virtualization, which is in common use today), and that machine was the only place where we could run a C program.

We had to copy data from one mainframe to the CP/CMS mainframe, and if I recall correctly, we loaded the data into a FOCUS database, then extracted it from there onto the CP/CMS system. The mainframe with the data was not directly accessible from the Windows PC where we would run the mail merge into Word, but the CP/CMS mainframe was.

In CP/CMS we ran a C program that would extract the data and produce a CSV-formatted file, which we then downloaded to the Windows PC.

On the Windows PC, we wrote a non-trivial Basic for Applications script that would choose the appropriate paragraphs to include for each letter, depending on what aid would be received, and apply it to that letter, along with supporting detail. We would run the script to produce a single, long document that could be visually verified for accuracy.

Cindy would run this process, verify the results, and print the letters. What had been a painstaking, error-prone (no fault of Cindy's) two-and-a-half week process became a one-and-a-half day process that produced much more accurate and timely results. What had been a miserable, dreaded, yearly task became just another simple task to be performed.

Not surprisingly, this changed everything for Cindy. Though not technically the most challenging project I've worked on, it is one of the most satisfying projects I've ever done. Why? Well, like many people that get into software development (or many other careers), I want to change the world, and change it for the better. Realistically, I probably won't do that, but I can change my little corner of the world, and this is one project where I did change my little corner for the better. I worked directly with a user, understood the need, met the need, and saw the benefit that I provided, one human being to another. In some small way, one person's life was better because of what I did, and I got to see it happen.

One of the core tenets of today's Agile development processes is continuous, daily, user interaction. I've seen this be effective from 1983 when my software development career began at Washington University with my first user, Neldeane P.

Cat Stove

2008-10-23T19:27:00.000-07:00

Okay, so for all of you out there that thought, "mmmmm, cat," shame on you. They're not that tasty. ;)

I think this guy is the inventor: THE CAT FOOD CAN ALCOHOL STOVE I read his instructions, but then read and followed these to make my stove: SGT Rock's Hiking H.Q. - Cat Stove.

The instructions were easy to follow, and an evening's work resulted in a new cat stove that fits nicely into my backpacking cook pot. My stove weighs 68.5 grams, which is about 2.4 ounces. I could potentially reduce that by trimming the hardware cloth, since its squares are 1/2 inch on a side, and probably 1 inch on a side would do the trick.

Chaco Shoes for Backpacking

2008-10-18T14:54:00.000-07:00

So, now that I'm getting back into backpacking after a long hiatus - like, decades - I've been trying learn from Ray Jardine's practices and apply what works for me. I've already mentioned about making his Tarp and Net-Tent kits.

Another piece of advice from Ray is about footwear. There's probably no more important gear choice you can make than what to put on your feet. Ray recommends hiking in trail running shoes. I agree. On the other hand, I don't push it, because some people genuinely need ankle support. But if you switch to trail running shoes, there are some advantages. One, obviously, is weight. Trail shoes beat boots hands-down. Another is flexibility, and that's very important. Again, trail shoes beat boots. Another is support, and again, shoes beat boots. Another is shock absorption. Again, shoes over boots.

So, about weight. The weight on your feet is more important than the weight on your back. Why? I think it's because you're constantly accelerating and decelerating your feet. Any weight there takes more of an energy toll than the weight on your back, which maintains a relatively constant speed. I'm a big fan of New Balance shoes. I switched to them for daily use some years ago when I found that a pair would last me a couple years instead of one year, like many other brands. I also like to buy Made in the U.S.A when I can, so I can ride my high horse when my job gets outsourced. ;) New Balance gives me more of an opportunity to do that. New Balance shoes are available in 2E width, which I need.

Given that background, I went and bought a pair of MT908's. They're advertised at about 12 ounces, which is close to Ray's ideal maximum of 11 ounces. They're made in China, unfortunately. I wore them on several hike at a parks nearby that sports woods, hills, and trail loops where I can get a 3 to 6 mile hike in fairly easily. I then wore them on an overnighter, about 15 miles. Total mileage on the shoes was probably less than 100 miles. And guess what? The soles started falling apart. Follow the link and look at the picture of the soles. You'll see that there are different colors. All of those different colors are actually pieces that are glued on. Duh. What were they thinking? Those pieces were starting to fall off. I'm going to go to the New Balance store, expecting a fight when I try to return them.

But let me tell you this - New Balance understands customer service like NOBODY else does these days. I take the shoes back to the New Balance store, and the employee there says that this is unusual, and they haven't had that problem with this model. She asks if I would like a total refund, would I like to try a brand-new pair of the same shoe, or would I like to try a different model. So, I like the shoe, it's light and comfortable, so I try a new pair. No charge. I love those guys, and they have a customer for life.

Within seven miles, the new pair is falling apart. That's right, seven miles. Now, I'm not huge. I'm a little over six feet tall, about 195 pounds. Heavy, but the shoes should be able to handle it. Back to the New Balance store. This time, by luck of the draw, I'm talking to the store manager. Same level of customer service. This time, I opt for a full refund.

So, I was disappointed in the shoe. They're made in China, and honestly, they know that Americans today are not like the previous generation. Most of us (not me) are happy to buy crap, and pay good money for it, so they sell us shiny crap at high prices, and we say thank you. Having said this, New Balance customer service is Made in the U.S. of A., the old-fashioned way. I WILL go back to them, largely because of their customer service. New Balance, please make all of your shoes in America. Why not outsource to small towns like Steelville, Missouri? You can still save money over big city labor costs, and those small-town folks remember what quality is. I guarantee it.

So now, what to do for a shoe? I went to one store, and the guy tried to jam Nikes on my foot. Nikes only come in narrow and narrower, and I need a 2E. Moron. I went to another store, and found the Chaco's Men's Redrock.

Here's what's right with the Chacos. Firstly, never in my life have I had a shoe where the arch of the shoe comes up and nestles in the arch of my shoe. I never knew they were supposed to do that! The Chacos do. Wow, arch support. So that's what they meant by arch support. Oooooh.

Second, one-piece soles, stitched to the uppers. That's way shoes are supposed to be made. There's no gluing in shoes.

The soles are some percentage recycled rubber from tires. Good for the environment, and that's a plus. And they've got good lugs for grip on the trail.

Also, all the standard stuff. Reasonable cushioning. Not as good as the NBs, but good. That's good for my knees. Breathable uppers, so that they walk dry after going through a creek. I also moved the laces so that the eyes closest to my toes are not used. This gives me the nice, floppy, barefoot feel up front without losing the heel-hugging feel in the back. The shoe laces could be better, but I might swap out for some of New Balance's bubble laces, which are great.

Okay, now the bad news. First, they're made in China. Not a show-stopper, but I'd like to keep shoe jobs here so that when someone's buying software development, my job stays here. Next, the weight. Chaco doesn't advertise the weight.I wear a men's size 9 in 2E width. My right shoe weighs 463.1 grams, or about 16.34 ounces. The left one weighs 472.2 grams, or about 16.7 ounces. Too heavy to be ideal for backpacking. They are noticeably heavier than the New Balance 908s. Finally, when I asked about the return policy, it's not as generous as NB's. They'd deduct from the refund for wear-and-tear.

The Chacos are for me, at least for the time being. Chaco, here's what I have to say to you. First, open a plant in Steelville, Missouri. Can you tell I love the place? It's not my home town, but it has a special place in my heart, for a variety of reasons. Chaco, if you build a plant there, you will be able to make shoes cheaper than in Colorado. Not as cheap as China perhaps, but it keeps jobs here in the U.S. Second, see if you can make a shoe that's just right for ultralight backpackers. Take the Redrock, reduce the weight by five to six ounces, if possible, while keeping as much ruggedness as possible. And third, think about your return policy. If you're making a quality shoe, and you are in the Redrock, then you can be more generous. See if you can match NB's policy.

Conclusion: I'm going to wear the Chacos for now, because they're the best shoe I've found so far.

Completing the Ray-Way Tarp and Net-Tent

2008-10-16T20:21:00.000-07:00

Well, it's taken us a while, but my Ray-Way Tarp and Net-Tent are now complete, having finished the Net-Tent tonight. My wife Dot did the lion's share of the work - it's a fair bit of sewing, to be sure. It's really been a labor of LOVE on her part, too - she doesn't backpack.

So, every good ultra light backpacker (which I am not, but let's pretend) will want to know the weight. When I weighed the tarp before, I mis-weighed it by counting my 200 gram weights as 100 grams. Oops! Anyway, here are the correct figures.

Component	Advertised weight	My weight
Two-person tarp	16.76 ounces (before sealing)	17.84 ounces (505.8 grams)
8 Stakes		2.87 ounces (81.5 grams)
Net-Tent	about 12 ounces	13.85 ounces (392.5 grams)
TOTAL		34.56 ounces (979.8 grams)

Also consider that with the Net-Tent completed, I no longer need the ground cloth, since the Net-Tent does double-duty, and that saves me 114.1 grams, or 4 ounces.

I added about 10 grams to the weight of the tent by swapping out all the brown flatlines for white cord. Why? Because I found that I absolutely cannot untie the broun flatline when wet.

PenWag and GWT update

2008-10-16T16:17:00.000-07:00

So, I mentioned that I'm using GWT to develop my new web site PenWag. I ran into an interesting problem with GWT, so let's talk about that in case you run into the same problem. When you do GWT development, most of the time spent viewing your app is in "hosted mode," that is, using their custom browser. Your app runs there as a java program. Once you've made some progress, you might deploy it to say, Tomcat, and hit it with a full-fledged browser such as Firefox.

So, I'm tooling along throwing this app together, viewing it in hosted mode, and all looks good. It's looking the way I want it, so I deploy it to Tomcat and view it in Firefox. Hmm, no love. I can see most of the content, but there's a blank space where there's supposed to be a grid. I'll cut to the chase here - after a fair amount of digging, I found the problem in this method in a class used to populate the grid:


 public List<T> getList() {
  return hasNextPage() ? results.subList(0, pageSize) : results;
 }

To my eye, it looks pretty innocuous. But no. Here's where the problem is. Apparently, not sure why, "results.sublist(...)" works fine in hosted mode as Java, but fails when it's compiled to Javascript and viewed with Firefox. Ah, well. The fix was to move this logic to the server side, and just store the desired value that we will return with getList(). So now, the method looks like:


 public List<T> getList() {
  return results;
 }

and all is well.

This was something of a difficult problem to solve. There was no error message to point to the problem, so I used a divide-and-conquer approach to solving the problem. Took out all the code, and the other surrounding components began to show up. Started to introduce pieces until it broke, then narrowed the problem down to sublist. There ya go, hope this helps.

Global Warming, A Skeptic's View

2008-10-15T19:43:00.001-07:00

Well, my friends, I am a Global Warming Skeptic. Is the Earth warming? I though it was, but the more I read, the more the data seems inconclusive. Does that mean I think we should go on polluting the Earth? Well, no. In practical terms, we will of course, continue to pollute. But I'll go out on a limb and say that less pollution is better than more pollution. There are enough valid reasons for this that I don't need your bogus Global Warming reasons to convince me.

After a unfortunately short conversation with a friend of mine who seems quite convinced of Global Warming, I sent him this email: Global Warming Letter It includes a link to the Junk Science article on Global Warming. If you're still one of those folks that think Global Warming's on the verge of destroying the planet, and you can only stomach one Global Warming Skeptic article, by all means make it this one. They've done some good study on the subject.

Now, you may be saying to yourself, "But there's a strong consensus among scientists that Global Warming is real!" There a many ways to achieve consensus. One is through reason, but there are other techniques available. Consider the many countries where "democratic" elections re-install presidents with 100% support. That's right. The Global Warming exaggerators may employ less-than-scientific methods of persuasion to change your mind. They may threaten to make you unemployable if you don't agree with them, effectively excommunicating non-believers from the scientific community. All while they're griping that one Global Warming Exaggerator might have been threatened once, according to a rumor from a credible source. Here's Dr. Cullen's threat. The Exaggerators are now making ties between skeptics and neo-Nazism by referring to skeptics as "deniers." The name-calling wouldn't be there if they had a valid case, would it? Read this one, too: The Real Inconvenient Truth About Global Warming: Skeptics Have Valid Arguments

There are also some interesting articles on Wikipedia. This one is biased towards Global Warming Exaggeration: Global warming controversy. You rarely hear about the List of scientists opposing global warming consensus. Who knew there were scientists that disagree with the consensus?

You might even remember dire predictions that 2006 would be the worst hurricane season ever in the Atlantic, surpassing even 2005, and that this would provide further evidence of global warming. In 2005 there were thirty storms, of which fifteen were hurricanes. In 2006 there were nine storms, of which five were hurricanes. That's right, nine storms. Hardly worth even getting the Hurricane Center all geared up for the season. Is that proof that there is no global warming? Of course not. It might be an indication that the exaggerators couldn't predict Christmas with a calendar, much less the average temperature in 2100.

Reuter's reports that the United States is "the world's largest greenhouse gas polluter accounting for nearly one quarter of all carbon emissions." That much may be true. What they're not telling you is that the U.S.'s "carbon uptake" matches our carbon output. The net effect? Zero. Contrast that with Japan, whose carbon output is seven times their uptake. Wow.

When I'm Not Coding

2008-10-15T18:06:00.000-07:00

Sure, I'm a coder, but like many others, coding's not my only interest. Many coders are fine musicians. Not me, sadly, but there are a couple other hobbies I enjoy.

I've been bowhunting for deer for a long time. Unsuccessfully. And I mean a LONG time. For many years, the land I had access to was thinly populated with very smart deer. I had precious few opportunities. Sometimes I went a few years without even seeing a deer. For the past several years, I've been hunting with a friend whose family has well-populated land. They're nice enough to let me go up there, and I say thankya. Well, last year I got a new bow. A Mathews, actually, and they make a very fine bow. It's very quiet and smooth, not to mention which it's beautiful in terms of craftsmanship and artistry. I had four clean misses last year, all easily make-able shots. This year, I finally got one. It's been a long time coming.

Current Projects

2008-10-15T17:53:00.000-07:00

I have a couple software projects underway that I do outside of work. The first is of interest only to developers of web applications, mostly, and it's called DonsProxy. I didn't name it that, but people started using that name for the original version which was developed over some years at a client site. The new version is a complete re-write, and is available at http://donsproxy.sourceforge.net/. I haven't made any updates for a couple months. The next big task on this is to do more work on the SSL/HTTPS side, and get that working better.

The project I've been active on lately is for a web site with somewhat broader appeal. It's for anyone who likes to write, but for whatever reason isn't diving in to a book project or short stories for some magazine. It's also for readers of those stories, and it can be found at https://penwag.com/. This site is in its early stages, so don't expect much yet. It's just that I wanted to share it while it's under development, so you can see the progress, much like the user on an Agile project would.

My Life with SOA

2008-10-15T17:42:00.000-07:00

I was recently asked to write about my history using SOA technologies. I was asked to write a paragraph. I failed. :) There's a lot to cover, and here it is:

I got my start with service-oriented architectures well before SOA was a buzzword in 1995 at Southwestern Bell Telephone, on a project by the name of Datagate. At the time, there were really no tools or libraries to help with the process, so we built everything in C from the ground up. There are a number of elements that are immediately associated with SOA in today's world, such as SOAP and UDDI. In Datagate, there were corresponding technologies. The exchange protocol that we used was a proprietary format designed to be concise and limit bandwidth use. Today, industry standards such as SOAP exist to allow interoperation and tools development. Most SOA environments rely on a Directory Service to allow client applications to discover active service instances. In today's world, that's typically UDDI, and in Datagate we built our own directory service that complied with the X.500 standards of the time. Current SOA implementations provide services hosted by an application engine such as WebLogic or JBoss. Datagate services ran as independent processes, but were managed by a central Resource Manager that monitored and reported the health state of infrastructure and business services. Eventually, we added a PKI to the system, and again, there were few tools at the time, so much of what we did we had to build ourselves.

One of the keys to success within a SOA environment is that mentality, a way of thinking about services, that is different from building stand-alone applications. SOA provides opportunities for high-availability and scalability that standalone applications cannot provide. But the real power of SOA, the idea that makes it really exciting, is the idea that we can capture complex business rules one time in a service, and re-use that service over and over. If we have a service that captures, for example, the rules that allow a customer to buy a certain stock, we can capture that once in a business service. That business service is then responsible to guard the data. We can test that service to a high standard, and any client that wants to buy a stock will go through that service, using well-tested logic. If we then decide to create a new application that requires the same logic, we don't have to recreate the logic that's already there. We can re-use it. This is important because we already have a service that we've been using, and trust to do the right thing. It's also important because it represents a cost savings to the business. We don't have to re-write, re-test and re-deploy a new implementation. This lets us provide the latest flashy interface, while using the same business logic that we've come to trust over time.

I started at my next employer, Connectria, in 1998, and worked on-site at A. G. Edwards, Inc. for six-and-half years. We did a number of projects there using SOA technologies as they developed into what they are today. My first SOA project there was a project named ClientOne, which delivered a new Broker Workstation to all of the brokers around the United States. We mentored application teams as they wrote thin-client applications that were delivered through the Broker Workstations.

My next big SOA project there was AGEconnect - the project which delivered the website agedwards.com for many years. I was an architect on this project's architecture teams through a couple major iterations. In the first iteration, we delivered a SOA-based architecture using two application engines, Dynamo and Tengah (now WebLogic). In the second iteration, we completely re-architected the framework to be based in WebLogic. Many supporting technologies were required and included in the new architecture, including LDAP, Apache HTTPD, F5's Big-IP appliance, and WSD appliances. We hosted the system on Sun hardware. There were four WebLogic engines each on four Sun machines in the home office, with the same on the failover site. We rolled out with about 80,000 clients on the system, which eventually grew to 300,000 or more.

On my final project at A. G. Edwards, I was the team leader for the BLServer component of their Gateway project. The Gateway project was focused on migrating some key functionality to an external provider. This provider was using some dated technologies to deliver content over the network, specifically, proprietary protocols. In addition, this external client had a nightly four-hour update window during which we needed to continue to accept updates. BLServer was one of a collection of services designed to wrap those proprietary services with standards-compliant services. BLServer included at its core a clustered message-driven web service that queued requests into JMS queues (the provider was Tibco) and then applied them in order during the external provider's uptime. The throughput requirements (about 300,000 transactions per 8-hour day) and the ordering requirements that exist for updating clients' financial accounts made this a particularly interesting and challenging project.

I started at my present employer, Asynchrony Solutions, Inc., in 2004, where I have also been a member of SOA projects. On the first I was the team leader on a project to deliver a system that could both push documents near to point-of-use around the world, and to pull data from disparate systems and deliver it around the world. That is, a federated system. This system employed Model 2 servlets hosted under Tomcat and JMS to meet the demanding performance requirements on networks that are often slow or unreliable. The JMS provider in this case was originally ActiveMQ, but we replaced it with JORAM when ActiveMQ was found to have difficulty reconnecting after network failures.

The next was to demonstrate to a client in the healthcare industry how SOA might work for them. We delivered a pilot application that allowed clients to view their prescriptions, appointments, doctors and other information using a web browser. It allowed healthcare providers to review patient information using a browser on workstations, iPods, and Blackberries. This application demonstrated the capabilities of both SOAP-based and RESTful services, and both Java and Ruby clients.

The next project made available an opensource SOA stack for the Army that developers could use to implement SOA solutions, then deploy them within a production-level environment using primarily commercial implementations of SOA stack components, such as WebLogic and Systinet. The developers' stack included a service compliance tool that would alert them to possible governance violations before deployment, and a stack compliance tool that would allow them to replace components of the SOA stack and verify that they delivered the same functionality.

Hadoop

2008-10-15T15:31:00.000-07:00

I'm now on my second project where we're using Hadoop/Hbase and the Google Web Toolkit, both of which I'm happy to get a chance to use at work. Gives me an excuse to play, er, work with tools I'd use at home anyway.

The first Hadoop project was for a client in the health industry. They needed to provide doctors with easy access to DICOM images using web browsers. This required a conversion process from DICOMs that browsers cannot display, in to browser-friendly formats such as JPG and AVI files. We used Hadoop to manage the conversion of large numbers of images from DICOM to JPG and to created varying sizes of images for thumbnails and such. We used Hbase to store the images. The use of Hbase on top of Hadoop greatly simplified the original approach where we stored the images on HDFS. Finally, we developed a snazzy GWT front-end for the doctors so that they could upload and manage images on the system.

The exposure to GWT was a great experience, and I was so impressed with it that I decided to use it for a web site I'm developing, PenWag. We use Agile-XP practices at work, and I try to follow those to the extent I can at home. So, yes, I have story cards, I have a continuously-demo-able product. PenWag is under development, go have a look at its current state. I'm just getting started, but thought I'd like to make it available all along the way.

The current Hadoop project is an R&D effort. Hadoop is relatively new to most companies. I'm starting to hear about opportunities to do Hadoop development on both coasts of the U.S. There are fewer such opportunities in the Midwest, but it's coming this way. Our company is preparing to be ready when it gets here. We already have some experience, but we're exploring the technology more thoroughly, because there's a lot we can do for our clients with this technology. Hadoop really, really simplifies the whole question of how to scale my app. Now, any problem I can express in MapReduce terms can be deployed to Hadoop. We can start with four or five commodity boxes, but could conceivably scale to 2,000 or 10,000 boxes if that's what the customer wanted. Assuming, you know, that they had a place for 10,000 Linux machines, and a way to cool them.