Wednesday, May 20, 2009
DonsProxy on github
I've been thinking about getting DonsProxy out on github for a bit now, but needed to break out the good stuff from the slough. So, it's finally out there. My next DonsProxy task is to get the SSL part working better - maybe by injecting Mina. Still trying some things out. You can clone DonsProxy from github at http://github.com/DonBranson/DonsProxy/
Thursday, May 7, 2009
Favorite Project Series - Datagate (14 Years of SOA)
SOA is one of the big buzzwords these days, and a lot of people have started doing SOA for the first time in the past few years. I have some good memories of the "good old days of SOA," and now it seems like everybody's using the groundwork that was laid in the 80s and 90s. Now, people can just go download Weblogic and Systinet or JBoss, and and the various pieces, build a system using those frameworks, and they're "doing SOA." There are so many powerful tools; the IDEs such as Eclipse and Intellij do a lot of the tedious work for you.
My first introduction to SOA was in 1995 on a project called Datagate at Southwestern Bell. I came on the project when it was about six months old. All of the things that people take for granted now, that they use and it "just works" (or not), we had to build our own. There was no app engine, no UDDI, no SOAP. The things that people download and use today, we built and used back then. If you're a computer geek, this is the project of a lifetime. It was all message-based (that's a MOM for all you acronymophiles). Now I can look around and see SOA everywhere, but we were, in our own way, pioneers of the technology, because we did things no one had ever done before. Sure, not everything we did was new - Tuxedo had been around, so MOMs weren't exactly new.
What we did in the mid-90s that was new was to make the technology more accessible. It was hard to write Tuxedo services back then. We made it accessible to every C, VB, and PowerBuilder coder out there. There are some ways in which today's technology is better. But, what we built was so smokin' fast, and there's nothing that SOA folks are doing today that even comes close to the performance we delivered on those old, slow machines. We had a great, usable middleware product that was the framework for developers to construct clients and reusable services on a variety of platforms - about 12 unixes, MVS running on IBM 390s, Windows 3.0, Windows NT, and Tandem. It's one of those experiences that you looked back on, and think, we did it before most anybody else. What people today use, we built.
I wasn't one of the visionaries for that group. I was brought in to develop an interface so that VB and PB programmers could write clients on top of our C framework. In the end, I help design and architect that, the Directory Service Replicator, and the Dashboard for this system, and we did good work.
There was no SOAP back then. The Datagate team developed its own protocol. It was message-based, so was much more performant, scalable, and simpler than the RPC-style calls are today. Yes, even the message-driven services are RPC under the covers, not the true MOMs like what we built. A true MOM lets you fire off multiple messages to services without waiting for the ack that the message was received, then process the responses as they come back in. All this with one thread, not multiple threads that RPC pushes you towards for this kind of processing.
There was no UDDI then for service lookup back then. There was no LDAP, either. We built our own distributed, replicated X.500 directory service. Infrastruture and business services registered and reserved leases dynamically. If a service went down, its service entry went away, and clients would call one of the other instances that was still registered. And of course, it was fast - 10s of millis to do a service lookup, not 100s or 1000s like you see with UDDI. Smokin' fast.
I architected and led the development of the Directory Replication Service along with my friend Dave C. This cell-based, scalable solution allowed us to run multiple Directory Services, and replicate data between them in a scalable, available fashion.
We used XDR for marshalling. That's one piece we did download. I lead the charge to unit test the whole thing - and found only one bug. There was an endian issue on that affected one of the platforms, and we found it and fixed it.
There were no application engines. What we built was a resource manager that would control service lifecycle and heartbeat services to quickly detect outages.
There was no PKI, or even SSL. We bought licenses for the encryption libraries, then designed and built our own secure message protocol (with mutual authentication where needed), plus the infrastructure to support it. Certificate Revocation Service, Certificate Authentication Service, and al the APIs for services to use to load their certs, encrypt their messages. We built the services and the GUIs so that administrators could manage the PKI we built.
There was monitoring software available - Tivoli - but it was far more than our budget would allow. We built our own Dashboard that allowed us to closely monitor infrastructure and business services running over much of the United States, including Missouri, Texas, California, and more.
We did training to teach developers how to build reusable business in C, and how to build clients in C, in VB, and in PowerBuilder. I taught the VB and PowerBuilder classes.
Our team consisted of three subteams - Infrastructure, which was the team I was on, the Service Writers team, and System Adminstration, which handled the care and feeding of production systems.
At A.G. Edwards, we leveraged SOA frameworks for 6-1/2 years as part of a broker workstation, to develop the pretty scalable agedwards.com (which, last I heard some years back, had about 300,000 users signed up). I was a member of the Lead Architect Team that designed and developed agedwards.com. We used SOA for some other pretty cool stuff. One of my favorites was the BLServer, that used a modified version of what is now called the "Competing Consumers" pattern in a highly-available cluster of message-based services that self-allocated using leases to process roughly 1.5 million messages a day for about 4-1/2 years before it was mothballed in February of this year.
The industry has come a long way since then. Like I said, what we built, you can now go download, and the SOA business has experience explosive growth. After 2-1/2 years at Bell on the Datagate project, I did 6-1/2 years of SOA work at A.G. Edwards, continuing to architect systems using new SOA technologies as they emerged - Tengah, (now Weblogic) Dynamo, Novell's LDAP server, SiteMinder, WSDs, Big/IP, and various JMS providers. Some of this stuff I had more exposure to than others, but that gives you an idea of how things have changed over the past 14 years. Now there's CXF and JAXB and other exciting new tools coming along.
Every Tom, Dick and Harry does SOA now. Years after that work at Southwestern Bell, a small company hired a "SOA expert" (after I was already there!). This poor fellow didn't even know what LDAP is. When I described how it could be used for service lookup, he said you can't use it for that - it's for authentication. I explained to him that it's just a database that's optimized for reads. You can use it for anything that falls into that category. Now, he's an expert, downloading what other people have built and putting it all together. One boss at that company tried to get me to join a project where I would have been writing architecture documents all day for six months, and building nothing. His angle? That the SOA experience would look good on my resume. I kid you not.
On Datagate, we built SOA sooner, faster, more scalable, and at least in some ways better than what exists today. The primary advantage I see in today's world is standards. Almost everything is standards-based, and that means vendors can provide tools to do many of the things we used vi for. It's a change for the better, in most every way, because SOA is much more accessible today that it was in 1995. Plus, now it's an acronym. ;)
My first introduction to SOA was in 1995 on a project called Datagate at Southwestern Bell. I came on the project when it was about six months old. All of the things that people take for granted now, that they use and it "just works" (or not), we had to build our own. There was no app engine, no UDDI, no SOAP. The things that people download and use today, we built and used back then. If you're a computer geek, this is the project of a lifetime. It was all message-based (that's a MOM for all you acronymophiles). Now I can look around and see SOA everywhere, but we were, in our own way, pioneers of the technology, because we did things no one had ever done before. Sure, not everything we did was new - Tuxedo had been around, so MOMs weren't exactly new.
What we did in the mid-90s that was new was to make the technology more accessible. It was hard to write Tuxedo services back then. We made it accessible to every C, VB, and PowerBuilder coder out there. There are some ways in which today's technology is better. But, what we built was so smokin' fast, and there's nothing that SOA folks are doing today that even comes close to the performance we delivered on those old, slow machines. We had a great, usable middleware product that was the framework for developers to construct clients and reusable services on a variety of platforms - about 12 unixes, MVS running on IBM 390s, Windows 3.0, Windows NT, and Tandem. It's one of those experiences that you looked back on, and think, we did it before most anybody else. What people today use, we built.
I wasn't one of the visionaries for that group. I was brought in to develop an interface so that VB and PB programmers could write clients on top of our C framework. In the end, I help design and architect that, the Directory Service Replicator, and the Dashboard for this system, and we did good work.
There was no SOAP back then. The Datagate team developed its own protocol. It was message-based, so was much more performant, scalable, and simpler than the RPC-style calls are today. Yes, even the message-driven services are RPC under the covers, not the true MOMs like what we built. A true MOM lets you fire off multiple messages to services without waiting for the ack that the message was received, then process the responses as they come back in. All this with one thread, not multiple threads that RPC pushes you towards for this kind of processing.
There was no UDDI then for service lookup back then. There was no LDAP, either. We built our own distributed, replicated X.500 directory service. Infrastruture and business services registered and reserved leases dynamically. If a service went down, its service entry went away, and clients would call one of the other instances that was still registered. And of course, it was fast - 10s of millis to do a service lookup, not 100s or 1000s like you see with UDDI. Smokin' fast.
I architected and led the development of the Directory Replication Service along with my friend Dave C. This cell-based, scalable solution allowed us to run multiple Directory Services, and replicate data between them in a scalable, available fashion.
We used XDR for marshalling. That's one piece we did download. I lead the charge to unit test the whole thing - and found only one bug. There was an endian issue on that affected one of the platforms, and we found it and fixed it.
There were no application engines. What we built was a resource manager that would control service lifecycle and heartbeat services to quickly detect outages.
There was no PKI, or even SSL. We bought licenses for the encryption libraries, then designed and built our own secure message protocol (with mutual authentication where needed), plus the infrastructure to support it. Certificate Revocation Service, Certificate Authentication Service, and al the APIs for services to use to load their certs, encrypt their messages. We built the services and the GUIs so that administrators could manage the PKI we built.
There was monitoring software available - Tivoli - but it was far more than our budget would allow. We built our own Dashboard that allowed us to closely monitor infrastructure and business services running over much of the United States, including Missouri, Texas, California, and more.
We did training to teach developers how to build reusable business in C, and how to build clients in C, in VB, and in PowerBuilder. I taught the VB and PowerBuilder classes.
Our team consisted of three subteams - Infrastructure, which was the team I was on, the Service Writers team, and System Adminstration, which handled the care and feeding of production systems.
At A.G. Edwards, we leveraged SOA frameworks for 6-1/2 years as part of a broker workstation, to develop the pretty scalable agedwards.com (which, last I heard some years back, had about 300,000 users signed up). I was a member of the Lead Architect Team that designed and developed agedwards.com. We used SOA for some other pretty cool stuff. One of my favorites was the BLServer, that used a modified version of what is now called the "Competing Consumers" pattern in a highly-available cluster of message-based services that self-allocated using leases to process roughly 1.5 million messages a day for about 4-1/2 years before it was mothballed in February of this year.
The industry has come a long way since then. Like I said, what we built, you can now go download, and the SOA business has experience explosive growth. After 2-1/2 years at Bell on the Datagate project, I did 6-1/2 years of SOA work at A.G. Edwards, continuing to architect systems using new SOA technologies as they emerged - Tengah, (now Weblogic) Dynamo, Novell's LDAP server, SiteMinder, WSDs, Big/IP, and various JMS providers. Some of this stuff I had more exposure to than others, but that gives you an idea of how things have changed over the past 14 years. Now there's CXF and JAXB and other exciting new tools coming along.
Every Tom, Dick and Harry does SOA now. Years after that work at Southwestern Bell, a small company hired a "SOA expert" (after I was already there!). This poor fellow didn't even know what LDAP is. When I described how it could be used for service lookup, he said you can't use it for that - it's for authentication. I explained to him that it's just a database that's optimized for reads. You can use it for anything that falls into that category. Now, he's an expert, downloading what other people have built and putting it all together. One boss at that company tried to get me to join a project where I would have been writing architecture documents all day for six months, and building nothing. His angle? That the SOA experience would look good on my resume. I kid you not.
On Datagate, we built SOA sooner, faster, more scalable, and at least in some ways better than what exists today. The primary advantage I see in today's world is standards. Almost everything is standards-based, and that means vendors can provide tools to do many of the things we used vi for. It's a change for the better, in most every way, because SOA is much more accessible today that it was in 1995. Plus, now it's an acronym. ;)
Tuesday, May 5, 2009
Migration from Subversion to Git
I'm migrating from Subversion to Git. It's about time, of course - decentralized SCMs have some significant advantages over the old-style SCMS such as CVS, Clearcase, Subversion and the like. I looked at Bazaar, which touts "Bazaar is a distributed version control system that Just Works." Sadly, I couldn't get it to work. It needed a lot of python modules that I, not being a python guy, don't really know where to get. Found some, not others. Funny thing is, Git "just works." Sure, had some hiccups with converting from Subversion before realizing that I had made an incorrect assumption about the repository structure in one case. Looks cool, though, especially when you throw github in the mix. Good stuff.
Recommended reading if you're thinking about converting from your current SCM to something new, whatever that may be: http://whygitisbetterthanx.com/
Recommended reading if you're thinking about converting from your current SCM to something new, whatever that may be: http://whygitisbetterthanx.com/
Saturday, March 28, 2009
Tell Your Story!
There's a web site that I'm building, penwag.com, that lets me enjoy two of my favorite interests: software development and story telling.
It lets me enjoy software development in that I'm using two technologies that I will gain experience with: the Google Web Toolkit (GWT) and Hibernate. The GWT is a great tool from Google that provides a framework for building AJAX-style applications. The framework includes the necessary pieces for client-side and server-side development, and manages the communication between the layers. Add in Hibernate and a data layer, and you have everything you need for easily building AJAX web sites.
Then there's the story-telling side. When I was very little, we visited my great-uncle Henry Shoop, who was my father's mother's brother. He was born near Cherryville, Missouri, but his family moved to California when he was six. Uncle Henry was a friend, a beekeeper, and one of the great storytellers, and I hung on his every word as he instilled in me a love for a well-told story. He was one of the best.
Another favorite story-teller of mine was Earl Halbert, a family friend. You can see his picture here from National Geographic Magazine. He's the one on the left in the overalls. Earl was born in 1910 in rural Missouri, and lived through the Great Depression, Prohibition and numerous other hard times. He knew my great-grandfather Thomas Jefferson Branson, and had a few good stories to share about him as well.
Both of these men, shall we say, infected me with a love of a story told friend-to-friend, perhaps while sipping some sassafras tea, or around a campfire, or just hanging out in the living room. I miss both of them not just for their story-telling and their wit, but for their character as human beings.
Certainly the art of a well-told story has not been lost, but is changing over time. How many people nowadays have had the chance to hear stories of hard times in the Great Depression, friends lost, good times, and funny times, from "way back when"? I'd hate to lose those stories. So join me, won't you, and let's tell each other those stories, like we're sitting over a beer or around a campfire, and let's enjoy a good laugh together, savor tales of clever dealings with shysters, and the occasional twist ending.
Just head out to penwag.com. You're getting in early at a site that will someday, hopefully, have many stories that we have shared with each other. But it's your stories that will make the site great. If you have even one story to share, come on out and put it up there.
This site is in its early stages, but everything you need to add a story and read others' stories is already there. Since it's in its early stages, there's still a lot of work that I need to do add features. Some are pretty obvious - you can create a story, but can't edit the story to make corrections yet. I have a long list of features that I will add as the site grows. But the feature you want may not be on my list! So, click on the Contact Support link, or just send an email to support@penwag.com, and give me a shout. Tell me what feature you want, and I'll add it to my list.
Let me reiterate - this site's success depends on people like you who have a story to tell. As you add your stories, others will visit the site just to read a great story. The stories are short. Don't feel like you have to write a novel. Just write it how you would share it with your friends over a beer.
Now, to get started, you have to register. For simplicity's sake, your email address is your user id, and you need to make up a password. Registration is quick and easy, and your email address will not be shared. If you still have questions or concerns, or if the site doesn't work right for you, use the Contact Support link and let me know!
Thanks, and hope to see you out there. Come tell your story!
It lets me enjoy software development in that I'm using two technologies that I will gain experience with: the Google Web Toolkit (GWT) and Hibernate. The GWT is a great tool from Google that provides a framework for building AJAX-style applications. The framework includes the necessary pieces for client-side and server-side development, and manages the communication between the layers. Add in Hibernate and a data layer, and you have everything you need for easily building AJAX web sites.
Then there's the story-telling side. When I was very little, we visited my great-uncle Henry Shoop, who was my father's mother's brother. He was born near Cherryville, Missouri, but his family moved to California when he was six. Uncle Henry was a friend, a beekeeper, and one of the great storytellers, and I hung on his every word as he instilled in me a love for a well-told story. He was one of the best.
Another favorite story-teller of mine was Earl Halbert, a family friend. You can see his picture here from National Geographic Magazine. He's the one on the left in the overalls. Earl was born in 1910 in rural Missouri, and lived through the Great Depression, Prohibition and numerous other hard times. He knew my great-grandfather Thomas Jefferson Branson, and had a few good stories to share about him as well.
Both of these men, shall we say, infected me with a love of a story told friend-to-friend, perhaps while sipping some sassafras tea, or around a campfire, or just hanging out in the living room. I miss both of them not just for their story-telling and their wit, but for their character as human beings.
Certainly the art of a well-told story has not been lost, but is changing over time. How many people nowadays have had the chance to hear stories of hard times in the Great Depression, friends lost, good times, and funny times, from "way back when"? I'd hate to lose those stories. So join me, won't you, and let's tell each other those stories, like we're sitting over a beer or around a campfire, and let's enjoy a good laugh together, savor tales of clever dealings with shysters, and the occasional twist ending.
Just head out to penwag.com. You're getting in early at a site that will someday, hopefully, have many stories that we have shared with each other. But it's your stories that will make the site great. If you have even one story to share, come on out and put it up there.
This site is in its early stages, but everything you need to add a story and read others' stories is already there. Since it's in its early stages, there's still a lot of work that I need to do add features. Some are pretty obvious - you can create a story, but can't edit the story to make corrections yet. I have a long list of features that I will add as the site grows. But the feature you want may not be on my list! So, click on the Contact Support link, or just send an email to support@penwag.com, and give me a shout. Tell me what feature you want, and I'll add it to my list.
Let me reiterate - this site's success depends on people like you who have a story to tell. As you add your stories, others will visit the site just to read a great story. The stories are short. Don't feel like you have to write a novel. Just write it how you would share it with your friends over a beer.
Now, to get started, you have to register. For simplicity's sake, your email address is your user id, and you need to make up a password. Registration is quick and easy, and your email address will not be shared. If you still have questions or concerns, or if the site doesn't work right for you, use the Contact Support link and let me know!
Thanks, and hope to see you out there. Come tell your story!
Wednesday, March 25, 2009
OT Trip Report - Bell Mountain to Council Bluff Lake
My friend James and I hiked Council Bluff Lake to Bell Mountain (reversed) March 22 and 23, in about 24 hours. It was our first hike of the year, and left me a little sore.
The view from Bell Mountain is a must-see, and is easy to get to on foot. It's pretty level all the way from the North Bell Trailhead. It'd make a fine afternoon hike, out and back.
Once you get past that, the rest of the trail has some pretty good hills, by our Midwest standards. There are long level parts along ridge-tops and in creek bottoms, but you'll have some climbs in the 200-400 foot range. There are some beautiful creeks along the way, too, so finding water was not a problem for us. Joes Creek is big enough that in wetter weather, it is probably hard to cross.
We heard from other hikers that there are feral pigs in the area, so we hung our food out of there reach overnight. We really saw very little animal life - a couple squirrels and birds along the whole trip. We saw lots of deer scat from last fall - many were full of persimmon seeds.
The view from Bell Mountain is a must-see, and is easy to get to on foot. It's pretty level all the way from the North Bell Trailhead. It'd make a fine afternoon hike, out and back.
Once you get past that, the rest of the trail has some pretty good hills, by our Midwest standards. There are long level parts along ridge-tops and in creek bottoms, but you'll have some climbs in the 200-400 foot range. There are some beautiful creeks along the way, too, so finding water was not a problem for us. Joes Creek is big enough that in wetter weather, it is probably hard to cross.
We heard from other hikers that there are feral pigs in the area, so we hung our food out of there reach overnight. We really saw very little animal life - a couple squirrels and birds along the whole trip. We saw lots of deer scat from last fall - many were full of persimmon seeds.
Tuesday, March 24, 2009
AIG Bonus Scandal
Okay, I haven't been political yet, but here goes...
I've been thinking about the whole AIG bonus thing. Correct me if I'm wrong, but here's the way it looks to me. Congress rushed through this legislation. It's fairly complicated, and they were in a hurry to get this thing done. It seems like you can pick two of: complicated, fast, and right. They picked all three, and they screwed it up. The issue of bonuses was in the legislation in black and white, and they missed it because they voted on something without taking enough time to read it. To me, it really seems that simple.
Now as far as AIG's role, maybe the folks deserved the bonuses, maybe not. I've heard the argument that things would have been a lot worse without the work of the executives that got the bonuses. That may or may not be true. It's a real possibility, but without knowing the inside info, I can't say. But it seems clear that AIG was contractually obligated to pay those bonuses, with Congress' backing per the legislation. If they didn't pay the bonuses, they could have been sued, and lost the bonus money plus punitive damages and court costs. And they wouldn't have had a leg to stand on in court, so they probably would have lost.
AIG was between a rock and a hard place. Congress put the rock there, and now they're whining about it. Congress should man up (if there are any there) and say "We screwed up. We know what we did wrong this time, and we won't make that mistake next time." Since they're not owning the problem, they'll probably screw it up next time, too.
Don't misunderstand me - I am not defending AIG. But Congress is pretty quick to shift the blame here, because they know if we stop and think about it, we'll realize what idiots we've elected.
Now, as complicated as the bailout is, it pales in comparison to what's coming down the pike with health care. If they can't even get bailouts right, can they be trusted to get health care right? At this point, I don't think they collectively have the skills to lance a blister, much less to do their jobs well.
I've been thinking about the whole AIG bonus thing. Correct me if I'm wrong, but here's the way it looks to me. Congress rushed through this legislation. It's fairly complicated, and they were in a hurry to get this thing done. It seems like you can pick two of: complicated, fast, and right. They picked all three, and they screwed it up. The issue of bonuses was in the legislation in black and white, and they missed it because they voted on something without taking enough time to read it. To me, it really seems that simple.
Now as far as AIG's role, maybe the folks deserved the bonuses, maybe not. I've heard the argument that things would have been a lot worse without the work of the executives that got the bonuses. That may or may not be true. It's a real possibility, but without knowing the inside info, I can't say. But it seems clear that AIG was contractually obligated to pay those bonuses, with Congress' backing per the legislation. If they didn't pay the bonuses, they could have been sued, and lost the bonus money plus punitive damages and court costs. And they wouldn't have had a leg to stand on in court, so they probably would have lost.
AIG was between a rock and a hard place. Congress put the rock there, and now they're whining about it. Congress should man up (if there are any there) and say "We screwed up. We know what we did wrong this time, and we won't make that mistake next time." Since they're not owning the problem, they'll probably screw it up next time, too.
Don't misunderstand me - I am not defending AIG. But Congress is pretty quick to shift the blame here, because they know if we stop and think about it, we'll realize what idiots we've elected.
Now, as complicated as the bailout is, it pales in comparison to what's coming down the pike with health care. If they can't even get bailouts right, can they be trusted to get health care right? At this point, I don't think they collectively have the skills to lance a blister, much less to do their jobs well.
Subscribe to:
Posts (Atom)