Archive for May 2006

If civil engineers built bridges like we built software…

May 29, 2006

Oracle’s Chief Security Officer made a remarkably silly comment in a talk at WWW2006. CNet covered it here.

First wrong thing: We know how to build bug-free software. If you’ve flown a Boeing airplane built in the last ten or fifteen years, you’ve trusted your life to a software system. The nav systems have been software driven even longer, and all the newer planes are fly-by-wire. We now how to build high-reliability, bug-free software. It’s just expensive.

So, what would it mean to build bridges like software? Well, you’d be able to automatically assemble a new lane overnight if you needed more capacity. Anybody driving down the road would be able to get their best route around traffic jams by pinging the road. In fact, the road would probably be doing most of the driving, so you could work, or talk on the phone, or … whatever else you wanted to do.

If you need another bridge, you can probably reuse the bridge you built last week, for rather less cost. Bridges are kindof an interesting problem, so you can probably get pretty good people to work on them. They are pretty amenable to agile methods: You can put the engineer on site with the customer, maybe try a couple of approaches and put just one kind of traffic across first. It’s probably best to start with just trucks, since that gives you an early payoff in commercial traffic, and anything that will hold a truck will probably do okay with ordinary cars. Trains will need some additional infrastructure, so do that later.

So building bridges the way we build software might not be so bad.

Now, turn it around. What if you built software the way the civil guys build bridges? First thing, it would be expensive. Bridges are all purpose-built – what the Brits call “bespoke” software. No trotting down to CompUSA to pick up a copy of your favorite game. First you need to get a game development permit, do a “sight” plan, (pun intended) and get the plan reviewed. Maybe for games under six minutes long (like fences under six feet high) you don’t need a permit. You need to have your development plan publicly reviewed, even on private computers. There may be public hearings if you’re doing something unusual. There will be endless inspections, and government inspectors may make you redo things if you use a technique they haven’t seen before, and approved in advance.

The problem isn’t a lack of regulation, or a lack of attention to detail on the part of developers. Rather, there is a basic unwillingness to spend the money required to “harden” applications and servers against ill intent. It’s not cheap, and everybody thinks that somebody else should pay for it.

Which is fine. If you want Windows XP (or Vista, whenever that happens) hardened against all threats, you need to do two things. First, expect to pay about ten to twenty times as much for the license. Second, you’ll need several weeks of training in order to learn to configure the damn thing.

Me, I’d like to sell you the training. I know some very good trainers, and I bet I can get repeat business.



How we work, and for whom, part 2

May 25, 2006

Alberto and I are splitting the DMS CE duties while ESS recruits a replacement for Warren. The main thing we hope we’ll be doing is tracking progress against the schedule, but inevitably things will come up. The interesting question will be to understand what happens when things to come up.

I understand that people find value in the CE role. The CE is our chance to coordinate architectures and implementations across missions, and get consistency across subsystems. I believe there might be even more value if each CE had a narrower focus. The number of CEs was rather arbitrary, so why have three? Why not have six or seven? Or even eight? If we had a strong-project-lead model, and treat each build as a separate project, the CEs would effectively direct both the work and the people, and the Branch Manager would have a limited, administrative role. (To the point that we might only need one.)

Matt talks about the importance of teams. For the big projects, we want to contribute team members that have a good understanding of products and practices, informed by their experience on previous efforts. For little projects, and for maintenance efforts, we want sustaining teams that work in sustainable ways, so the project managers should expect to be “stuck with” the products they build. Put baldly, the CE needs much of the Branch Manager role to be effective.

Peg talks about organizing around the work: Look at where work gets done, and create an organization that supports that, recognizing that different work has different needs, and may need different organizations. Again, what ESS works on is products, so we should organize around products. When we contribute people to cross-functional teams, we should pick them based on their understanding of existing products, recognizing that they may be asked to build an entirely different product.

My notion of the Product Engineer is a CE with staff responsibilities. It is, in a sense, “back to the future” in the sense that ESS used to be organized around product teams. It is, however, informed by the experience of the reorg. We can have larger branches, and have products in rational groups. We can matrix people, though we’ve learned that we need to not split them so finely.

We’re going through an exercise of understanding what works and what doesn’t about the current organization. It may be that I’ve prejudged the outcome, so I’m anxious to see where that exercise takes us. What I don’t accept is that we can’t change roles simply because we’ve invested effort in getting acceptance of the CE role. That’s a classic case of the sunk cost fallacy.


How we work, and for whom.

May 24, 2006

It’s often said, among people who think seriously about how ESS works, that we are pathologically incapable of building the wrong system just because of specification. That’s true, as far as it goes. We certainly can build the wrong system, and to some extent we have in the past. We probably will again. But we don’t do it because of specifications. We do it from a combination of misunderstanding and pig-headishness.

The old “Tom’s two rules” are

  1. The customer is always right
  2. Users are stupid

Again, both are true, as far as they go.

The problem with users is that if there is a way to misuse your software, users will find it. So there’s a strong need to make sure that if user do misuse your software, the result isn’t a disaster. (Annoying, however, is fine.) But you can’t assume that because the user can do stupid things, they are stupid people. They in fact do know what they want to do, even if they don’t always have the best notions of how to simplify, improve, and accellerate the work they do. If you believe your users are simply stupid, you will not do a good analysis of their problem, and you will almost certainly do the wrong thing.

The problem with customers is that they usually aren’t users. The people who buy the software (in our case, Missions) are almost never stuck with actually using the software. When dollars are tight, as they are now, they tend to think in terms of what they can afford in the next year, and lose track of what the people using the systems will need in three or five or ten years. Not because they are short-sighted, but because their vision is reasonably constrained by what they can afford, and what they know themselves.

I wrote recently that, regardless of the project that builds ground system software, ESS gets left holding the bag for that software. Way too often, our users also get left holding the bag. Unlike us, they can’t “look inside the bag” to see why things have gone wrong. Most of the time we work closely with our internal users, and listen carefully to our external users. The problem we have, for the moment, is that we have very few real users other than for HST. Even for HST, it is sometimes hard to find the right users when the Mission Office takes a greater interest.

ESS has for many years now been about sustaining HST. It will continue to serve that role for a while yet, but the future is not HST. The future is MAST, and SAGE, GALEX and Kepler, and hopefully most of all JWST. We need to figure out how to balance the customer and user both now, and for the future.


Institutional change

May 22, 2006

Matt announced today that the changes to the organization are complete for now – no more staff cuts.

The Office of Technology has been eliminated, and the separate Program Management activity has been merged into BRC. The three Mission Heads now report directly to the Director, and the Divisions report to the Deputy Director. Matt will talk more about the changes and what they mean on June 6th.

For ESS, there are no immediate changes. We are thinking about the organization, and how ESS works, but our focus is about how we relate to the Missions and the other Divisions.

For CPT, there is essentially no change in organization, but there is change in focus. Doris still leads CPT, but the list of projects that CPT must execute is being reviewed. As a result, ESS’s agreement to loan a couple of half-people to do project management is on hold. That work could pick up quickly, but it depends on both how long it takes to do the reprioritization, and the results of that exercise.

Warren Miller is now also admitting that he is leaving ESS. He isn’t sure how this will work exactly, but DMS is short a CE. We still have a CE position, and it’s likely that we’ll see something posted, at least internally, in the near future.

I am concerned about the lack of urgency I feel from the ESS Division Office to deal with the internal and external changes. We seem to be hung up in talking about talking, and not even talking about changing. I think we need change, and I think we need leadership to achieve change. We face declining budgets, a shrinking staff, and dissatisfaction from the people who pay for us. If that doesn’t call for change, I don’t know what does.


FiOS is coming

May 21, 2006

I spent the day finishing up the desk in the living room. Now all that’s left is trim pieces, and the living room will finally be done. Only what, 18 months?

The outside fiber cable was installed today for my new FiOS service. The service guy is coming Tuesday. I’d like to have CAT-5 already run, but if not, I’ll let the Verizon guy run it. Either way, by Tuesday night I should have better, faster, cheaper phone, internet and (best of all) TV service.

No more cable Nazis, as Mary puts it.


What's different about software developers

May 19, 2006

I spent much of today thinking and writing about what makes ESS different from other parts of the Institute. In particular, I wondered why the original plan, which had people moving from an HST-directed part of ESS to a JWST-directed part of ESS.

A few years ago, it seemed likely that ESS would fission into two or three parts to support the three Mission Offices. I don’t believe that will actually work, chiefly because we’ve already observed that the systems for JWST are the same as those being used by HST. I see this with Kepler, too, and I believe Kepler is doing it incorrectly. They’ve branched the DMS systems so that Kepler and HST will evolve separately. That means that when we find generic problems with HST (like needing to change something for a language upgrade) we will have to first decide, and then do, the same same update to Kepler, but separately. That will, inevitably, cost more.

If we really wanted to do the Mission-directed work separately, we should have a CE for each mission, or have people matrixed to each mission. We don’t because most of the time the Missions can’t afford whole bodies, and because most of our work is maintenance work. We have few projects, because at the end of the project, somebody is left holding the bag. Usually ESS.

So Joe’s challenge to find a new organization isi the right challenge. But I wonder if it comes too late for anybody to listen.

A bit of History

May 18, 2006

I wasn’t working today, but took the day off to chaperone Teela’s class trip to Historic Saint Mary’s City. It’s a fairly cool place to visit, but I wouldn’t want to live there.

St. Mary’s had two main crops: Tobacco for cash, and corn to eat. Tobacco was both the main crop and the main currency. Getting it grown was mostly the work of indentured servants, and being indentured was both a tough way to make a living and an effective way to get an education. A boy who spent anywhere from four to ten years indenture learned all about growing tobacco, and at the end of his indenture got cash, tools, and a land grant. If you paid any attention to what you were doing (and survived illness, injury, cold and possibly famine) you could grow tobacco and at least your children would be wealthy.

Without local sources of metal, and a very limited middle class, most manufactured goods (and all metals) were imported. The stuff came over by ship, but very, very small ships. They had a replica of a fast trader, and it was a tiny place to spend a couple of months crossing the Atlantic, especially in winter. This was pre-industrial-revolution, of course, so doing simple things, like raising a 350 pound anchor, was hard work.

I don’t think we do hard work anymore. We do difficult things that require a lot of thought, but not a lot of real work, in the personal or physical sense. I don’t quite believe what I do, sitting in front of a keyboard, qualifies as “hard work.”