Where's Waldo?

Now that edX has been launched, there is a chance that life can get back to something like normal (what is that at Harvard?). It was odd spending a significant portion of my time on a project that I couldn’t, until yesterday, talk about. It was exciting, it was strange, but it also made me appear more flakey and irresponsible than I would like. Also more tired.

It will come as no surprise to learn that the partnership with MIT was not the only choice that had been offered to Harvard. The usual suspects had all come by. It’s nice to have the Harvard brand, in that educational ventures are all anxious to have us join in.

I was also heartened (and impressed) by the considerations that made Harvard decide to go the edX route. One of the core reasons that we are taking the direction that we are taking is that we can approach on-line education (and how it impacts the on-campus experience) as a research project. The folks who are guiding the educational ship have the Socratic wisdom to know that we don’t know how to approach the use of technology to teach. The folks at MIT who are doing this have the same wisdom. So this isn’t a venture-funded concern where we are shipping a product. Instead, edX is a not-for-profit run by two universities with the stated goal of finding out, through real research, how best to use on-line technologies to enhance our core missions of learning and research.

This is not a simple research endeavor. I’ve been known to characterize this as something on the order of the Manhattan project or the Apollo program. It is going to take time. It is going to take money (fortunately, we are already being approached by foundations and other donors who are excited about this). It will take cooperation and the suppression of some standard primate behaviors. Most importantly, we don’t know what the end result will be. But we do know that it will be transformational in the areas of teaching and research. Which are the areas that a university like Harvard should be transforming.

I think the whole thing is pretty exciting…

A gentle nudge from a reader made me realize how long it has been since I’ve posted. Time to get back into the habit.

This has been a particularly busy semester, both from the point of view of my academic life and as CTO. The academic side has been great– I’ve been teaching CS 105, Privacy and Technology, which is always more fun than I should be allowed to have. This is a class that looks at technologies that are seen as privacy-invasive (things like surviellance cameras, wire tapping, and Facebook), dives into the technology and policy, and tries to figure out what can be done. I co-teach with Latanya Sweeney, who really knows this stuff, is a great lecturer, and a better friend. But what made this semester fantastic was the best group of students I’ve ever had in a class–smart, engaged, funny, and fun. On days (and they happen) when I wondered why I was doing all of this, I just had to go to this class to be reminded what fun it is to be at Harvard.

The CTO work has been a lot more scattered, but has also been interesting. Probably the biggest change in my life as I moved to the CTO position was finding that I have very few concentrated, extended periods of time to think about things and get things done. The life of a CTO is one of constantly swapping context, trying to help others (who I hope have concentrated periods of time for their work) to move forward or course correct.

There is also another, odder, part of my job which I characterize as being a technology canary. Canaries were used as early warning systems in mines, organic sensors for dangerous gases. My role of technology canary is to be an early warning system for HUIT on technology trends that are going to change the way we do our jobs. There are lots of these changes coming around, like the changes in client devices (moving from desktops to laptops to tablets and phones, a change that had a pretty disastrous impact on the University of California’s email system). But the most interesting whiff of the future that I’ve seen had to do with a bill from Amazon.

First, some context. All colleges and universities are supposed to offer a net price calculator, a tool that will allow prospective students and their parents to estimate what their college educations will really cost at a particular school (anyone who has to worry about this doesn’t pay list price, at least at Harvard). The financial aid folks here have done a very nice web tool, which they decided to host on Amazon.

Recently, I got a copy of their bill for a month. They had had about 300,000 hits, most from the U.S. but others from all over the world. And the total bill for running this site? $0.63. That’s right, sixty-three cents.

Now, not everything we do at Harvard in IT can be farmed out in this way. This is a simple site, and doesn’t have to be up all the time. It doesn’t have a lot of computation associated with it, and there isn’t a lot of data being moved around. More important, there is no confidential or protected data. But there is a lot of computing at Harvard which has similar characteristics. And at this price, we need to figure out what we can host elsewhere. It may cost more than this example, but even if it is one or two orders of magnitude more it will be less expensive than setting up our own servers and running them here.

This will change a lot of things. We need to figure out what will be changing rather than having it done to us. As the canary, I get to think about these things early on. Which makes life more, um, exciting. But also a lot of fun.

I was reminded once again this morning of how the language of programmers is not the language of everyone else (“doh”, I hear you say). For most of my adult life I lived in a society of software developers, and the linguistic patterns developed are hard to shake (even if, contrary to fact, I was trying to shake them). There are some phrases that just elicit blank stares from my non-programming associates. I’ve come to realize that “paging in the context” is not something most people rightly understand, nor do I expect them to understand where /dev/null is (or, more importantly, isn’t).

What is more distressing is the terms that are understood, but in a different way, by the general community. In particular, I find that I often get worried looks from my colleagues when I talk about doing some hacking, or being a hacker. Those not in the programming community understand hacking to be the act of breaking into a computer system. But the older meaning, still in use by many in the programming world, is much different. On that meaning, “hacking” is the activity of writing code that is particularly clever, elegant, or that solves a particularly difficult problem, and hackers are those who have shown a consistent ability to write such code.

In the programming culture, being called a hacker is an honorific. This is the sense of the term that Steven Levy wrote about in his book Hackers (subtitled Heroes of the Computer Revolution). To be called a hacker in this sense is a considerable honor, and (at least in the old days) was not a title that you could bestow upon yourself, but one that had to be bestowed on you by someone who was himself (or herself) a hacker.  A kind of hacker writes good, clean, understandable code that runs well and does the job at hand. It is a term denoting craftsmanship and art.

But now the term “hacker” is being used to denote those who break into networked systems. There is a connection between the old meaning and the new– some of the early (code) hackers believed that they should be able to look at any code to try to make it better, and would use their (considerable) skills to break into computers that those who were not hackers would try to close off to them. But the goal, in those cases, was always to make the underlying code better, not to steal information or shut a system down. That is a new phenomenon, or at least newer than the term. I’m afraid that the press has made this meaning of the term the dominant one, and trying to change it is a battle already lost.

But it does lead to confusion. When I tell people that I am going to do some hacking, I get odd looks, and have to explain to them that I just mean writing some code. Even worse, when people ask me if I am a hacker, I have to ask what they mean by that. On one sense of the term, I am proud to say that I am (or at least once was). But in the more popular sense of the term, I am not (by choice, rather than because I can’t).

I, like many in the tech world, was saddened at the news of the death of Steve Jobs. I had met Jobs a couple of times, but he hardly counted as a friend or even acquaintance. I’d experienced the reality distortion field around him, but his main impact on my life is on the computing environment that I use, and the gadgets that are part of my daily life. Even my choice of a phone (Android based) was made as a conscious decision not to buy another Apple product rather than a decision to buy something else.

But I’ve also found it a bit odd that everyone talks about what an innovator he was. He wasn’t, really. Apple didn’t do things first. Xerox PARC did the windows/mouse/icon interface well before Apple. Sony did personal and portable music before the iPod. Smart phones existed before the iPhone, and tablets were around for a couple of years before the iPad.

What Apple under Jobs did so well was to design products that were beautiful and a joy to use. Much of this had to do with the design aesthetic that Jobs brought. But just as important was that Jobs trusted his customers. He felt, in the face of all the business advice to the contrary, that building a beautiful product that was easy to use would be appreciated, and that his customers would be willing to pay extra for the beauty and ease of use. And he built a company around that, which was successful. You can see the same sort of faith in the customer in the products of his other company, Pixar– the animated movies didn’t condescend to the audience, but expected a level of intelligence and sophistication that differentiated those movies from the usual animated stuff.

I hope those now running Apple are allowed to continue on the assumption that their customers care about more than cost, and that design is important. We won’t know until the product pipeline that is currently filled has emptied out. I sincerely hope that they will, and that I’m given the choice to fill my computing world with objects that make me smile (mostly). Otherwise the world will have lost much more than just another innovator.

I still think the best exemplar of the Jobs attitude is the famous 1984 ad. This is how I will remember what Steve Jobs did to the technology industry, and I will always be grateful.

A recent thread on an email list I subscribe to started with the question of why there are so many roles for technically oriented people at Google (engineer, lead, manager, product manager, production assistant, etc.) and then moved to the question of how decisions can be made in such an environment. All of this was around the more general topic of how to encourage innovation. I was asked to comment as someone who had some experience in the tech industry. But this is a hard problem, needing more space than a good email. It is also a topic that might be of interest to more than the subscribers of that list. So I decided to move the discussion over here.

Let me start by saying that, in my experience, the worst (and least innovative) technical organizations are those in which there is a single career path that starts with individual contributor and moves to various levels of manager. This might work for some functions (although I can’t think of an example right off hand), but is very bad for engineering. The only real advantage that this sort of organization has is that it is clear who gets to make decisions. The higher up on the organizational tree one is, the more authority one has. It is efficient, lines of authority are clear, and if you are trying to do something technically innovative you are doomed.

This kind of organization has the well-known drawback of making bad managers out of good engineers, but it also means that many of the most important decisions that are technical in nature are made by people who are spending their time on management. Having spent time as both a technical contributor and a manager, let me assure you that it isn’t just the skill sets that are different. The main difference is the way you spend your time in the different roles.

Engineers (and, by that term, I mean anyone who is building something, whether it is a web site, a new OS, or a service) need blocks of time to think about what they are doing. There is a lot of context that needs to be swapped in to the brain, and a lot of concentration that is required to delve into all of the corners of the implications of any decision. They shouldn’t spend much time in meetings, and the unit of time for their schedule should be no less than an hour (it’s even better if it is the day).

Managers, on the other hand, are interrupt driven. If you get a full hour of time to spend on some topic, it is a bonus. A really effective manager may occasionally disappear to do long-range strategy or planning, but once that plan is in place the decision process is to compare an action to the plan and then do the action if it fits and do something else if it doesn’t fit. Contemplation is not part of the job description.

Dual Track Systems

A much better organization is one in which there is a dual career path. One path takes a person into management. The other keeps a person in engineering. And each lets a person move up the organization, increasing scope of authority, pay, and prestige. The first company that I can remember having this sort of plan was Digital Equipment Corporation. I lived in the dual track at Sun for a long time. Most high tech companies have adopted some variation of this scheme, and many that you might not think of as high tech have also decided that this is a good idea. I’d love to see something like this within the IT organization at Harvard.

Most dual track system will have a number of points at which your career can branch. You start as an individual contributor, but at some point (sometimes early on) you can opt to go into line-management or become a technical lead. Line managers and technical leads share responsibility for a development group. The technical lead makes the technical decisions, while the manager makes business decisions, deals with budgets, does reviews, and the like. Clearly, these two people need to work closely together. The manager needs to be technical enough to understand what the technical lead is doing (if for no other reason than to explain the work to other managers) while the technical lead needs to understand the business case for the technology. But the creation of the business case is the job of the manager, while the creation of the technology is the job of the technical lead.

At the best companies, this dual track continues all the way up the organization. At the old Sun (and now at Microsoft, IBM, and lots of others) there was the job of Distinguished Engineer, which was the technical equivalent of a director or first-level vice president. The D.E.s (generally) had no reports, but designed and led the implementations of major parts of the technology. Becoming a D.E. was not something that just happened as a matter of course, just as becoming a director or a v.p. did not just happen. These engineers were often associated with a manager at the same level to oversee larger and larger parts of the company. At the very top of the company, the Chief Technical Officer was the technical representative on the executive team.

Decision Making

This did mean that the authority to make decisions was often unclear. While the distinction between a technical decision (made by the technical person) and a business decision (made by the manager) might seem clear, it often isn’t. The best groups had a team of leaders who could work well together, when those two didn’t work well together, it was not a pretty. When D.E.s decided that they could manage, bad things happened. When managers decided that they were sufficiently technical to make design decisions, worse things happened.

And then there are those companies that have decided that two decision makers is not enough. The thinking there is that while there is a clear distinction between the technical aspects of a project or product and the management of that project or product, there are other stake-holders that need representation. So the role of product manager is introduced to speak for the customer. VMware had such a system, which I believe is also used at Microsoft. At its best, this form of user-representation can provide valuable input into design. At its worst, it introduces the moral equivalent of the political officer in the Soviet military– someone who can second-guess any decision without having the responsibility to actually do the work.

From what I’ve heard, Google has introduced even more roles. At best, this will allow more points of view to be represented in the decision making around what is to be done. At its worst, this will make it harder to move quickly and will make everyone feel less responsible (since there were so many voices). It will be interesting to watch; I’ll express my own skepticism that it will work, but we will see.

Missing the Point

But at bottom, all of this talk about roles, and career paths, and tracks misses the most important point. It confuses management, organizational structure, and process with leadership, which is the thing that is really important.

All of the successful groups, products, and technology development I’ve ever seen or heard about were organized (intentionally or, more often, by chance) around a strong leader. Sometimes this was a manager, more often it was a technical person, and on occasion I’ve seen it be a marketing person or a product manager. But someone needs to be willing to insist that they know where everyone should be going, and have the force of both vision and personality to get others to sign on.

This is not something that you are going to get out of any definition of roles, or career track, or process. My own observation is that when a company tries to insure innovation through one (or more) of these things, it is probably time to look around for another gig. Innovation is the sort of thing that happens in all sorts of ways, almost none of which can be replicated or scaled. It is like good design– all of the best studies show that good design comes from good designers. Innovation comes from (and is driven by) innovative people, no matter what their title or role and no matter what process they use. This makes it impossible to manage, which in turn makes many managers nervous. But it is, I’m afraid, the truth…

Time to try another open office hour. Fridays seem to work best, and this time I think I’ll try the suggested location of the Fisher Family Commons in the Knafel Building of the CGIS complex (1737 Cambridge Street). I’ll show up around 3:30 (the lunch crowd should have left by then), and stick around until 4 or no one is around, whichever comes last.

All topics are open. If nothing else, come with the feature of Java that you hate the most, and I’ll tell you why it happened that way (working on the principle that history clarifies stupidity).

Programmers don’t juggle anymore.

There was a time, many moons ago, when you could pretty much assume that anyone who wrote code in a serious fashion knew how to juggle. Maybe not well, but at least well enough to keep three balls in the air for an extended period of time. Maybe even well enough to do passing patterns. You could go to a programmer’s office or desk and find a set of lacrosse balls (much better for juggling than tennis balls, until you miss).

There were other skills that were reasonably common, as well, such as lock picking. It was always nice to know that you couldn’t really be locked out of anyplace. Some programmers had real lock picking tools, while others made do with home-grown picks. It was part of what made the culture of programming somewhat unique.

I started thinking about this recently when I realized that none of the students that are around have any of these hobbies or skills. I wondered what had changed. And then I realized that it was because of the speed of the machines that we all use.

It used to be that part of programming was long periods of waiting. You put together a program of reasonable size, and it had to be compiled, which would take a while. And by a while I mean half an hour or more (I won’t bore you with the tales of waiting overnight for your deck to be run; go ask your parents). You needed something to do for the time it took the program to compile. You didn’t want to start on another program, because you wanted to be ready to fix any errors or start working on tests once the program compiled. So we did things that took enough attention to pass the time, but not so much brain power as to be actually useful. Juggling. Picking locks.

The exceptions to this were the programmers who worked in Lisp, who had instantaneous feedback from their integrated development environments. The Lisp community produced very good programmers, but they were (and this is going to get me in a lot of trouble) less interesting as people. Not only were they intolerably smug about their programming environments, but they didn’t juggle. How could they be real programmers?

Now, of course, our machines are so fast that we all use integrated programming environments that compile the whole system between our keystrokes. These interactive environments have brought all the speed and convenience of the Lisp environment to those of us who use compiled languages. The only time we wait for compiles now is when we download open-source software from someplace else and have to build the entire system from scratch (or while we wait for our IDE to initialize). But that wait isn’t long enough to do anything else other than, perhaps, read some email.

Mind you, I’m not really complaining. I wouldn’t want to go back to the days of 45 minute compiles (especially when those compiles found one typo, which got fixed in a few seconds and then required another 45 minute compile). But all this speed has made the community of programmers less interesting. No one juggles. No one picks locks. I sort of miss it.

At least the Lisp folks aren’t so smug anymore.

Well, something clearly went wrong with the last open office hour. Whether it was the day (Friday is probably better than Monday), or the weather (variously terrible), or the fact that there were preparations for a BBQ going on at the location (the perils of doing something unofficial), no one showed up. I hung around for 15 or so minutes and then headed off myself (sorry for anyone who came late that I missed). I’ll try again later this month.

Rather than talking about what went on there (a pretty short post), I thought I’d say a bit about some of my current thinking about open source. This can be something of a hot-button topic. But having lived in the open-source world for the last 10+ years of my time at Sun, I’ve thought about the topic quite a bit, and have some experience, as well.

Let’s start with a basic premise. Harvard is not a software company. There are those who say that Harvard is really a real-estate trust, or a hedge fund, but I’ll stick with the idea that Harvard is an institution that builds its value around education and research. The software we (at least those of us who are in the IT organization) build is meant to support that basic value, and to keep the overall institution running smoothly and efficiently. Other software is built to test research hypotheses. But none of it is built to sell. Since software is expense to develop and (especially) to maintain, we should only build our own software when we absolutely have to.

This means that, if we can fit our needs to a commercially available software package, we should probably do so rather than build the software ourself. But higher education is a small niche for a software solution, so there aren’t a lot of major players building software for this market. Where Harvard is like other businesses (for things like human resource management or ERP systems) we can find commercial offerings. But there are other areas (like student information systems, or classroom scheduling) where the market is just too small for the major players.

This puts us in an interesting bind. We don’t want to be writing and maintaining software, because doing that is expensive and not part of our core mission. But we are not part of a large enough market to attract the major (or even many minor) software companies. And, more and more, we need software to support our core mission. This need will only expand in the future, as we figure out new ways of using technology to support both teaching and research.

Open source is one of the ways to get around this dilemma. There is a lot of open source software that supports various academic endeavors. This existing code may not be exactly what we want, but we have the source so we can change it as needed. Even if an open-source package only does 50% of what we want, that is more than what we have if we start from scratch. It isn’t actually 50% more, since there is time and effort in learning an open-source code base. But it is a start.

Even more, it means that there are already some developers who are working on the problem, who can be used to leverage our own development efforts (although they may view us as leveraging theirs). And, if the package is successful, there is a group of maintenance engineers larger than the staff we can use to maintain the package (which is also true for everyone else that is working on the package).

Open source gives a nice mix of low price, high safety, and the ability to customize. The low price is actually quite a bit higher than free; you need to learn the software, be ready to use the open-source community for maintenance, and be prepared to join and support the community. And you could be stuck with maintaining the software by yourself (which is one of the things that argues against writing our own) if the community around the open source evaporates. But you still have the source, and can make changes, and the initial price is low.

If there is no open-source package, then we may have to write some software from the ground up. But if we do so, we should build that package with the intention of making it open-source, or just start the endeavor as an open-source project. With any luck, others will join in and help in the development or the maintenance. There is some cost to this (building and working with an open-source community takes effort), but the end result can be much better than if we keep the code to ourselves and will never be much worse.

Harvard has perhaps the best brand in the world of higher education. If we start writing open-source software in support of our teaching and research missions, I’m pretty sure we would have others joining the effort. If we are really lucky, someone will start a company around the software that we help build (perhaps current members of the IT staff, or perhaps students). If that happens, we can trade our intellectual property rights for maintenance, freeing up our development resources for something new.

So, to a first approximation, here is my crude view on how we should approach software at Harvard. If there is an existing commercial application offered by a major vendor, we should buy it. If there is an open-source offering that even approximates what we need, we should adopt it, join in the community, and alter it (with the alterations going back to the community) for our needs. And if there is nothing, we should write the software as an open-source project, inviting others in to help in the development and maintenance.

One of the suggestions that came out of the discussion when I spoke to the ABCD group was that I hold some sort of open office hours. The idea is that I could just be available for anyone who wanted to come by and talk. No agenda. No plan. Just discussion.

I thought it was a great idea, but wondered if it would work. But I tried it out a couple of weeks ago, and to my surprise (and delight) about 8 people showed up. We had a terrific conversation about what they did, what got in their way, and what made their part of Harvard unique (the person from Classics spends lots of time support odd and non-standard fonts, which was interesting to hear and not surprising once she said it, but hadn’t occurred to me before). The second attempt (last Friday) had about a dozen people, and we ended up talking a lot about open source and the role it could play at Harvard. Both great discussions.

So I’m going to have another. While Friday afternoon is a good time for this sort of thing, I’m going to be out of town (vacation) the end of next week, so I’m going to try varying the day to see what happens. So the next CTO open office hour will be next Monday, August 8, at 4 p.m. in the ground floor Maxwell Dworkin lounge. I’ll put up a sign of some sort (probably on my computer screen) to identify myself.

So come one, come all. The discussions are good, I’m learning a lot, and it is your chance to be heard…

And in an attempt to get some comments, where are some other places on campus that would be good to hold this? I’m looking for someplace informal, that needs no particular scheduling, and where I won’t be embarrassed if I end up spending the hour by myself but where a group of 10 to 20 can sit and chat. I know the Engineering part of campus pretty well (which is why I’ve been doing this in Maxwell Dworkin and Northeast Labs), but it would be good to move to other parts of campus as well. So if you have a suggestion, tack a comment on this post and I’ll entertain other options.

The great thing about starting up a blog is that you get asked interesting questions. Even something as vanilla as my introductory post brought some good comments (and some help in setting up the Word Press theme; thanks!). Then along comes Ruby Bright and asks one of the big questions. As Ruby so beautify put it,

I’m not all that sure that I have the answer to this question; if I did I’d be rich and famous. People have written books about the problem (Crossing the Chasm is probably the most famous). I’ve thought about the problem a lot. And so have a lot of others, much smarter than I. When I worked at Sun Microsystems, this was a constant topic of conversation. It caused the company to move from workstations to servers (an innovation that worked), and to support Java (much more innovative, but maybe not so successful financially), but in the end we didn’t find another innovation that the company could support. I’m sure that this is a topic of conversation (all the time) at places like Google, and Facebook, and Microsoft. Doing something new is really hard, especially when what you are currently doing is working well. It is even harder when you are well known, because then everything you do is watched by everyone, and there is the worry that failure will dull (or even ruin) your reputation.

I remember seeing this risk-aversion develop in the Java organization. When Java first came out, the people working on the project thought of it as a new way of writing software, with the virtual machine, object-oriented language, and portability from machine to machine no matter what the operating system. There were all sorts of new things that we wanted to try. Innovation piled on innovation. It was really fun.

But then Java took off, and we had other companies as partners and the price of the company stock was determined by the latest news story on something that had gone wrong (or, less often, something that had gone right) with Java. Within a couple of months the question stopped being “what new can we do with this shiny thing?” and became “how do we keep from making any mistakes?” And a lot of innovation was stopped or never started.

Fortunately for those of us who want to innovate in the IT world of Harvard, we aren’t under quite the same set of constraints. Harvard is not, at core, an IT organization (or at least it doesn’t believe that it is, at core, an IT organization– this may be the subject of a future post). Since IT is a support for the core activities of teaching and research, we are not under quite the same microscope as we would be if Harvard were a hi-tech company.

And innovation, as I continually tell my colleagues in HUIT, is not an activity that can be planned, or managed, or controlled by some process. It is messy, and unpredictable, and requires leaps of faith. It is hard. But there are a couple of things that seem to help.

I’m a big fan of skunkworks projects– small groups of engineers that go off and build something out of sight of most everyone. This is a great way to build a prototype. You need to do it fast, and you need to understand that you might fail, and be willing to take the risk. But with the right people this can work amazingly well. The first version of what became the Common Object Request Broker Architecture (CORBA) was written by a group of 5 engineers in six months in a skunkworks (I had the great pleasure of leading that team). When we did the Jini work at Sun, the team was hidden from the rest of the organization by being put (organizationally) in Human Resources. Making such an investment is something I hope we soon have the freedom to do, at least at some scale. It may be that we involve students to keep the costs down, but we should try something.

I’m also a fan of importing good ideas from elsewhere, and then making them our own. This may be in the form of adopting an open-source solution that we then customize for Harvard. It may be in the form of taking work someone else has done here, and adapting it for a new purpose. To quote the sage Tom Lehrer, “brilliance borrows, genius steals.” We should be unashamed of using the work of others (always respecting IP laws, of course, and giving credit where it is due).

But most of all, we need people to be willing to take risks. Managing projects in any organization is like managing a financial portfolio– if you want major rewards, you need to allow some risk. You don’t want every investment to involve major risk, but you need to have some to balance the low-risk parts of what you do. It is best to manage risk with some vision of where you want to go; this is a vision of the future. I’m trying to encourage some risk taking in the organization, and there is plenty of support for trying this out. We will see if it will allow us to innovate. I think it will, if we let it.