Why isn’t there a glut of good software engineers?

As Google hires all of the world’s good software engineers and my friends with startup companies fight over the scraps I am left to wonder how everyone could have been so wrong in predicting that the world would be glutted with good programmers and sysadmins by now.

In the early days of programming there was not considered to be a limit on how many people could be trained to do the job. IBM would hire graduates from all kinds of college majors, e.g., music, and train them (there were hardly any computer science programs at universities back in the 1950s). A popular idea among corporate managers is that this could be a good job for women, who were readily available at reasonable rates.

With the rise of the worldwide Internet, open-source, and inexpensive microprocessors, it seemed inevitable that the world would be glutted with technically skilled people. Programming is easier to learn than a foreign language (a friend who has a PhD in physics and is an expert hardware and software developer says that learning Mandarin and to read Chinese was by far the hardest thing that he ever did). Getting hired as a programmer does not require a specific degree or training course. Programming or system administration can be learned by anyone worldwide who has access to a $300 personal computer and an Internet connection. Why wouldn’t the millions of unemployed Americans train themselves to code? Why wouldn’t middle class people in China, India, and Africa?

Admittedly in the U.S. folks with an organized mind, the grit to get through technical subjects, and the drive to go to work every day can find higher paying jobs that involve more social interaction (e.g., medical doctor, Wall Street banker, etc.). But that doesn’t explain why people in the Philippines or China aren’t training themselves en masse to be able to soak up the $30-100/hour jobs that would be readily available to them if they could demonstrate the ability to turn a customer request into working code. Why are the Greeks, Portuguese, and Spaniards going bankrupt instead of learning to administer Cisco and Linux?

Any good futurist circa 1990 would have predicted that by now most of the world’s programming would be happening in low-to-medium-wage countries and that customers wouldn’t have to pay a lot for high quality software. But it hasn’t happened (e.g., see my posting about California’s state government spending $327 million on a straightforward ecommerce site (I visited http://www.coveredca.com just now and it seems to be a work in progress)). Why instead do we have catfights over mediocre talent in Kendall Square and Silicon Valley?

79 Comments

  1. J

    May 2, 2013 @ 9:26 am

    1

    “Why are the Greeks, Portuguese, and Spaniards going bankrupt instead of learning to administer Cisco and Linux?”

    Portuguese national here.

    Long story short, the general feeling is that a good software engineer isn’t much different from being a mediocre software engineer because companies prefer to employ 1 or 2 code monkeys (and pay the least possible) instead of a better, more expensive software engineer.

    I believe this happens because Portugal’s IT market only cares about two things: either accounting software or consulting services for ISPs. R&D is either limited to the academia – completely cut-off from the market – or to specialized centers inside the large ISPs, so why pay two or three times more for a better engineer if you’re not going to need his expertise?

    The idea is to pay as less per person as possible and still demand they do overtime because “the market is bad” and if you don’t like the conditions too bad, someone will. Outside IT it’s even worst, as some companies are offering a ridiculous 300 $ per month (that’s 200 $ below the minimum wage!!) and they even have to nerve to tell the candidate that it’s their way or the highway.
    Screw motivation, get back to work grunt.

    PS: mods, why did you delete the previous post … ?

  2. Chad

    May 2, 2013 @ 9:50 am

    2

    Intellectual property rights are not well respected globally. Every thief out there, most don’t realize they are a thief, contributes to the culture such that investing (always a long range activity) in software is too hazardous or even impossible.

    The U.S. is a leader in defending intellectual property, therefore they are the sweet spot for software development.

  3. Dan Weber

    May 2, 2013 @ 10:19 am

    3

    Chad hits a good point: if I have my software built by a Chinese company, it’s very likely that, if it works, they’re going to sell it to someone else, too.

    It’s possible to work around this by planning for it to be open-source, but then what are *you* doing? You surely don’t know the code as well as the Chinese people writing it.

    Speaking of open-source, that didn’t put software developers out of jobs, although a very simplistic view of code would say that was inevitable. The thing is, there is not a finite amount of software to be written. If someone writes a bunch of code for free, there is *still* a lot more gains to be gotten on top of that.

  4. Nuno Venâncio

    May 2, 2013 @ 10:21 am

    4

    As software developer here in Portugal i don’t know what is the reality in the US, but in Portugal we also have a lack of people to work in the industry, and the main reasons are that :
    a) The young people don’t want to learn the skill set to do the job, because they know they are going to be ostracized by others;
    b) The pay is low, $30/$50 would be very nice;
    c) Because the pay is low many IT workers go to Germany, Uk, France even Spain;

  5. George

    May 2, 2013 @ 11:04 am

    5

    A good software engineer is someone who can solve problems, someone who can see past the lines of code and see the many other variables in play, and someone who can communicate in plain English about how a system works in and around your product.

    When I started 20+ years ago I knew C/C++ down to earth. Today, I code in 5 languages and don’t call myself an expert in any of them. However, I’m regarded by my organization as indispensable engineer because I’m a problem solver.

    When I interview new candidates, I care less about their domain, or programming language experience. What I want is someone who knows how to find answers, and someone who sees past the given use-case or logical flow in the code.

  6. Gary Henson

    May 2, 2013 @ 11:22 am

    6

    I am stunned by the tone of some of these posts. ‘IQ of > 125’, ‘mediocre’, ‘code monkeys’, ‘the best and the brightest’. Look around, guys, really look at what you do, day to day.

    Programmers are not superior beings, able to gaze at their navels, hum a mantra, and then spout perfect or even good code.

    The variables in developing robust, viable software systems are enormous. The challenges vary from mind-numblingly boring to moments of ‘oh my God, how cool it that’ and everything inbetween.

    ‘Good’ programmers have one common trait; they care about their work. Everything else is learned, easily or painfully hard, it doesn’t matter.

  7. Brian Balke

    May 2, 2013 @ 11:32 am

    7

    The fact that the market hasn’t responded with higher salaries reflects that fact that making money on software is a crap-shoot.
    The economics of software are different from anything that preceded it. Software does not just manifest the insights of the engineer – it propagates them at almost zero cost. This means that one great software engineer at a competently managed company can wipe out a whole industry of less competent practitioners. This is why we see cannabalism in our hiring practices: senior engineers develop platforms, and then are replaced by the junior developers they empower.
    What this also means is that the great developers, if they have domain and business savvy (or link themselves to someone with that savvy) can make a pile of money.
    Well – of course – they can make a pile of money until hungry developers looking for an opportunity copy the superficial functionality and offer it to the world for free, given that they can’t make any money in direct competition. Then for the original developer it becomes a hard battle of extension, integration and refactoring to maintain a competitive posture in the marketplace.
    So why pay a lot of money to produce something when the risks are so enormous?

  8. Bob

    May 2, 2013 @ 11:41 am

    8

    There isn’t a glut of good software engineers because the problem isn’t learning the programming languages. Basic grammar is something most people in developed countries have been taught before they hit puberty but there are not many high schoolers would be considered on par with the Bard. Because mastering grammar is not the same as mastering writing. The same is true in programming. Just because someone can use an if/else does not mean they are capable of writing a large application or even a small one.

    Good programming starts with logical thinking, a skill in short supply. Once the problem has been analyzed, devising a solution that solves the problem without a bunch of excess cruft around it. Then the slinging of code starts.

    I am a self taught developer and after a couple years of learning I was hired at the upper end of the pay scale discussed in the article. I work with a number of developers who have been “professional” programmers for 5-15 years who would not be considered good by any definition of the word. They know the grammar but they struggle to write code that is clean, efficient and effective or in other words not awful.

  9. Dan Sutton

    May 2, 2013 @ 12:03 pm

    9

    The supposition that the world will be glutted with good programmers has a presupposition that the world will be glutted with intelligent people, as though by magic. Not going to happen…

  10. Clay Shannon

    May 2, 2013 @ 12:04 pm

    10

    I reckon it depends on the person, but in my case, I found learning German well (I grew up speaking and hearing English only) was, although not easy, quite a bit easier than learning to program well. Mandarin Chinese I realize is a harder language for English speakers to learn than German is. But just as there are big differences between the difficulty of programming languages (Basic and C++, for instance), there are large differences in the difficulty of human/natural languages, too.

  11. Jeff Jones

    May 2, 2013 @ 12:20 pm

    11

    There are several reasons (IMHO).
    1 – Good software engineers are often overlooked because of age. Many companies have no interest in quality software engineers when they get near 50.
    2 – Cheap, low quality, non-American software engineers help the short-term financial statements, but not the long-term health of the company.
    3 – The myth persists that non-American software engineers are smarter and do better work for a cheaper price. The cheaper price is true, but those of us who have dealt with offshoring and H-1B visa holders who work cheap know the quality is rarely there, and the ability to figure out problems that don’t fit the textbook is rarely there, either.
    4 – American education, at least in government schools, has been dumbed down the last 30 or more years. Today’s 20 and 30 somethings are more likely to have been indoctrinated, not educated. The basic knowledge set of a 1970s high school or college graduate is generally superior to that of such graduates in the 2000s and later. That has led to a smaller pool of people with the character, critical/logical/reasoned thinking necessary to be a good software engineer.

  12. Dan Weber

    May 2, 2013 @ 12:30 pm

    12

    The fact that the market hasn’t responded with higher salaries reflects that fact that making money on software is a crap-shoot.

    1. Salaries have gone up in certain localities, like SV. If you want to live there, you can make $100K out of college.

    Now, the better question is why salaries haven’t gone up as much in other areas.

    2. Software companies make boatloads of money. Here:

    http://www.techspot.com/news/43846-apple-overtakes-google-becomes-first-in-yearly-profit-per-employee.html

    They aren’t quite in Goldman Sachs territory, but Goldman Sachs will proudly give all that profit to the employee instead of the shareholder. If software engineers can manage to get better at negotiating, they too could have $200K – $300K salaries.

    Power to the proletariat!

  13. Steve

    May 2, 2013 @ 3:35 pm

    13

    I would guess that outsourcing nations at least have a culture which discourages and suppresses talented engineers. The IT industry in India and China is built largely on providing cheap software to the kind of business that looks at software as a cost center to be minimized, rather than as a product of variable quality. Therefore, anyone who really does feel passionate about the work either gives up or moves to North America. Most people are likely so grateful to have a good job that they don’t think twice about the lack of attention to quality.

  14. Kaveh

    May 2, 2013 @ 5:30 pm

    14

    I graduated with a CS degree. Big companies pass on. Tons of resume without any answer from them. I have to settle for small ones which pay so little. What is the chance of self taught programmer get a job? very little.

  15. Jeff

    May 2, 2013 @ 6:39 pm

    15

    There was that comment in “snow crash” about how (once everything that could possibly be outsourced from the US has been) the US is left with “music, movies, microcode, and high speed pizza delivery.” Three of the four make sense, but I’m not sure why microcode (he clearly means software) would be in here. Maybe there’s some social paradox where top coders can only be drawn from a social environment that actively (perhaps unconsciously) discourages people from getting good at programming.

  16. SeattleC++

    May 2, 2013 @ 7:01 pm

    16

    I can teach a Greek to speak English, but that doesn’t make him William freakin’ Shakespeare. I can teach a clerk to write Excel macros, but that doesn’t make her Ada Lovelace. There are more programmers than there are good programmers, and more novice programmers than there are experienced ones.

    A four year Computer Science degree is about 90 credit hours of specialization. It takes 2 full-time years (on top of college) to acquire a journeyman’s competency with C++. Your 10-year-old’s six months of python does not make him employable.

    There is no shortage of talented programmers. There is only a shortage of talented US programmers with a CS degree and 10+ years professional experience willing to work for $35/hr. All shortages can be cured with better money, more perks, and greater respect. There are sometimes surpluses of programmers, like during the Great Recession when some companies were laying off and nobody was hiring. That there are never shortages but there are sometimes surpluses is a factor in students’ willingness to blow 4 years getting a degree, when they can get other jobs with less up-front effort.

    Google and Facebook have revenue streams so wide they can afford to pay top dollar and awesome perks for programming talent. That’s why they get the talent. If startups offered great wages instead of a dream of wealth and a year living on ramen, then Google and Facebook wouldn’t get all the top talent. Plus if you’d worked with this top talent, you might be unimpressed with what they get for their money.

  17. philg

    May 2, 2013 @ 10:46 pm

    17

    Folks: I don’t think it is apt to compare programming to being the author of a novel or a play (see above comment above Shakespeare!). A programmer who simply listens carefully to user requirements and then implements a system to meet those requirements will be considered at least a “good” programmer. He or she need not create an entire world the way that an Edith Wharton or Tolstoy might. A better comparison, in my opinion, would be to journalism. And most people do accept that a capable journalist can be produced by training.

  18. Trevis Rothwell

    May 2, 2013 @ 10:57 pm

    18

    Philip: Elsewhere you’ve said that a lot of computer science university graduates don’t learn how to be an engineer, but how to work for an engineer. Is it fair to think that someone who self-studies programming through books and free online resources would be capable of deciphering user requirements and building a software system that meets them, when formally-trained computer scientists seem unable to?

  19. John Galt

    May 3, 2013 @ 1:45 am

    19

    Why are the Greeks, Portuguese, and Spaniards going bankrupt instead of learning to administer Cisco and Linux?

    “We’ve all heard that a million monkeys banging on a million typewriters will eventually reproduce the entire works of Shakespeare. Now, thanks to the Internet, we know this is not true.” – Prof. Robert Silensky

    That’s why.. :))

  20. Alex Simi

    May 3, 2013 @ 11:09 am

    20

    @Jeff Jones

    “The myth persists that non-American software engineers are smarter and do better work for a cheaper price. The cheaper price is true, but those of us who have dealt with offshoring and H-1B visa holders who work cheap know the quality is rarely there, and the ability to figure out problems that don’t fit the textbook is rarely there, either”

    I’m not sure who believes that non-American sw engineers are better and cheaper, but I can share a couple of personal stories as a proud former H1-B visa holder and, now, as an even prouder American citizen. More than half my colleagues at my various sw engineering jobs in US were born outside the country. The proportion of good or even outstanding engineers was the same among the immigrant and American groups of people. My compensation package has been competitive.

    My experience with the offshore teams has been somewhat less encouraging. With the exception of one country, where the company I work for has had a presence for a long time, the other remote sites, which were newer, had a lot of junior engineers. Among these junior engineers there were a few who were very good and a couple who were outstanding but the problem was that the attrition was high and whoever was a good engineer could quickly find a job somewhere else, with better pay. The most recent example includes a person who was hired by Facebook and relocated to SV (and yes, the compensation package was competitive and it fit that person’s experience). Another issue was that even for those who stayed with the company for a longer period of time it wasn’t easy to advance in their careers due to a lack of good projects – the older teams, which were usually based in US, had priority, were closer to the management, sometimes closer to the customers etc.

  21. Jim Howard

    May 3, 2013 @ 3:05 pm

    21

    Really great discussion here, and relevant to me because I’ve just a fair of amount over the last week interviewing intern candidates here in Austin.

    My background is working a software developer in a number of small companies here in Austin since I retired from the military in 1994. I’ve been at present gig for ten years, and was the day 1 employee.

    It became clear to me very quickly that in the civilian/commercial world hiring the right people is critical to the success of a software company. For that reason I’ve always stayed involved in recruiting as much as I can.

    I mention this because I don’t think at all like a big organization HR person, I’m probably an outlier.

    I’ve never liked the term ‘software engineer’, and only a handful of people can legally use that term here in Texas, as ‘engineers’ are required to be licensed.

    I strongly prefer ‘software developer’ because all my software work has been for the purpose of delivering a commercial software product to our management for them to sell.

    There’s a lot more to product development than just ‘programming’, although programming is the core task. I wish colleges did a better job of teaching their students how to deliver a finished product that another person could use.

    Psb aks:

    “If someone’s resume said that they learned how to program on the internet but that they have no programming work experience, would you call them in for an interview?”

    In my case, yes, absolutely, if they can show that they’ve written software that others are using. An application in the Apple App Store is the gold standard, not because Apple products are that special, rather because those apps are curated and will meet a known quality standard. (The demand for iOS programmers here in Austin is huge. Anyone with an non-trivial app in the App Store can get a job inside of about three days here, nobody will care about their actual background.) In our case a Google Playstore app or recognized contributions to a significant open source project would also justify at phone screen and probably an in person interview. That’s as least as good to me as a CS degree with no practical experience.

    In the case of entry level technical degree graduates I want to see some sort of non-coursework self initiated software development project. Often it’s a game, I’ve seen pretty decent chat applications and interesting web applications from the more motivated students. The better the application, the less I care about their gpa.

    I’m not interesting a recent graduate with no self-motivated work, no matter how high the gpa.

    The thing is this. Lots of people major in CS and even work in the field because its a good paying job that is inside, air conditioned, with little or no heavy lifting. Or because their parents told them to major in CS or EE for purely economic reasons.

    If you don’t actually like to develop software, you’ll never be good at it IMHO. Most people simply lack the mental configuration that motivates good developers. That’s why it’ll always be hard to find good software developers.

    This concept also applies to welders.

    One last thought. Every good software developer (or ‘software engineer’, or ‘architect’) is a good programmer.

    That’s why we start every software development interview with a fizz buzz type question, and then ask increasing difficult questions until we feel like we’ve reached the candidate’s comfort level.

    We understand that many good developers can’t just jump up to a whiteboard and write perfect code to reverse a linked list with no notice in an interview situation. But at least they ought to be able to make credible effort.

  22. Michael Moser

    May 4, 2013 @ 12:27 am

    22

    Phil, you site open source projects as reputation builders.

    Do employees really value open source contributions?
    My experience is that management sees them as distractions; things that
    can steal valuable company time;

    Other indications are that open source people are seen as not pliant
    enough; and people with too strong opinions of their own are not valued
    int the market.

    In general I don’t know if I should hide my open source projects on the CV or not.

  23. Michael Moser

    May 4, 2013 @ 5:11 am

    23

    On a practical note; I think the open source movement comes with its culture of sharing and its own set of ideas; the whole thing came out of the Hippy movement (just look at the long beard of Mr. Stallman 😉
    Now with all things cultural there comes the same problem: it is hard to transplant a given culture into the third world; all third world countries come with their own culture, with their own system of values. Western countries can’t be simply transplanted into the third world, it can be adopted, but the process takes time.

  24. Adrian Miu

    May 6, 2013 @ 4:35 pm

    24

    Exceptional programmers are those that have
    1. Incredible high IQ
    2. Determination to learn.

    While you can blame the lack of glut of good software engineers on the lack of determination (there are a lot of programmers who treat their job as any other a 9-to-5 job) the high IQ is actually the big problem. I have an IQ of 130 and, looking at the source code of some open source project, I feel stupid (not because I don’t understand the code but by the elegance of those solutions).

    http://www.iqcomparisonsite.com/iqtable.aspx shows how few people are candidates to become good programmers. Obviously there are people with high IQ that are not interested in programming so the pool shrinks even more.

  25. Bob Van Wagner

    May 6, 2013 @ 8:15 pm

    25

    There are many more good programmers, but toxic is the ground that has developed in the industry for allowing the flowers of intellect and creativity to blossom. Let me get back to that but first allow me to disrupt the topic:

    Why interview? Why cull resumes? Both are LOW INFORMATION. They are very low information. Few humans except newborns with cute puffy faces and babyfat don’t show well in either. But babies can’t code or engineer useful systems or attack human misery using technology.

    Stop interviewing. Dump all the resumes. Fire more often. Hire more often. Somebody who comes in, they want to see you and your work just as much as you want to see how they fit in that garden of delights known as a productive workplace and how they can bloom and with that bloom add to the beauty of your garden.

    So let them actually work for a day, a week with you. Come in the door and follow, as an extra hand, extra eye, extra mind at the shoulder of a key worker. See how it goes. Fire them! Or hire them. Either way a far far better hire than all the lying resumes and all the bipolar social phobic or social gadfly personified interviews.

    The best chefs do this. A sous chef walks into a kitchen, the lead chef says “Follow me!” and that means follow the lead and pitch in. A chef looking for a new position quickly finds one that is compatible with his skills and personality. Many trials, but REAL trials. As we who follow the Food and Cooking shows know, great chefs are not unlike great programmers. A creative and cookbook mix, the clerical mundane processes of programming and engineering are matched by all the rigorous prep work and maintenance of larder and fridge that a master of a kitchen must ever continue, and yet no good restaurant survives without a chef staff committed to lifetime learning and innovation.

  26. Bert

    May 7, 2013 @ 2:33 am

    26

    I recommend reading ‘I am a Bad Programmer’ on Hacker News for relevant perspective. It recounts the travails of an experienced programmer who cannot get past the interview stage eventhough he desperately needs a job. If it’s no longer on the first page, you can use their search box to find it.

  27. Rokahn

    May 10, 2013 @ 2:08 pm

    27

    Seems like there’s two groups posting in these comments–the disenfranchised and the “enfranchised”. The disenfranchised developers describe a system with arbitrary HR departments and companies hiring for quantity over quality. The “enfranchised” founders/recruiters and high-end programmers say it’s very hard to recruit someone with high enough quality to develop challenging products and that Silicon Valley startups pay well and still have trouble finding staff.

    As a technical founder at a challenging startup who is deeply involved in recruiting; for local talent, we expect to pay $90k for a college graduate showing some initiative and $110-200k for senior developers…and it’s very hard to find the right people (I sympathize with the Rackspace gentleman). For remote developers, I’m currently paying about $90k for senior developers (and they probably have a higher quality of living than our local senior developers due to lower costs of living (I’m unlikely to consider an inexperienced person because our product is sufficiently challenging and the coordination overhead makes it unfeasible…other companies may differ). Developers who maintain an open source project relevant to our product are clearly desirable and our current tech lead was found that way–it demonstrates both skill and an understanding of what’s useful (they anticipated what we’d be looking for by developing and maintaining a project we later find useful).

    For projects such as my company’s which can dramatically benefit from exceptional programmers (and many Silicon Valley companies are similar) it remains hard to find good staff, local or remote.

    Yes, recruiting is a deeply flawed process, but that’s the nature of the beast. Also, I find most developers incredibly passive. If finding a great job is so important, why would they not develop a portfolio of (open source) code they can show to a potential employer; why would they not look into themselves to identify technologies and products with which they’re passionate and directly contact relevant companies. Instead, they spam companies without regard for goodness of fit (rarely do I receive a cover letter explaining why a candidate’s skills/interests match our company’s and although we have a publicly available, free product, rarely do I see a candidate try it out) and many use recruiters who will admit to not even superficially understanding the relevant technologies.

    I think there’s a bifurcation between commodity software companies and the commodity developers who work for them…and high-skill companies and the high-skill developers who work for them. We sometimes have to interact with the former companies and although the job titles and CV’s might look similar, they’re worlds apart in ability to get things done. Exceptional developers know many technologies and easily pick up new ones, so high-skill companies don’t require an exact fit (e.g. a developer just needs to know any SQL variant or any JavaScript framework…not our specific variant/framework). Yes, this is implying that “if you’re whining, you’re probably a commodity developer”, but I haven’t met any exceptional developers who have trouble getting work while I haven’t met any high-skill companies who can easily hire for open positions.

  28. Mark Lutton

    May 12, 2013 @ 6:09 pm

    28

    Last week I had to conduct a job interview at my company, which is a very large corporation. We were provided with the corporate instructions for conducting interviews. The interview must be 45 minutes in length. Greet the candidate, explain the interview process, etc.

    30 minutes of the interview must be spent asking the candidate questions to determine whether he/she is a team player, communicates well, has initiative, is good at problem solving etc. Tell me about the most effective presentation you ever gave. Tell me about a time when you had difficulty communicating with a co-worker, and how you resolved that problem. Tell me about a time when you had an idea that was rejected, and how you eventually got the idea accepted. Tell me about a time when you helped a co-worker learn something. Tell me about something you learned on your own initiative.

    No time need be spent determining if the candidate can actually write a program.

  29. Insight247

    May 15, 2013 @ 8:49 am

    29

    Interesting point.

    As the Marketing Manager for a small business, we face difficult decisions when hiring freelance software engineers for our cloud-based recruitment software.

    However the trend that we’re witnessing in the UK is that the cost of acquiring the talent is dropping. The market has become saturated to a certain degree, with our universities churning out capable graduates hand-over-fist.

    I’d point to Mark Lutton’s point about the depth and relevance of the contemporary interview process. Perhaps too much time is being spent on ‘culture fit’ rather than capabilities.

Log in