نوشته زیر کمی طولانی است. اما ارزش مطالعه دارد. مطلب زیر به یک معضل بسیار مهم در تیمهای نرمافزاری اشاره دارد. جایگاه برنامهنویسان خبره. تفاوت برنامهنویسان خبره با سایر برنامهنویسان. اصل مطلب را با گزیدهای از توضیحاتی که خوانندگان بر آن گذاشته بودند در زیر آوردهام. حوصله کنید و آن را تا آخر بخوانید و نظراتتان را برای استفاده دیگران برایم بنویسید. بحث بسیار جالب و چالشبرانگیزی است.
I was having a conversation with a friend of mine (we’ll call him Eric) about my article on rewriting software and he made an astonishing claim. “I think you low-balled the amount that the rewrite should reduce the code size. I’d be willing to bet that I could rewrite almost any piece of software in half the lines of the original. In fact, I’d take a fixed priced bid on a rewrite for anything up to a quarter million lines of code, get it done in a year, meet their coding standards, have 80% test coverage to boot, and do all the work by myself.” That was a bit hard to swallow. A quarter million lines rewritten in a year by one person? IBM once did a research report that indicates that the average developer writes about 10 lines of functional tested code in a day. Here was Eric telling me that he could reliably write tested code at a rate of closer to 1000 lines per day, nearly 100 times that average.
I took it as a bit of a boast, but Eric is really a pretty hard-core guy. He’s built at least six systems in the 100,000 to 250,000 lines of code range and he knows what it takes. He’s in my top 5 developers list and since I’ve worked with him quite a bit I know that he can crank out solid code consistently and at a speed that is fairly unbelievable. While I doubt his claim that he could rewrite most quarter-million line programs in a year (turning them into 125,000 line programs), I don’t doubt that he could do it in two. And I don’t doubt that the code would be a lot better off when he was done.
This brings me to the point of this article, that there is at least an order of magnitude of skill difference between the average programmer and the best programmer, and maybe even two orders of magnitude. I’ve made this observation to quite a few programmers and managers and in listening to people’s answers I’ve noticed the following:
1. Everyone agrees that good developers are many times better than average ones, but people give widely different numbers from 4x to 100x.
2. Everyone agrees that the worst programmers are a net negative (they actually slow a project down).
3. The more complex the type of work the estimator does the higher the multiplier they choose. People who spend all their time writing small scale CRUD apps tend to pick numbers between 4x and 10x, but people working in extremely complex or performance critical domains pick numbers between 10x and 100x.
4. Everyone agrees that larger teams are less effective (on a per person basis), but that you can have up to about five or six people working on a team before it starts to fall off in productivity
5. Most people agree that there are some problems (especially some performance ones) that will NEVER be solved unless you have one really great engineer.
6. Most people agree that the best coders are better at all aspects of the development life-cycle: analysis, design, coding, debugging (however you want to break it down).
7. Most people agree that the code written by the best coders is smaller and contains fewer bugs per line than that written by average coders.
While a few people whose opinions I respect may quibble with some of these points, most people seem to accept them at some level, which begs a question: why aren’t the best coders paid more? A LOT more.
Let’s go back to Eric’s point. Let’s say a company has decided to rewrite some 250,000 line piece of code for some reason (usually because it’s broken), on the one hand they can choose to let their team of 10 average developers work on it for the next two years and end up getting it done and pay about $2.5M in fully loaded costs. Since they are average coders there will be a lot of bugs and while they may improve it somewhat, the improvement will be less than expected. What if they took Eric up on his offer, and let’s say (for the sake of argument) that Eric needs to bring in 2 equally stellar friends (Eric-2 and Eric-3). Anything a team of 10 average coders can do in two years the three Erics can do in one year. If you offered them $1.5M fixed for the whole contract they could walk away at then end of that year with $500k each (which is probably a tad bit more than their current salaries). The odds are that the final product would be better, smaller, faster, and more reliable as well. That’s just the kind of code that people like Eric tend to write. In fact, they HAVE to write better code or they couldn’t become an Eric. People who write bad code spend their lives fixing it. That’s one big reason they’re so slow.
This same kind of logic applies even more for moderately large projects (say 1-2 Million lines of code). A five or six person team of 99.7th percentile developers can write a project like this in less time than any team of 40 average coders. Who wouldn’t gladly pay them each $300,000/year each to get it done?
In practice, this turns out to be pretty hard to do. First, most companies can’t tell a 66th percentile developer from a 99th percentile one (some can’t even seem to tell the difference between a 50th percentile one and a 99th percentile one), so that makes this strategy pretty darn near impossible for them to carry out. Second, many of these companies haven’t the slightest idea how to use a top developer. They find ways to dump them in meetings, bother them with continual interruptions, and drown them in useless unproductive tasks. This is part of the reason they can’t tell them apart: they have an almost insidious ability to bring everyone back down into the big pile that is “average”. In these environments other engineers generally know the great developer is better than everyone else, but they have almost no idea just how much better they really are.
Actually, it’s nearly impossible for anyone to interview for the top one percent. There is too much distinction between programming skill and interviewing skill to have such a refined process. With work you can weed out bad and average engineers, but getting great ones is extremely hard. This means that the only reliable way to find the truly exceptional engineers is to know them. To have worked with them and seen them in action. Since you can’t interview for greatness there are two options left: The first approach is the Paul Graham approach, get them to do a startup for next to nothing and if they turn out to be great you’ll know it (even if the startup fails). The second approach is the buddy system: after being around the block for a while, call up the five best engineers you know and convince them to join you. Then, if you still need a few more, get them to call the five best developers they know. In short order you have a very strong team.
The buddy system has its limits. At most you probably can’t get more than fifteen or so people this way before the diffusion effect has moved you out of the top one percent and into the top ten percent. This assumes you have actually been around enough good companies to START in the top one percent. The bigger problem with the buddy system, though, is how to get those five people to join and what to do with them when you do. One possibility is to create a startup company. Unfortunately, startup companies have no guarantee of success, regardless of how good their engineers are. Lots of companies with mediocre engineers succeed while others with great engineers fail. Coding is only one small piece of the puzzle and many of your top coders know this. Another option would be to form a consulting firm, but consulting is a sales driven activity and just because you actually have a great team of six amazing coders doesn’t mean you can convince some large company to pay you $3M to write them a big piece of software. EVERY consulting company claims to have the world’s best coders. A company that can’t hire the top coders internally won’t be any better at selecting the top coders externally. Unless you have the contract in hand, getting the five best coders you know to join your team will be a bit of a struggle.
So maybe it’s a challenge to bring together a small group of t
op coders and guarantee that they get paid close to what they’re worth, but why can’t a really top coder just demand more from the company they are at? Wouldn’t the company pay a really large sum just to keep them? The answer is almost assuredly no. Let’s say the company is a big company with mostly mediocre coders. In that case they probably have fairly fixed salary structures with limited ability for a particular manager to pay much above or below the standard. Even if the manager would agree to pay $300,000 a year it would be virtually impossible for them to get approval. Also, it’s likely that the manager has only a fairly limited capacity to recognize the skill of the engineer. They may know the programmer is better, but they really can’t quantify how much better so they aren’t going to be able to recognize that skill with a commensurate compensation package.
Small companies can’t do it either. Even if the manager recognizes that the developer is really great, there is a good chance that all the developers in the company are pretty good anyway. So while they may be 10x or 15x better than the average coder overall, they may only by 2x or 3x better than the average coder at the company. Also, if they agree to pay more there is the very real chance that everyone in the company will soon be asking for more, and that’s not something they can afford. It’s better to lose the one great coder than to have to pay the other twenty 25% or 50% more. The last reason the great developer can’t get 3x more than the people around him is that he probably doesn’t know just how great he is. Really great developers know they are good, but very few have the slightest idea how much better they are than their peers.
In the end, this means that really great coders will keep getting paid less than they are worth and average ones will keep getting paid more, so the economic benefits of great skill will go primarily to the companies with the best employees and not to the employees themselves. Is it fair? No, but it is how the market works. Who ever said that fair can or should be factored into the equation. Sure I’d love to get that phone call asking me to get the five best coders I can find to write a million line application for $3M dollars, but I’m certainly not going to waste any time sitting by the phone waiting for it to ring!
برخی از نظرات جالب در مورد این نوشته:
Peter Says:
I’m a Ph.D student at MIT. What I’m finding is that the top people here do one of three things. One group will start businesses (or consulting work) where they can be paid on par with productivity. Another, but shrinking, group will go into acadamia. This group is shrinking because the odds of tenure are pretty low. Others quit programming, and work in the finance/management (venture capital, management consulting, investment banking, etc.). Very few of the top people end up in industry. This is moving back further and further — this used to happen mostly to people in graduate school or even industry, but now, increasingly, technical undergrads no longer want to go into engineering. For high levels of intelligence, the payoff just isn’t there, in the way that it is in financial and management sectors.
Industry acts as an insurance company. It compresses the pay range. If you’re unproductive and generally suck, your salary goes up to some minimum. If you’re highly productive, it drops, often by a factor of 10x. This absorbs risk — you don’t know which of the things you’ll do will have a payoff (often, good engineers will work on unprofitable projects), which is nice. It does, however, strongly discourage great people for working in industry.
Also, the industry keeps complaining about a labor shortage, while employees about a job shortage. I finally got to the bottom of this. Industry is finding a shortage of good people, whereas crappy people can’t get jobs. Universities, with dropping enrollment in engineering programs, are lowering standards so that more people can train to be engineers. Poor engineers enter the job market, can’t get jobs, and when they do, they lower the overall pay range. As a result, good people see low wages and a job shortage, and go into other fields, increasing the problem.
Hildo Says:
The problem with this is that the code written by excellent programmers like Eric will not always be understandable by coders in the lowr 50% of the bell curve. The code will be concise and well-designed, but as a result not like like the typical “write it all out” code that typical programmers churn out.
This matters because the code will have to maintained and enhanced by such programmers for the remainder of its life, which is likely to be far longer and far more costly than the development time.
I see this all the time at work ($VERYBIGCOMPANY). Programs that need to survive for a long time (5-10 years) are on purpose given to large groups of mediocre programmers, so the code will be at their level and can be maintained by them. Short-term high-priority projects are given to really great programmers, but if theu survive longer than expected, they often have to be rewritten from scratch 2-3 years down the road when requirements have changed fundamentally and the original developer is working on another important project.
Chuck Says:
Laws of the universe like “diminishing returns” and Heisenberg-ian observation prevention paradoxes prevent the optimal team from working on a system that is optimized to their skillset. I worked on a project that went through several iterations on several teams, ensuring that the talented developers stayed on and the bottom-feeders were weeded out. I call it Darwinistic Developing. Of course, this only works with highly flexible time constraints (ie. research).
Obviously, this philosophy has its perks and plugs, but one of the more fantastic advantages is that the top 97 percentile are quite easy to spot. Right out of the unit testing stage, it is pretty clear who is holding the project back. After a few iterations, the project was error-free to a very high degree and the team was a highly-cohesive and productive unit.
Enter the next project, and suddenly the team is no longer optimal as there are no two projects exactly alike (even if just the client changes but the general project scope does not). The result of this research was that the process of evolving and optimal development team results in a team that is optimized to a single case. When the case changes, the optimization fails and productivity de-evolves.
Small team development, even single-person team development will always be hindered by one or more team members who detract from the efficiency of the development process. It is certain the one or more members of the team deserve to be paid the lion’s share of the wages and the rest earn a proportion of that amount, but it is nearly impossible to always determine the 97 percentile developer(s).
That is where another law comes into play: the law of averages, providing everyone on a team with an average wage based on overall productivity. Even with a team of one, where the average should be 1:1, problems in development like requirement and scope drift, challenges in implementation on a foreign platform or other unknowns will prevent even a 97 percentile developer being paired with a project that is perfectly suited to their skills.
Lee Says:
You missed the main reason in this article, although you already explained it (in part) in a previous article when you said:
“Mommy, mommy, Bobby has two more gigs of RAM than I do and he won’t share.” Pretty soon managers start liking conformity more and more. Same salary, same environment, same blue suit.
The rest of the reason is that if they paid the coder more, then the manager (considering him/herself MORE important than a mere “coder”) would want to make slightly more, etc. on “up” the proverbial ladder.
Ben Says:
I agree fully with you, especially with the problem (ability motivation) has to recognize difference in ability. You overlook another important factor though — the ability to translate programming ability into money. For many companies this may not come that often, or the difference between good and stellar results isn’t that high. For them, it’s not worth paying any coder ‘what they’re worth’ at least in the long term, because they can’t profit from it. I think this blends in with the start-up problem. You might be a great coder, but can you code something great that is worth a large chunk of change to someone?
مرجع: http://codecraft.info/index.php/archives/78
گزیده:
Love cannot be found where it doesn’t exist, nor can it be hidden where it truly does.
– David Schwimmer, in the movie “kissing a fool”
