Aug 9, 2010

The Developer Experience Index

I’ve been musing on developer skills, certification rorts, hiring, and a whole lot more recently and I’ve been thinking that it’s about time we had something akin to the Windows Experience Index, but for developers – what I’ve whimsically titled the Developer Experience Index.  A representation of a persons skills and abilities in various areas of development and maybe some way of rating them overall as a developer.

Because my thinking on this is still solidifying, instead of writing down all my thoughts and having a long rambling post that gets misunderstood, I’d instead love for you to have a listen to Episode 24 of Talking Shop Down Under and get a feel for what I’m really talking about. It’s only 24 minutes so it won’t take forever to listen to.

I’m after feedback and thoughts on this to decide if it’s worth pursuing, so even if you don’t subscribe to my awesome podcast (and why not!?), have a listen to this episode and see what you think.  Even better, spread the word about it so that others can talk about it as well and we can get some conversation happening.

Would you be willing to contribute to such a community? Would you be willing to go through a community certification process?  If you are in a position to hire people, would you rather interview people who are rated well by their peers or those who are certified by an organisation that makes money from people attempting certification exams? Am I completely bonkers and this is a really dumb idea? Let me know

P.S. For those wondering (and who haven’t yet listened to the episode) I am aware of the Java Black Belt exams, and while that’s a good approach and shows the potential of this, it’s not quite what I’m after.


  1. Hey, just thought I'd leave a note and say that there was one place you didn't think of where it could be useful...

    At the place of employment, as a standard of skills test for employers potentially hiring.

    However, of course, if someone can find a site with the standard, then they'd have practice to get past it/get a rating.

  2. I have been following a few mailing lists that have been attempting something similar. One is:

    Of course, it is agile specific, and focuses mainly on the XP practices etc. (And to be honest, I find the XP practices fairly easy compared to algorithms and data structures, programming languages, and the larger topic of software engineering)

    A concept of Badges is the latest idea under discussion on that list.

    There are actually a few similar ideas floating around out there, there used to be a web of trust thing called WeVouchFor that supported the idea of vouching for people you have worked for. I think the ruby guys have something similar but I can't remember what it is called. They both have the potential disadvantage of being a bit like an old-boys-network / popularity thing rather than a true measure of competence.

    The programmer competency matrix seems like a more even coverage of developer skills than something that is completely "agile" focused, or .net focused, and of course it is not a certification so you cannot take it to a potential employer or anything.

    Right now, the best method seems to be building up a reputation by writing and contributing to open source software, otherwise, IF you need to prove your competence to someone, the best things are getting a formal education and degree, so you know the basics, algorithms and data structures etc, and beyond that, certifications (they have to be difficult though, and involve a lot of work) that show more specific competence in specific areas, such as programming languages, or software engineering in general (IEEE CSDP for example).

    I suppose the third option would be blogging and speaking at conferences etc, but to be honest that is probably better for coaches than practitioners, as talking about development and actually doing it are two different things.

    But I am always open to other ideas.

  3. One other thing I'd just like to point out - I'm haven't been developing a long time, and I don't know where I'd fit on your "levels" probably closer to 2, than to 5+.

    But you used the example of having people work with an IOC container.

    That's something I get - I can work with, I may not know all the ins and outs, but I've added them to projects, I can use them.

    But you'd rank that as level 5 knowledge or something.

    All I'm trying to say is, if you start pointing at different concepts and say "this one makes you a level 2, this one 3, 4, 5"... somewhere that becomes troublesome.

    While at the same point in time - it would be good to be able to say, "to be at this level of programmer, these are the things we'd expect you to understand - go and research them".

  4. An interesting but potentially long and controversial discussion, but as a starting point and some feedback, I'll throw in a few comments and questions.

    What sort of time investment on the part of the assessee would be required for it to have real value? As you raised during the podcast, the certifications where you really need to put in the greatest effort are those that really mean something. So what would be the value proposition to the developer and industry? Because which ever way you look at it undertaking decent education and certification is a non-trivial process for the individual.

    Devils advocate question: What specific problem would it be solving that's not already covered by various colleges, universities, representative bodies and industry certification programs?

    I really like the idea of submitting a "mini project" for peer review. It really embraces that community/open source concept. I would however be slightly concerned over how you would verify an original piece of work or deal with plagiarism.

    I hadn't seen the programmer competency matrix before, as mentioned by Kurt, but it reads (mostly) like what you might expect from a 'generic' computer science/software engineering degree, except that you can have varying levels of competency on points within the matrix. It's a good template, although as already pointed out, not an actual certification.

    This made me think, perhaps there is an analogy to pilot training. As a pilot you must first have the proper base knowledge of flying (practical instructor led and theory), of which there are various levels of course. That is your starting point. From there you must then be endorsed for particular skills, competencies and aircraft types. Perhaps that analogy could be extended?

    Would I be willing to interview people who are well rated by their peers over those who aren't? That would entirely depend on the credibility and more importantly integrity of the peer review process. If it descended into a self congratulatory knob polishing in-group, then no. I don't think being just peer reviewed is enough, more critically, it would need to be independent and unbiased. I think they would absolutely have to be core principles, otherwise to my mind it would be worthless. However those principles might just also be the differentiator that could make it work and give it credibility.

    A lot of random thoughts and questions I know. I hope I haven't totally missed the point? :)