The Core Developer Apprenticeship Program

The opinions in this article are my own and do not represent any "official" sentiment by either the Ethereum Foundation or the Ethereum Core Development Team.

We need more people working on the core Ethereum protocols.  The roadmap for Stateless Ethereum has solidified in significant ways over the last months.  Recent work on fleshing out the high level structure of State Expiry and the underlying Verkle Trie and Address Space Expansion (ASE) have filled in a gap on the roadmap that was previously unsolved.  

There is a big difference between knowing what needs to be done and doing it.  All three of these are deep changes to the protocol and require significant prior knowledge and ability to execute them.  All of the capable people/teams are already working on something equally important.  This has been a growing problem in the Eth1.x space.

Over the last two years we have gotten better organized.  More often than not we are moving as a coordinated group.  We've come to agreement on not only major protocol improvements, but also addressing long standing protocol problems.  We have more work queued up than at any prior point in Ethereum's history and we need more people capable of doing this work.  We don't have enough "Core Developers".

What is a "Core Developer"?

The term "Core Developer" is a bit of a loaded term.  I say this because there's no roster or official list of who is or is not a core developer.  The lack of a formal definition is not exactly something that was intentionally arrived at, but I believe it is something we preserve because it works.  Often, those that want power are those you actively want to keep from having it.  The same may be true for the "core developer" title.  If the title becomes official then with it will come power and those that will actively seek it.

An interesting case study on this is a recent proposal to govern the gas limit using a token.  I'll state very clearly that I am firmly against this proposal for multiple reasons, but one in particular was very interesting in the context of "who is a core developer".  The proposal included language which would distribute the token to the set of core developers with the expressed intention of having them use the token to vote on the gas limit.

On the surface this might seem reasonable.  There's a group known as the core developers.  They tend to be very knowledgeable about the protocol.  Let's give them a token they can use to govern with.  Simple right?

Nope...

This is exactly the kind of degenerative stuff that would naturally spill out of making the title of "core developer" official.  By granting tokens which have economic value to the people who officially carry this title, you also introduce perverse incentives for acquiring this title.  In this case, if this proposal were to have been accepted, anyone carrying this title would be entitled to receiving these governance tokens.  Tokens which have economic value.  Tokens which put a direct monetary value on holding the title of "core developer".

For me, one of the main reasons that I'm strongly opposed to defining membership into this group is to avoid this type of situation.  The best way I know how to keep the group populated with the right people, is to try and maintain the current status quo of people being accepted into the group by informally defined social rules based on their ability to contribute positively to the protocol.

Privilege and Systemic Discrimination

I'm a caucasian male in my late 30s from a privileged background living in Boulder CO.  I'm very aware of systemic discrimination and privilege but largely from the side of them benefiting me.  I'm going to take a small moment to acknowledge these things in the context of the core developers.

The core development team is predominantly caucasian males in their 20s and 30s.  We are not a diverse group.  One unfortunate but real side effect of the manner by which core developer status is defined is that it likely reinforces our lack of diversity.

The Next Generation of Core Developers

The status quo today is that there is a group of "us" who are the "core developers" and we run this multi-billion dollar financial platform called Ethereum.  The ambiguous nature of the title provides us with some security.  Joining this group requires some amount of significant investment of time and expertise.  We have a roadmap with more work to be done than can reasonably be executed by the existing group.  The natural conclusion here is that we need to grow the group.

"How can we get more people working on the protocol?".

Enter the "Core Developer Apprenticeship Program" aka CDAP.  The original idea came about during a lunch conversation with some of my team while we discussed this very problem.  The problem seemed quite similar to the general problem of "how do we hire the right engineers".  There is no shortage of opinions on how to effectively interview for technical roles.  In fact there are entire companies built around this topic.

I've done a lot of technical interviewing and hiring in my career.  I've been on both sides of the table.  I'm lucky that interviewing is a skill I excel at.  The companies that have hired me are also lucky that I'm actually good at my job because the two things are not necessarily related.  The ability to perform well in a technical interview isn't necessarily indicative of whether the candidate will actually be any good at the job.  Over the years, I've settled on the following approach to interviewing.

• Talk with them about who they are, what the job is and why they want to do it.  Make sure they seem like someone you might like to work with.  Make sure you both have clearly stated expectations about what the job looks like and how your working relationship will work.
• Make sure they can code their way out of a wet paper bag.  Anything from looking at existing github contributions to pairing on a small exercism.io problem.  This isn't a determination of whether they can do the job, but rather a filter to catch woefully under qualified candidates.
• Give them a chance to do the actual job on a 3 month contract-to-hire.

You may notice that there is no part of this process that tries to measure their technical ability beyond ensuring that it is above some very basic "low water" mark.  The real "interview" happens over the course of the contract where they do actual work.  This is exactly how CDAP was designed.  The program aims to mirror what it is actually like to be a core developer.

• More often than not you will choose what you want to work on.
• Being "taught" something is rare.  The default is figuring it out for yourself.
• Being able to communicate your ideas in writing is everything.

A Beautiful Nightmare

On May 13th 2021 we opened up the program for applications.  Over the next two weeks we got a total of 379 applications.  We only had budget to fund four people.  It was a beautiful nightmare to find myself in.  It was amazing to see such a massive show of interest, and so incredibly sad to have to say "no" to so many.

The subtle part is that the program is wide open for anyone to participate, regardless of whether we "accept" them.  Acceptance is simply the promise of funding.  Everything else about the program is fully open to anyone who wishes to show up and do the work.  This closely mirrors core protocol development itself which is open to anyone who wants to participate.  In some ways it makes things accessible to anyone willing and able to contribute.

I'm glad to be able to offer this as an avenue for people to get involved. I also recognize the same systemic discrimination against the less privileged at play.  The program being "open" will mean that more people will participate than would have otherwise, and this should have a positive impact on the protocol.  But it also has the potential to reinforce the existing lack of diversity among the core developers.  To say this keeps me up at night would be an overstatement.  I try to keep it in the front of my mind.  I know there's no single sweeping solution, but I also know I'm in a position to influence and implement real change and so I have a responsibility to try.

What comes next?

The first cohort of CDAP is just getting off the ground.  Selecting the candidates to fund was an exercise in eating an elephant.  There is no quick way to properly review 379 applications, other than one at a time.  Making choices that have significant impact on other people's lives is taxing.  Making that kind of choice 379 times over the course of two weeks put me at a deficit of decision points.

I'm cautiously optimistic.  It's actually very rare for me to just be plain "optimistic" so maybe you could say I'm at my peak level of "optimism".  Not to be confused with Optimism which is also cool. I try hard to keep my expectations in check with reality.  Things are looking quite good.  

We've been able to fund more candidates than we initially expected.  That feels good.  

We've been able to engage multiple candidates independent of funding.  How do I ensure that every one of them gets a real chance?  How do I give them the best chance to find a permanent and sustainable place in this ecosystem?

I try to ignore that in just a few months I'm going to be faced with more of the same tough decisions.  Who do we accept into the second cohort? How do I balance rewarding those that are showing up and doing the work with the desire to push back against the systemic discrimination that such rewards reinforce?  How do I avoid spreading myself too thin?  How do I support the successful candidates in finding a sustainable long term home doing this work?  Is this my job now? What if they figure out that I'm just making this all up as I go?

This remains the best job I expect to ever have.  I still haven't lost sight of that piece.  I would have permanently burned myself out years ago if this was "just a job" to me.  I'll be happy if I help even just one candidate find that for themselves.  A job free of all the rat-race bullshit where you get to do rewarding work for a mission you believe in.  A job that's really difficult to explain exactly what it is that you do.  A job that I'd probably be doing whether you paid me for it or not.

Do you want to do this type of work?  Want to tell me about it? piper@ethereum.org