Jamie Bernardi

Earlier this year, I changed careers from a full-time technical AI safety research trajectory to full-time AI safety field building [1].

I’m writing this to share the considerations I made, for a few reasons:

1. Potential considerations for others trying to decide whether to work on community building for AI safety
2. Expose my reasoning to hear thoughts from others or blindspots
3. General interest / letting others see what careers look like; a human account of what (my version of) making career changes for impact looks like

Past and present

My technical career path: 2015-2021

It’s worth sharing a little context about what I’d done in the past. Before deciding to run the AGI safety fundamentals programme full time, I:

• Studied Physics at Cambridge 2015 – 2019. I was pretty involved with Effective Altruism throughout that time. I learned deep learning in my final year.
• Was a Machine Learning Engineer from 2019-2021 (from 2020, I split my time 50-50 between ‘product ownership’ and engineering). I learned a bunch of reinforcement learning on the side during the covid lockdowns.
• Simultaneously participated in the virtual AI safety camp 2021 (still during lockdown). I worked with Michael Cohen (Future of Humanity Institute) on his pessimistic RL agents.
• Got a Long Term Future Fund grant to continue with that project full time, and quit my MLE job (Sept 2021).

I started working on the AGI safety fundamentals courses in 2022

Before the grant period was up for the research project, I was pitched on taking on the AGI safety fundamentals (AGISF) programme. I’d been running a local AIS paper reading group in Cambridge from 2021, and that was where the AGISF programme happened to have been based (by Dewi Erwan).

I wish I could say I immediately had a vision that this would be a highly impactful path for me to go down, but the truth is there was some uncertainty about the capacity for the programme to run if I hadn’t agreed to help out. On that basis, it seemed worth spending 50% of my time on it for 3-6 months. It seemed clear that the community, network and knowledge gained by its participants would easily replace the lost time in my own research, if just one of them went on to counterfactually get a project working on AIS 6 months sooner.

Deciding to do field building full-time

Emotional processing

I’ll lay out the cold, hard considerations I made around impact next, but it’d be wrong not to talk about the emotional aspect of changing careers, too.

I didn’t find it a particularly easy decision to even temporarily pause my research, to run AGISF 2022. Whilst I knew that skills like management and entrepreneurship were important, I had spent years intrinsically valuing hard skills like maths & computer science. I didn’t particularly feel like the new skills I’d be learning and exercising were valuable or like something I wanted.

I think there’s also an element of identity involved with a big career change like that. I had identified as a nerdy programmer or engineer, and again intrinsically valued the image of myself as an engineering whiz as that’s what I’d admired others for.

I was a little worried about the changes it would imply for my personality; people who I talked to about that would say “you’d still be you! It’s only your work that’s changing”. That doesn’t feel quite right to me. The industry you’re in dictates the people you associate with, and the media you consume, thus could significantly change your outlook on topics. I’d love to see that measured, or something. (What % of things you thought 5 years ago do you still agree with? Have you had a major career change in that time?)

All in all, I think it’s hard to change your goals away from the goals you already have. The Effective Altruism community certainly helps with that, since the overarching goal of reducing AI x-risk held firm, and I could keep most of my friends even though I was switching industry.

People’s reactions to the change

Speaking to friends outside of EA they found it hard to understand why I was even considering leaving my technical job if I knew I enjoyed it. I discuss some arguments below why I still chose to go through it.

Within the EA network, the reception was more understanding of why I was considering the change. However, some people felt that someone with a technical skillset shouldn’t be abandoning that path to do meta work. I think that exposed an interesting bias against informed people doing community building, for me. This idea is somewhat adjacently explored in this post; technical work seems to be the more highly-valued work in EA, but I think there are arguments below justifying that the path I’m on now is more impactful (for me and my disposition).

Emotionless processing: crunching the considerations

These were the considerations I made around the relative impact of the 2 roles, my fit for each of them, and what they’d imply for my long term career.

Pro: multiplier effect clearly checks out. The AGI safety fundamentals programme was better for me to have ran than to have spent 4 more months on my project (incidentally, it was also looking like it’d be hard to get the project to work, at that time). The programme had run over the summer, and I had a few anecdotal cases in mind that had led to plausibly-counterfactual work being done on alignment, at least replacing my contributions.

Pro: short(er) feedback loops. Some of the positive feedback loops in top-of-the-chain field building are shorter. It can take months-years to complete a paper, and you don’t find out for a while if it’s accepted at a journal. Running an experiment in the middle of a field building programme is short and you get to find out if you’ve substantially helped someone with their career within months.

Uncertainty: am I as ‘good’ at field building as the next person? I hadn’t felt particularly entrepreneurial up til that point, but I had gravitated towards people-leadership in general. But what skills does it even take to do AIS field building?

Uncertainty: is my comparative advantage in research or field building? My research so far hadn’t been super fruitful, and my particular project felt like it was grinding to a halt. I think if I’d kept trying and persevering with research I could have eventually made progress on it, but I clearly wasn’t a 1 in a billion mathematician so I felt I didn’t have a lot to lose there.

Relief: I could return to industry if needed. The engineering skillset I had is valuable in industry, and I could probably return to an engineering job with a few months of refreshing my knowledge (note: make sure I keep 6+ months of runway). In terms of going back into research, I had been out of academia for a few years already so I wasn’t losing more by continuing to stay out of it.

Pro: exploration is good and I am young enough. At 25, I still felt I had time to go hard on one path later.

Pro: there are not many examples of technical people working on serious AIS field building. Before AGISF, I claim there were not many examples of field building being done. I’m not sure why, but one hypothesis I have is that there’s a lack of technical people feeling it is their job to do it. I think the effect of that is nobody doing major work in AI safety field building has any technical expertise. I’m still developing my thoughts on how important it is that I have a technical background, but I do find it easier to network with people I had some connection to back in my technical days, understand their work (mostly), and speak to them on their level.

In the end, I put most of the weight on the replaceability argument. There are few people with a technical skillset and experience in the AIS field that are willing to do community building, and I expected I could do a good job of counterfactually replacing my own technical contributions with a graduate of AGISF quite quickly.

But not everyone should go into field building (obviously)

I don’t want this post to advocate for everyone technical to go into field building, using the multiplier effect as their justification. I’m not sure where to draw the line on who should or shouldn’t, but the dilemma is briefly discussed in point 4 in this post on the meta trap (now with a recent update from Rohin): “at some point you have to get object level”. I feel obliged to make this point as the object of my work now is to make sure at least some people stick at the technical work.

There are some circumstances where you might make the same choice as me. I feel I had a particularly good opportunity with the already-established AGI safety fundamentals programme, to take that on and build a top-of-the-chain onboarding programme into the field of AI safety. I also felt particularly well-suited to work that involves working with and managing people, as well as having product ownership experience. You might find similarly good opportunities and good fit for a project, in which case, great!

Another way of viewing this is that the best field building interventions are produced by empirical researchers creating examples of misalignment and alignment strategies that make people take misalignment seriously, and feel compelled to contribute to the field, respectively. My job is just to platform that work and expose it to newcomers in a systematic way that helps them break into the field.

My main mistake: splitting my time for too long

For a while, I tried to split my time between doing research and running the programme. Trying to do that was likely a mistake, for me at that time.

It was hard to focus on getting really good at one of the career paths. It was also hard to immerse myself in media and information related to either career, and instead I got a smattering of both.

As a result of that experience, I usually recommend people don’t split their time between two quite different roles which require intensity. I’d likely make an exception if one of the roles was: temporary, relatively low-stress, and impactful.

Potentially this would have been fine if I had been better at time management. One reason this was difficult for me to get immersed in either role is I found it quite difficult to draw solid lines in my day around which things I work on when, especially if something from the other job felt like it was ‘on fire’.

Thanks!

Feel free to contact me if you have any feedback on any of my claims, or questions related to this post.

---

[1] I think ‘field building’ is quite a fuzzy term. What I do feels like slightly more than community building, which I simply see as “people meeting each other and maybe developing their ideas”. You might claim field building should be reserved, as a term, for researchers who are defining the alignment research agenda. I am somewhere in between.