AI ALIGNMENT FORUM
AF

All of Judd Rosenblatt's Comments + Replies

There Should Be More Alignment-Driven Startups
Judd Rosenblatt36

Thanks, appreciate your wanting these efforts not discouraged!

I agree there's certainly a danger of AI safety startups optimizing for what will appeal to investors (not just with risk appetite but in many other dangerous ways too) and Goodharting rather than focusing purely on the most impactful work.  

VCs themselves tend not to think as long-term as they should (even for their own economic interests), but I'm hopeful we can build an ecosystem around AI safety where they do more. Likely, the investors interested in AI safety will be inclined to think ... (read more)

Reply
There Should Be More Alignment-Driven Startups
Judd Rosenblatt32

Yes, you're right, and most startups do fail. That's how it works! 

Still, the biggest opportunities are often the ones with the lowest probability of success, and startups are the best structures to capitalize on them. This paradigm may fit well to AI safety.

Ideally we can engineer an ecosystem that creates enough that do succeed and substantially advance AI safety.  Seems to me that aggressively expanding the AI safety startup ecosystem is one of the highest-value interventions available right now.

Meanwhile, strongly agreed that AI safety driven startups should be B corps, especially if they're raising money.

Reply
2Zac Hatfield-Dodds
Without wishing to discourage these efforts, I disagree on a few points here: If I'm looking for the best expected value around, that's still monotonic in the probability of success! There are good reasons to think that most organizations are risk-averse (relative to the neutrality of linear $=utils) and startups can be a good way to get around this. Nonetheless, I remain concerned about regressional Goodhart; and that many founders naively take on the risk appetite of funders who manage a portfolio, without the corresponding diversification (if all your eggs are in one basket, watch that basket very closely). See also Inadequate Equilibria and maybe Fooled by Randomness. Technical quibble; "B Corp" is a voluntary private certification; PBC is a corporate form which imposes legal obligations on directors. I think many of the B Corp criteria are praiseworthy, but this is neither necessary nor sufficient as an alternative to PBC status - and getting certified is probably a poor use of time and attention for a startup when the founders' time and attention are at such a premium.