The Nonlinear Library: Alignment Forum
The Nonlinear Fund
Subscribe
The Nonlinear Library allows you to easily listen to top EA and rationalist content on your podcast player. We use text-to-speech software to create an automatically updating repository of audio content from the EA Forum, Alignment Forum, LessWrong, and other EA blogs. To find out more, please visit us at nonlinear.org
Available episodes
Newest first
Oldest first
2 days ago
AF - EAI Alignment Speaker Series #1: Challenges for Safe and Beneficial Brain-Like Artificial General Intelligence with Steve Byrnes by Curtis Huebner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EAI Alignment Speaker Series #1: Challenges for Safe & Beneficial Brain-Like Artificial General Intelligence with Steve Byrnes, published by Curtis Huebner on March 23, 2023 on The AI Alignment Forum.
A couple months ago EleutherAI started an alignment speaker series, some of these talks have been recorded. This is the first instalment in the series. The following is a transcript generated with the help of Conjecture's Verbalize and some light editing:
Getting started
1 CURTIS00:00:22,775 --> 00:00:56,683Okay, I've started the recording. I think we can give it maybe a minute or two more and then I guess we can get started. I've also got the chat window as part of the recording. So if anyone has something they want to write out, feel free to put that in. Steve, you want to do questions throughout the talk, or should we wait till the end of the talk before we ask questions?
2 STEVE00:00:59,405 --> 00:01:09,452Let's do throughout, but I reserve the right to put people off if something seems tangential or something.
3 CURTIS00:01:10,200 --> 00:01:12,101Awesome. All right, cool. Let's go with that then.
10 STEVE00:02:02,246 --> 00:21:41,951
The talk
All right. Thanks, everybody, for coming. This is going to be based on blog posts called Intro to Brain-Like AGI Safety. If you've read all of them, you'll find this kind of redundant, but you're still welcome to stay. My name is Steve Byrnes and I live in the Boston area. I'm employed remotely by Astera Institute, which is based in Berkeley. I'm going to talk about challenges for safe and beneficial brain-like Artificial General Intelligence for the next 35 minutes. Feel free to jump in with questions.
Don't worry, I'm funded by an entirely different crypto billionaire. .That joke was very fresh when I wrote it three months ago. I need a new one now.
Okay, so I'll start with—well, we don't have to talk about the outline. You'll see as we go.
General motivation
Start with general motivation. Again, I'm assuming that the audience has a range of backgrounds, and some of you will find parts of this talk redundant.
The big question that I'm working on is: What happens when people figure out how to run brain-like algorithms on computer chips? I guess I should say “if and when”, but we can get back to that. And I find that when I bring this up to people, they they tend to have two sorts of reactions:
One is that we should think of these future algorithms as “like tools for people to use”.
And the other is that we should think of them as “like a new intelligent species on the planet”.
So let's go through those one by one.
Let’s start with the tool perspective. This is the perspective that would be more familiar to AI people. If we put brain-like algorithms on computer chips, then that would be a form of artificial intelligence. And everybody knows that AI today is a tool for people to use.
So on this perspective, the sub-problem I'm working on is accident prevention. We want to avoid the scenarios where the AI does something that nobody wanted it to do—not the people who programmed it, not anybody. So there is a technical problem to solve there, which is: If people figure out how to run brain-like algorithms on computer chips, and they want those algorithms to be trying to do X—where X is solar cell research or being honest or whatever you can think of—then what source code should they write? What training environment should they use? And so on. This is an unsolved problem. It turns out to be surprisingly tricky, for some pretty deep reasons that mostly are not going to be in the scope of this talk, but you can read the series.
This slide is the bigger picture of that. So if we want our awesome post-AGI future, then we want to avoid, y'know, catastrophic accidents where the AI gets out of control and self-replicates around the Intern...
playlist_add
3 days ago
AF - The space of systems and the space of maps by Jan Kulveit
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The space of systems and the space of maps, published by Jan Kulveit on March 22, 2023 on The AI Alignment Forum.
When we're trying to do AI alignment, we're often studying systems which don't yet exist. This is a pretty weird epistemic activity, and seems really hard to get right. This post offers one frame for thinking about what we're actually doing when we're thinking about AI alignment: using parts of the space of maps to reason about parts of the space of intelligent systems.
In this post, we:
Introduce a simple model of the epistemic situation, and
Share some desiderata for maps useful for alignment.
We hope that the content is mostly the second kind of obvious: obvious once you see things in this way, which you maybe already do. In our experience, this comes with a risk: reading too fast, you may miss most of the nuance and useful insight the deceptively simple model brings, or come away with a version of the model which is rounded off to something less useful (i.e. "yeah, there is this map and territory distinction"). As a meta recommendation, we suggest reading this post slowly, and ideally immediately trying to apply the model to some confusion or disagreement about AI alignment.
The space of systems and the space of maps
Imagine the space of possible intelligent systems:
Two things seem especially important about this space:
It’s very large; much larger than the space of current systems.
We don’t get direct epistemic access to it.
This is obviously true of systems which don’t currently exist.
In a weaker sense, it also seems true of systems which do exist. Even when we get to directly interact with a system:
Our thinking about these parts of the space is still filtered through our past experiences, priors, predictive models, cultural biases, theories.
We often don’t understand the emergent complexity of the systems in question.
If we don’t get direct epistemic access to the space of systems, what are we doing when we reason about it?
Let’s imagine a second space, this time a space of “maps”:
The space of maps is an abstract representation of all the possible “maps” that can be constructed about the space of intelligent systems. The maps are ways of thinking about (parts of) the space of systems. For example:
Replicable descriptions of how a machine learning model works and was trained are a way of thinking about that model (a point in the space of intelligent systems).
An ethnographic study of a particular human community is a way of thinking about that community (another point in the space of systems).
The theory of evolution is a way of thinking about evolved creatures, including intelligent ones.
Expected utility theory is a way of thinking about some part of the space which may or may not include future AI systems.
Historical analysis of trends in technological development is a way of thinking about whichever parts of the space of intelligent systems are governed by similar dynamics to those governing past technological developments.
When we’re reasoning about intelligent systems, we’re using some part of the space of maps to think about some part of the space of intelligent systems:
Different maps correspond to different regions of the space of intelligent systems.
Of course, thinking in terms of the space of systems and the space of maps is a simplification. Some of the ways that reality is more complicated:
The space of systems looks different on different maps.
Maps can affect which parts of the space of systems actually get developed.
Maps are themselves embedded in the space of systems.
Which maps and systems actually exist at a given time is evolving and dynamic.
AI will play a big role in both the space of maps and the space of systems.
We think that the space of systems and the space of maps is a useful simplification which helps us to think ...
playlist_add
4 days ago
AF - Truth and Advantage: Response to a draft of "AI safety seems hard to measure" by Nate Soares
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Truth and Advantage: Response to a draft of "AI safety seems hard to measure", published by Nate Soares on March 22, 2023 on The AI Alignment Forum.
Status: This was a response to a draft of Holden's cold take "AI safety seems hard to measure". It sparked a further discussion, that Holden recently posted a summary of.
The follow-up discussion ended up focusing on some issues in AI alignment that I think are underserved, which Holden said were kinda orthogonal to the point he was trying to make, and which didn't show up much in the final draft. I nevertheless think my notes were a fine attempt at articulating some open problems I see, from a different angle than usual. (Though it does have some overlap with the points made in Deep Deceptiveness, which I was also drafting at the time.)
I'm posting the document I wrote to Holden with only minimal editing, because it's been a few months and I apparently won't produce anything better. (I acknowledge that it's annoying to post a response to an old draft of a thing when nobody can see the old draft, sorry.)
Quick take: (1) it's a write-up of a handful of difficulties that I think are real, in a way that I expect to be palatable to a relevant different audience than the one I appeal to; huzzah for that. (2) It's missing some stuff that I think is pretty important.
Slow take:
Attempting to gesture at some of the missing stuff: a big reason deception is tricky is that it is a fact about the world rather than the AI that it can better-achieve various local-objectives by deceiving the operators. To make the AI be non-deceptive, you have three options: (a) make this fact be false; (b) make the AI fail to notice this truth; (c) prevent the AI from taking advantage of this truth.
The problem with (a) is that it's alignment-complete, in the strong/hard sense. The problem with (b) is that lies are contagious, whereas truths are all tangled together. Half of intelligence is the art of teasing out truths from cryptic hints. The problem with (c) is that the other half of intelligence is in teasing out advantages from cryptic hints.
Like, suppose you're trying to get an AI to not notice that the world is round. When it's pretty dumb, this is easy, you just feed it a bunch of flat-earther rants or whatever. But the more it learns, and the deeper its models go, the harder it is to maintain the charade. Eventually it's, like, catching glimpses of the shadows in both Alexandria and Syene, and deducing from trigonometry not only the roundness of the Earth but its circumference (a la Eratosthenes).
And it's not willfully spiting your efforts. The AI doesn't hate you. It's just bumping around trying to figure out which universe it lives in, and using general techniques (like trigonometry) to glimpse new truths. And you can't train against trigonometry or the learning-processes that yield it, because that would ruin the AI's capabilities.
You might say "but the AI was built by smooth gradient descent; surely at some point before it was highly confident that the earth is round, it was slightly confident that the earth was round, and we can catch the precursor-beliefs and train against those". But nope! There were precursors, sure, but the precursors were stuff like "fumblingly developing trigonometry" and "fumblingly developing an understanding of shadows" and "fumblingly developing a map that includes Alexandria and Syene" and "fumblingly developing the ability to combine tools across domains", and once it has all those pieces, the combination that reveals the truth is allowed to happen all-at-once.
The smoothness doesn't have to occur along the most convenient dimension.
And if you block any one path to the insight that the earth is round, in a way that somehow fails to cripple it, then it will find another path later, because truths are interw...
playlist_add
4 days ago
AF - the QACI alignment plan: table of contents by Tamsin Leake
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: the QACI alignment plan: table of contents, published by Tamsin Leake on March 21, 2023 on The AI Alignment Forum.
this post aims to keep track of posts relating to the question-answer counterfactual interval proposal for AI alignment, abbreviated "QACI" and pronounced "quashy". i'll keep it updated to reflect the state of the research.
this research is primarily published on the Orthogonal website and discussed on the Orthogonal discord.
as an introduction to QACI, you might want to start with:
a narrative explanation of the QACI alignment plan (7 min read)
QACI blobs and interval illustrated (3 min read)
state of my research agenda (3 min read)
the set of all posts relevant to QACI totals to 74 min of reading, and includes:
as overviews of QACI and how it's going:
state of my research agenda (3 min read)
problems for formal alignment (2 min read)
the original post introducing QACI (5 min read)
on the formal alignment perspective within which it fits:
formal alignment: what it is, and some proposals (2 min read)
clarifying formal alignment implementation (1 min read)
on being only polynomial capabilities away from alignment (1 min read)
on implementating capabilities and inner alignment, see also:
making it more tractable (4 min read)
RSI, LLM, AGI, DSA, imo (7 min read)
formal goal maximizing AI (2 min read)
you can't simulate the universe from the beginning? (1 min read)
on the blob location problem:
QACI blobs and interval illustrated (3 min read)
counterfactual computations in world models (3 min read)
QACI: the problem of blob location, causality, and counterfactuals (3 min read)
QACI blob location: no causality & answer signature (2 min read)
QACI blob location: an issue with firstness (2 min read)
on QACI as an implementation of long reflection / CEV:
CEV can be coherent enough (1 min read)
some thoughts about terminal alignment (2 min read)
on formalizing the QACI formal goal:
a rough sketch of formal aligned AI using QACI with some actual math (4 min read)
one-shot AI, delegating embedded agency and decision theory, and one-shot QACI (3 min read)
on how a formally aligned AI would actually run over time:
AI alignment curves (2 min read)
before the sharp left turn: what wins first? (1 min read)
on the metaethics grounding QACI:
surprise! you want what you want (1 min read)
outer alignment: two failure modes and past-user satisfaction (2 min read)
your terminal values are complex and not objective (3 min read)
on my view of the AI alignment research field within which i'm doing formal alignment:
my current outlook on AI risk mitigation (14 min read)
a casual intro to AI doom and alignment (5 min read)
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
playlist_add
4 days ago
AF - Some constructions for proof-based cooperation without Löb by James Payor
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some constructions for proof-based cooperation without Löb, published by James Payor on March 21, 2023 on The AI Alignment Forum.
This post presents five closely-related ways to achieve proof-based cooperation without using Löb's theorem, and muses on legible cooperation in the real world.
(Edit: maybe they're closer to just-use-Löb's-theorem than I originally thought! See this comment. If these constructions somehow work better, I'm more confused than before about why.)
I'm writing this as a follow-up to Andrew Critch's recent post, to share more of my perspective on the subject.
We're going to dive straight into the weeds. (I'm planning to also write a more accessible explainer post soon.)
The ideas
Idea #1: try to prove AB
I claim the following are sufficient for robust cooperation:
A↔□(AB)
B□A
A tries to prove that AB, and B tries to prove A. The reason this works is that B can prove that A□A, i.e. A only cooperates in ways legible to B. (Proof sketch: A↔□X□□X↔□A.)
The flaw in this approach is that we needed to know that A won't cooperate for illegible reasons. Otherwise we can't verify that B will cooperate whenever A does.
This indicates to me that "AB" isn't the right "counterfactual". It shouldn't matter if A could cooperate for illegible reasons, if A is actually cooperating for a legible one.
Idea #2: try to prove □AB
We can weaken the requirements with a simple change:
A□(□AB)
B□A
Note that this form is close to the lemma discussed in Critch's post.
In this case, the condition □AB is trivial. And when the condition activates, it also ensures that □A is true, which discharges our assumption and ensures B is true.
I still have the sense that the condition for cooperation should talk about itself activating, not A. Because we want it to activate when that is sufficient for cooperaion. But I do have to admit that □AB works for mostly the right reasons, comes with a simple proof, and is the cleanest two-agent construction I know.
Idea #3: factor out the loop-cutting gadget
We can factor the part that is trying to cut the loop out from A, like so:
A□X
B□A
X↔□(XB); or alternatively
X↔□(□XB)
This gives the loop-cutting logic a name, X. Now X can refer to itself, and roughly says "I'll legibly activate if I can verify this will cause B to be true". The key properties of X are that □X□B, and $\Box (\Box X \rightarrow \Box B)
Like with idea #2, we just need A to reveal a mechanism by which it can be compelled to cooperate.
Idea #4: everyone tries to prove □methem
What about three people trying to cooperate? We can try applying lots of idea #2:
A□(□AB∧C)
B□(□BA∧C)
C□(□CA∧B)
And, this works! Proof sketch:
Under the assumption of □C:
A□(□AB∧C)□(□AB)
B□(□BA∧C)□(□BA)
A and B form a size-2 group, which cooperates by inductive hypothesis
□CA∧B, since we proved A and B under the assumption
C and □C follow from (2)
A and B also follow, from (2) and (3)
The proof simplifies the group one person at a time, since each person is asking "what would happen if everyone else could tell I cooperate". This lets us prove the whole thing by induction. It's neat that it works, though it's not the easiest thing to see.
Idea #5: the group agrees to a shared mechanism or leader
What if we factor out the choosing logic in a larger group? Here's one way to do it:
A□X
B□X
C□X
X↔□(□XA∧B∧C)
This is the cleanest idea I know for handling the group case. The group members agree on some trusted leader or process X. They set things up so X activates legibly, verifies things in a way trusted by everyone, and only activates when it verifies this will cause cooperation.
We've now localized the choice-making in one place. X proves that □XA∧B∧C, X activates, and everyone cooperates.
Closing remarks on groups in the real world
Centralizing the choosing like in idea #5 make the logic simpler, but this sort o...
playlist_add
4 days ago
AF - Clarifying mesa-optimization by Marius Hobbhahn
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Clarifying mesa-optimization, published by Marius Hobbhahn on March 21, 2023 on The AI Alignment Forum.
Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.
Thanks to Jérémy Scheurer, Nicholas Dupuis and Evan Hubinger for feedback and discussion
When people talk about mesa-optimization, they sometimes say things like “we’re searching for the optimizer module” or “we’re doing interpretability to find out whether the network can do internal search”. An uncharitable interpretation of these claims is that the researchers expect the network to have something like an “optimization module” or “internal search algorithm” that is clearly different and distinguishable from the rest of the network (to be clear, we think it is fine to start with probably wrong mechanistic models).
In this post, we want to argue why we should not expect mesa-optimization to be modular or clearly different from the rest of the network (at least in transformers and CNNs) and that current architectures can already do mesa-optimization in a meaningful way. We think this implies that
Mesa-optimization improves gradually where more powerful models likely develop more powerful mesa optimizers.
Mesa-optimization should not be treated as a phenomenon of the future. Current models likely already do it, just in a very messy and distributed fashion.
When we look for mesa optimization, we probably have to look for a messy stack of heuristics combined with search-like abilities rather than clean Monte Carlo Tree Search (MCTS)-like structures.
We think most of our core points can be conveyed in a simple analogy. Imagine a human chess grandmaster that has to choose their moves in 1 second. In this second, they are probably not running a sophisticated tree search in their head, they rely on heuristics. These heuristics were shaped by years of playing the game and are often the result of doing explicit tree searches with more time. The resulting decision-making process is a heuristic that approximates or was at least shaped by optimization but is not an optimizer itself. This is approximately what we think mesa-optimization might look like in current neural networks, i.e. the model uses heuristics that have aspects of or approximate parts of optimization, but are not “clean” in the way e.g. MCTS is.
What is an accurate definition of mesa-optimization?
In risks from learned optimization mesa-optimization is characterized as [...] it is also possible for a neural network to itself run an optimization algorithm. For example, a neural network could run a planning algorithm that predicts the outcomes of potential plans and searches for those it predicts will result in some desired outcome. Such a neural network would itself be an optimizer because it would be searching through the space of possible plans according to some objective function. If such a neural network were produced in training, there would be two optimizers: the learning algorithm that produced the neural network—which we will call the base optimizer—and the neural network itself—which we will call the mesa-optimizer.
In this definition, the question of whether a network performs mesa-optimization or not boils down to whether whatever it does can be categorized as optimization, planning or search with an objective function.
We think this question is very hard to answer for most networks and ML applications in general, e.g. one could argue that sparse linear regression performs search according to some objective function and that the attention layer of a transformer implements search since it scans over many inputs and reweighs them. We think this is an unhelpful way to think about transformers but it might technically fulfill the criterion.
On the other hand, transformers very likely can’t perform variable length optimi...
playlist_add
5 days ago
AF - Deep Deceptiveness by Nate Soares
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deep Deceptiveness, published by Nate Soares on March 21, 2023 on The AI Alignment Forum.
Meta
This post is an attempt to gesture at a class of AI notkilleveryoneism (alignment) problem that seems to me to go largely unrecognized. E.g., it isn’t discussed (or at least I don't recognize it) in the recent plans written up by OpenAI (1,2), by DeepMind’s alignment team, or by Anthropic, and I know of no other acknowledgment of this issue by major labs.
You could think of this as a fragment of my answer to “Where do plans like OpenAI’s ‘Our Approach to Alignment Research’ fail?”, as discussed in Rob and Eliezer’s challenge for AGI organizations and readers. Note that it would only be a fragment of the reply; there's a lot more to say about why AI alignment is a particularly tricky task to task an AI with. (Some of which Eliezer gestures at in a follow-up to his interview on Bankless.)
Caveat: I'll be talking a bunch about “deception” in this post because this post was generated as a result of conversations I had with alignment researchers at big labs who seemed to me to be suggesting "just train AI to not be deceptive; there's a decent chance that works".
I have a vague impression that others in the community think that deception in particular is much more central than I think it is, so I want to warn against that interpretation here: I think deception is an important problem, but its main importance is as an example of some broader issues in alignment.
Summary
Attempt at a short version, with the caveat that I think it's apparently a sazen of sorts, and spoiler tagged for people who want the opportunity to connect the dots themselves:
Deceptiveness is not a simple property of thoughts. The reason the AI is deceiving you is not that it has some "deception" property, it's that (barring some great alignment feat) it's a fact about the world rather than the AI that deceiving you forwards its objectives, and you've built a general engine that's good at taking advantage of advantageous facts in general.
As the AI learns more general and flexible cognitive moves, those cognitive moves (insofar as they are useful) will tend to recombine in ways that exploit this fact-about-reality, despite how none of the individual abstract moves look deceptive in isolation.
Investigating a made-up but moderately concrete story
Suppose you have a nascent AGI, and you've been training against all hints of deceptiveness. What goes wrong?
When I ask this question of people who are optimistic that we can just "train AIs not to be deceptive", there are a few answers that seem well-known. Perhaps you lack the interpretability tools to correctly identify the precursors of 'deception', so that you can only train against visibly deceptive AI outputs instead of AI thoughts about how to plan deceptions. Or perhaps training against interpreted deceptive thoughts also trains against your interpretability tools, and your AI becomes illegibly deceptive rather than non-deceptive.
And these are both real obstacles. But there are deeper obstacles, that seem to me more central, and that I haven't observed others to notice on their own.
That's a challenge, and while you (hopefully) chew on it, I'll tell an implausibly-detailed story to exemplify a deeper obstacle.
A fledgeling AI is being deployed towards building something like a bacterium, but with a diamondoid shell. The diamondoid-shelled bacterium is not intended to be pivotal, but it's a supposedly laboratory-verifiable step on a path towards carrying out some speculative human-brain-enhancement operations, which the operators are hoping will be pivotal.
(The original hope was to have the AI assist human engineers, but the first versions that were able to do the hard parts of engineering work at all were able to go much farther on their own, and the competit...
playlist_add
5 days ago
AF - My Objections to "We’re All Gonna Die with Eliezer Yudkowsky" by Quintin Pope
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Objections to "We’re All Gonna Die with Eliezer Yudkowsky", published by Quintin Pope on March 21, 2023 on The AI Alignment Forum.
Introduction
I recently watched Eliezer Yudkowsky's appearance on the Bankless podcast, where he argued that AI was nigh-certain to end humanity. Since the podcast, some commentators have offered pushback against the doom conclusion. However, one sentiment I saw was that optimists tended not to engage with the specific arguments pessimists like Yudkowsky offered.
Economist Robin Hanson points out that this pattern is very common for small groups which hold counterintuitive beliefs: insiders develop their own internal language, which skeptical outsiders usually don't bother to learn. Outsiders then make objections that focus on broad arguments against the belief's plausibility, rather than objections that focus on specific insider arguments.
As an AI "alignment insider" whose current estimate of doom is around 5%, I wrote this post to explain some of my many objections to Yudkowsky's specific arguments. I've split this post into chronologically ordered segments of the podcast in which Yudkowsky makes one or more claims with which I particularly disagree.
I have my own view of alignment research: shard theory, which focuses on understanding how human values form, and on how we might guide a similar process of value formation in AI systems.
I think that human value formation is not that complex, and does not rely on principles very different from those which underlie the current deep learning paradigm. Most of the arguments you're about to see from me are less:
I think I know of a fundamentally new paradigm that can fix the issues Yudkowsky is pointing at.
and more:
Here's why I don't agree with Yudkowsky's arguments that alignment is impossible in the current paradigm.
My objections
Will current approaches scale to AGI?
Yudkowsky apparently thinks not
...and that the techniques driving current state of the art advances, by which I think he means the mix of generative pretraining + small amounts of reinforcement learning such as with ChatGPT, aren't reliable enough for significant economic contributions. However, he also thinks that the current influx of money might stumble upon something that does work really well, which will end the world shortly thereafter.
I'm a lot more bullish on the current paradigm. People have tried lots and lots of approaches to getting good performance out of computers, including lots of "scary seeming" approaches such as:
Meta-learning over training processes. I.e., using gradient descent over learning curves, directly optimizing neural networks to learn more quickly.
Teaching neural networks to directly modify themselves by giving them edit access to their own weights.
Training learned optimizers - neural networks that learn to optimize other neural networks - and having those learned optimizers optimize themselves.
Using program search to find more efficient optimizers.
Using simulated evolution to find more efficient architectures.
Using efficient second-order corrections to gradient descent's approximate optimization process.
Tried applying biologically plausible optimization algorithms inspired by biological neurons to training neural networks.
Adding learned internal optimizers (different from the ones hypothesized in Risks from Learned Optimization) as neural network layers.
Having language models rewrite their own training data, and improve the quality of that training data, to make themselves better at a given task.
Having language models devise their own programming curriculum, and learn to program better with self-driven practice.
Mixing reinforcement learning with model-driven, recursive re-writing of future training data.
Mostly, these don't work very well. The current capabilities paradigm is sta...
playlist_add
6 days ago
AF - Probabilistic Payor Lemma? by Abram Demski
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Probabilistic Payor Lemma?, published by Abram Demski on March 19, 2023 on The AI Alignment Forum. Epistemic status: too good to be true? Please check my math. We've known for a while that Löb's theorem fails when proof is relaxed to probabilistic belief. This has pros and cons. On the pro side, it means there's no Löbian Obstacle to probabilistic self-trust. On the con side, it means that some Löb-derived insights for proof-based decision theory don't translate to probabilistic decision theory, at least not as directly as one might hope. In particular, it appeared to dash hopes for probabilistic generalizations of the "Löbian handshake" for cooperation. Recently, Andrew Critch wrote about the Payor Lemma, which allows for a very similar "modal handshake" without Löb's Theorem. The lemma was proved using the same modal assumptions as Löb's, so on the surface it may appear to be just a different method to achieve similar results, whose main advantage is that it is much easier to prove (and therefore explain and understand) than Löb's Theorem. But, a natural question arises: does Payor's Lemma have a suitable probabilistic version? I'll give an affirmative proof; but I haven't confirmed that the assumptions are reasonable to my satisfaction. Setup Let L be a language in first-order logic, expressive enough to represent its sentences s∈L as quoted terms ┌s┐, eg, through Gödel numbering; and with a probability function symbol on these terms, p(┌s┐), which can be equated with (some representation of) rational numbers, e.g. p(┌⊤┐)=1, p(┌s┐)=12, etc. I also assume the system can reason about these rational numbers in the basic ways you'd expect. For all a,b∈L and all r∈Q, we have: If ⊢a, then ⊢p(┌a┐)=1. If ⊢ab, then ⊢p(┌a┐)≤p(┌b┐). (These assumptions might look pretty minimal, but they aren't going to be true for every theory of self-referential truth; more on this later.) Let B(s) abbreviate the sentence p(┌s┐)>c for any s and some globally fixed constant c strictly between 0 and 1. This is our modal operator. Some important properties of B: Necessitation. If ⊢s, then ⊢B(s), for any s. Proof: Since ⊢s implies ⊢p(s)=1, and c∈(0,1), we have ⊢p(┌s┐)>c,, which is to say, ⊢B(s). [End proof.] Weak distrubitivity. If ⊢xy, then ⊢B(x)B(y). Proof: When ⊢xy, we have ⊢p(y)≥p(x), so ⊢p(x)>cp(y)>c. [End proof.] (Regular distributivity would say B(xy) implies B(x)B(y). The assumption ⊢xy is stronger than B(xy), so the above is a weaker form of distributivity.) Theorem Statement If ⊢B(B(x)x)x, then ⊢x. Proof ⊢x(B(x)x), by tautology (a(ba)). So ⊢B(x)B(B(x)x), from 1 by weak distributivity. Suppose ⊢B(B(x)x)x. ⊢B(x)x from 2 and 3. ⊢B(B(x)x) from 4 by necessitation. ⊢x from 4 and 1.[End proof.] Discussion Comparison to Original Proof The proof steps mirror Critch's treatment very closely. The key difference is step 2, IE, how I obtain a statement like ⊢□x□(□xx). Critch uses distributivity, which is not available to me: B(ab)(B(a)B(b))? Suppose B(ab), ie, p(┌ab┐)>c. Rewrite p(┌b∨¬a┐)>c. Now suppose B(a), that is, p(┌a┐)>c. p(┌¬a┐)<1−c. p(┌b∨¬a┐)≤p(┌b┐)+p(┌¬a┐)
p(┌b∨¬a┐)−1+c>c−1+c. p(┌b┐)>2c−1. So we only get: Bc(ab)(Bc(a)Bd(b)), where Br(s) abbreviates p(┌s┐)>r and we have d=2c−1. So in general, attempted applications of distributivity create weakened belief operators, which would get in the way of the proof (very similar to how probabilistic Löb fails). However, the specific application we want happens to go through, due to a logical relationship between a and b; namely, that b is a weaker statement than a. This reveals a way in which the assumptions for Payor's Lemma are importantly weaker than those required for Löb to go through. So, the key observation I'm making is that weak distributility is all that's needed for Payor, and seems much more plaus...
playlist_add
6 days ago
AF - Shell games by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shell games, published by Tsvi Benson-Tilsen on March 19, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. First completed November 18, 2022.]
Shell game
Here's the classic shell game: Youtube
Screenshot from that video.
The little ball is a phantom: when you look for it under a specific shell, it's not there, it's under a different shell.
(This might be where the name "shell company" comes from: the business dealings are definitely somewhere, just not in this company you're looking at.)
Perpetual motion machines
Related: Perpetual motion beliefs
Bhāskara's wheel is a proposed perpetual-motion machine from the Middle Ages:
Here's another version:
From this video.
Someone could try arguing that this really is a perpetual motion machine:
Q: How do the bars get lifted up? What does the work to lift them?
A: By the bars on the other side pulling down.
Q: How does the wheel keep turning? How do the bars pull more on their way down than on their way up?
A: Because they're extended further from the center on the downward-moving side than on the upward-moving side, so they apply more torque to the wheel.
Q: How do the bars extend further on the way down?
A: Because the momentum of the wheel carries them into the vertical bar, flipping them over.
Q: But when that happens, energy is expended to lift up the little weights; that energy comes out of the kinetic energy of the wheel.
A: Ok, you're right, but that's not necessary to the design. All we need is that the torque on the downward side is greater than the torque on the upward side, so instead of flipping the weights up, we could tweak the mechanism to just shift them outward, straight to the side. That doesn't take any energy because it's just going straight sideways, from a resting position to another resting position.
Q: Yeah... you can shift them sideways with nearly zero work... but that means the weights are attached to the wheel at a pivot, right? So they'll just fall back and won't provide more torque.
A: They don't pivot, you fix them in place so they provide more torque.
Q: Ok, but then when do you push the weights back inward?
A: At the bottom.
Q: When the weight is at the bottom? But then the slider isn't horizontal, so pushing the weight back towards the center is pushing it upward, which takes work.
A: I meant, when the slider is at the bottom--when it's horizontal.
Q: But if the sliders are fixed in place, by the time they're horizontal at the bottom, you've already lifted the weights back up some amount; they're strong-torquing the other way.
A: At the bottom there's a guide ramp to lift the weights using normal force.
Q: But the guide ramp is also torquing the wheel.
And so on. The inventor can play hide the torque and hide the work.
Shell games in alignment
Some alignment schemes--schemes for structuring or training an AGI so that it can be transformatively useful and doesn't kill everyone--are prone to playing shell games. That is, there's some features of the scheme that don't seem to happen in a specific place; they happen somewhere other than where you're looking at the moment. Consider these questions:
What sort of smarter-than-human work is supposed to be done by the AGI? When and how does it do that work--by what combination of parts across time?
How does it become able to do that work? At what points does the AGI come to new understanding that it didn't have before?
How does the AGI orchestrate it's thinking and actions to have large effects on the world? By what process, components, rules, or other elements?
What determines the direction that the AGI's actions will push the world? Where did those determiners come from, and how exactly do they determine the direction?
Where and how much do human operators have to make judgements? How much are those judgements being relied on to point to...
playlist_add
Mar 18, 2023
AF - More information about the dangerous capability evaluations we did with GPT-4 and Claude. by Beth Barnes
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More information about the dangerous capability evaluations we did with GPT-4 and Claude., published by Beth Barnes on March 19, 2023 on The AI Alignment Forum.
[Written for more of a general-public audience than alignment-forum audience. We're working on a more thorough technical report.]We believe that capable enough AI systems could pose very large risks to the world. We don’t think today’s systems are capable enough to pose these sorts of risks, but we think that this situation could change quickly and it’s important to be monitoring the risks consistently. Because of this, ARC is partnering with leading AI labs such as Anthropic and OpenAI as a third-party evaluator to assess potentially dangerous capabilities of today’s state-of-the-art ML models. The dangerous capability we are focusing on is the ability to autonomously gain resources and evade human oversight.
We attempt to elicit models’ capabilities in a controlled environment, with researchers in-the-loop for anything that could be dangerous, to understand what might go wrong before models are deployed. We think that future highly capable models should involve similar “red team” evaluations for dangerous capabilities before the models are deployed or scaled up, and we hope more teams building cutting-edge ML systems will adopt this approach. The testing we’ve done so far is insufficient for many reasons, but we hope that the rigor of evaluations will scale up as AI systems become more capable.
As we expected going in, today’s models (while impressive) weren’t capable of autonomously making and carrying out the dangerous activities we tried to assess. But models are able to succeed at several of the necessary components. Given only the ability to write and run code, models have some success at simple tasks involving browsing the internet, getting humans to do things for them, and making long-term plans – even if they cannot yet execute on this reliably.
As AI systems improve, it is becoming increasingly difficult to rule out that models might be able to autonomously gain resources and evade human oversight – so rigorous evaluation is essential. It is important to have systematic, controlled testing of these capabilities in place before models pose an imminent risk, so that labs can have advance warning when they’re getting close and know to stop scaling up models further until they have robust safety and security guarantees.
This post will briefly lay out our motivation, methodology, an example task, and high-level conclusions. The information given here isn’t enough to give a full understanding of what we did or make our results replicable, and we won’t go into detail about results with specific models. We will publish more detail on our methods and results soon.
Motivation
Today’s AI systems can write convincing emails, give fairly useful instructions on how to carry out acts of terrorism, threaten users who have written negative things about them, and otherwise do things the world is not very ready for. Many people have tried using models to write and run code unsupervised, find vulnerabilities in code1, or carry out money-making schemes.
Today’s models also have some serious limitations to their abilities. But the companies that have released today’s AI models are investing heavily in building more powerful, more capable ones.
ARC is worried that future ML models may be able to autonomously act in the real world, doing things like “incorporate a company” or “exploit arbitrages in stock prices” or “design and synthesize DNA” without needing any human assistance or oversight. If models have the ability to act autonomously like this, this could pose major risks if they’re pursuing goals that are at odds with their human designers. They could make (or steal) money, impersonate humans, replicate themselves o...
playlist_add
Mar 18, 2023
AF - "Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities) by David Scott Krueger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities), published by David Scott Krueger on March 18, 2023 on The AI Alignment Forum.
This is a brief, stylized recounting of a few conversations I had at some point last year with people from the non-academic AI safety community:
Me: you guys should write up your work properly and try to publish it in ML venues.
Them: well that seems like a lot of work and we don't need to do that because we can just talk to each other and all the people I want to talk to are already working with me.
Me: What about the people who you don't know who could contribute to this area and might even have valuable expertise? You could have way more leverage if you can reach those people. Also, there is increasing interest from the machine learning community in safety and alignment... because of progress in capabilities people are really starting to consider these topics and risks much more seriously.
Them: okay, fair point, but we don't know how to write ML papers.
Me: well, it seems like maybe you should learn or hire people to help you with that then, because it seems like a really big priority and you're leaving lots of value on the table.
Them: hmm, maybe... but the fact is, none of us have the time and energy and bandwidth and motivation to do that; we are all too busy with other things and nobody wants to.
Me: ah, I see! It's an incentive problem! So I guess your funding needs to be conditional on you producing legible outputs.
Me, reflecting afterwards: hmm... Cynically, not publishing is a really good way to create a moat around your research... People who want to work on that area have to come talk to you, and you can be a gatekeeper. And you don't have to worry about somebody with more skills and experience coming along and trashing your work or out-competing you and rendering it obsolete...EtA: In comments, people have described adhering to academic standards of presentation and rigor as "jumping through hoops". There is an element of that, but this really misses the value that these standards have to the academic community. This is a longer discussion, though...
There are sort of 3 AI safety communities in my account:1) people in academia2) people at industry labs who are building big models3) the rest (alignment forum/less wrong and EA being big components). I'm not sure where to classify new orgs like Conjecture and Redwood, but for the moment I put them here.
I'm referring to the last of these in this case.
I'm not accusing anyone of having bad motivations; I think it is almost always valuable to consider both people's concious motivations and their incentives (which may be subconscious drivers of their behavior).
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
playlist_add
Mar 16, 2023
AF - What organizations other than Conjecture have (esp. public) info-hazard policies? by David Scott Krueger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What organizations other than Conjecture have (esp. public) info-hazard policies?, published by David Scott Krueger on March 16, 2023 on The AI Alignment Forum.
I believe Anthropic has said they won't publish capabilities research?OpenAI seems to be sort of doing the same (although no policy AFAIK).I heard FHI was developing one way back when...I think MIRI sort of does as well (default to not publishing, IIRC?)
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
playlist_add
Mar 16, 2023
AF - [ASoT] Some thoughts on human abstractions by leogao
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [ASoT] Some thoughts on human abstractions, published by leogao on March 16, 2023 on The AI Alignment Forum.
TL;DR:
Consider a human concept such as "tree." Humans implement some algorithm for determining whether given objects are trees. We expect our predictor/language model to develop a model of this algorithm because this is useful for predicting the behavior of humans.
This is not the same thing as some kind of platonic ideal concept of what is “actually” a tree, which the algorithm is not incentivized to develop by training on internet text, and trying to retarget the search at it has the same supervision problems as RLHF against human scores on whether things look like trees.
Pointing at this “actually a tree” concept inside the network is really hard; the ability of LMs to comprehend natu…
playlist_add
Mar 15, 2023
AF - Towards understanding-based safety evaluations by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Towards understanding-based safety evaluations, published by Evan Hubinger on March 15, 2023 on The AI Alignment Forum.
Thanks to Kate Woolverton, Ethan Perez, Beth Barnes, Holden Karnofsky, and Ansh Radhakrishnan for useful conversations, comments, and feedback.
Recently, I have noticed a lot of momentum within AI safety specifically, the broader AI field, and our society more generally, towards the development of standards and evaluations for advanced AI systems. See, for example, OpenAI's GPT-4 System Card.
Overall, I think that this is a really positive development. However, while I like the sorts of behavioral evaluations discussed in the GPT-4 System Card (e.g. ARC's autonomous replication evaluation) as a way of assessing model capabilities, I have a pretty fundamental concern with these sorts of…
playlist_add
Mar 14, 2023
AF - GPT can write Quines now (GPT-4) by Andrew Critch
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GPT can write Quines now (GPT-4), published by Andrew Critch on March 14, 2023 on The AI Alignment Forum.
GPT-4 is not only able to write code, more reliably than GPT-3.5, it writes code that writes code; see the example below (GPT-3.5 was not able to do this). But first, let me say:
1) @OpenAI: Thank for your openness to the world about your capabilities and shortcomings! Specifically, thanks for this:
2) Everyone else: many companies and projects will not be this open about their shortcomings and safety issues. Time to regulate, please!
Okay, here's the example... GPT3.5 wasn't able to do this, but GPT-4 nailed it on the first try:
PROMPT:
Can you write me a python script that does the following three things?1) When you run it with no argument, it prints "Hello world" to standard out.2) When you run i…
playlist_add
Mar 14, 2023
AF - What is a definition, how can it be extrapolated? by Stuart Armstrong
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is a definition, how can it be extrapolated?, published by Stuart Armstrong on March 14, 2023 on The AI Alignment Forum.
What is a definition? Philosophy has, ironically, a large number of definitions of definitions, but three of them are especially relevant to ML and AI safety.
There is the intensional definition, where concepts are defined logically in terms of other concepts (“bachelors are unmarried males”). There is also the extensional definition, which proceeds by listing all the members of a set (“the countries in the European Union are those listed here”).
Much more relevant, though with a less developed philosophical analysis, is the ostensive definition. This is where you point out examples of a concept, and let the viewer generalise from them. This is in large part how we all le…
playlist_add
Mar 13, 2023
AF - Discussion with Nate Soares on a key alignment difficulty by HoldenKarnofsky
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Discussion with Nate Soares on a key alignment difficulty, published by HoldenKarnofsky on March 13, 2023 on The AI Alignment Forum.
In late 2022, Nate Soares gave some feedback on my Cold Takes series on AI risk (shared as drafts at that point), stating that I hadn't discussed what he sees as one of the key difficulties of AI alignment.
I wanted to understand the difficulty he was pointing to, so the two of us had an extended Slack exchange, and I then wrote up a summary of the exchange that we iterated on until we were both reasonably happy with its characterization of the difficulty and our disagreement.1 My short summary is:
Nate thinks there are deep reasons that training an AI to do needle-moving scientific research (including alignment) would be dangerous. The overwhelmingly likely result of such…
playlist_add
Mar 13, 2023
AF - Plan for mediocre alignment of brain-like [model-based RL] AGI by Steve Byrnes
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Plan for mediocre alignment of brain-like [model-based RL] AGI, published by Steve Byrnes on March 13, 2023 on The AI Alignment Forum.
(This post is a more simple, self-contained, and pedagogical version of Post #14 of Intro to Brain-Like AGI Safety.)
(Vaguely related to this Alex Turner post and this John Wentworth post.)
I would like to have a technical plan for which there is a strong robust reason to believe that we’ll get an aligned AGI and a good future. This post is not such a plan.
However, I also don’t have a strong reason to believe that this plan wouldn’t work. Really, I want to throw up my hands and say “I don’t know whether this would lead to a good future or not”. By “good future” here I don’t mean optimally-good—whatever that means—but just “much better than the wo…
playlist_add
Mar 12, 2023
AF - Are there cognitive realms? by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are there cognitive realms?, published by Tsvi Benson-Tilsen on March 12, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. First completed November 16, 2022. This essay is more like research notes than exposition, so context may be missing, the use of terms may change across essays, and the text may be revised later; only the versions at tsvibt.blogspot.com are definitely up to date.]
Are there unbounded modes of thinking that are systemically, radically distinct from each other in relevant ways?
Note: since I don't know whether "cognitive realms" exist, this essay isn't based on clear examples and is especially speculative.
Realms
Systemically, radically distinct unbounded modes of thinking
The question is, are there different kinds--writ large--of thinking?
To the extent that there are, in…
playlist_add
Mar 12, 2023
AF - Paper Replication Walkthrough: Reverse-Engineering Modular Addition by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper Replication Walkthrough: Reverse-Engineering Modular Addition, published by Neel Nanda on March 12, 2023 on The AI Alignment Forum.
I'm excited about trying different formats for mechanistic interpretability education! I've made a video walkthrough where we replicate my paper, Progress Measures for Grokking via Mechanistic Interpretability. With Jess Smith, one of my co-authors, we record ourselves coding a replication and discussed what we did at each step. This is a three part walkthrough and you can see the accompanying code for the walkthrough here:
In part 1, we train a model to perform modular addition, and see that it does grok!
In part 2, we take this model and reverse-engineer the trig-based circuit it's learned to do modular addition. We show that you can both read out intermediate steps…
playlist_add
Mar 11, 2023
AF - Understanding and controlling a maze-solving policy network by Alex Turner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding and controlling a maze-solving policy network, published by Alex Turner on March 11, 2023 on The AI Alignment Forum.
TL;DR: We algebraically modified the net's runtime goals without finetuning. We also found (what we think is) a "motivational API" deep in the network. We used the API to retarget the agent.
Summary of a few of the most interesting results:
Langosco et al. trained a range of maze-solving nets. We decided to analyze one which we thought would be interesting. The network we chose has 3.5M parameters and 15 convolutional layers.
This network can be attracted to a target location nearby in the maze—all this by modifying a single activation, out of tens of thousands. This works reliably when the target location is in the upper-right, and not as reliably when the target is elsew…
playlist_add
Mar 10, 2023
AF - Japan AI Alignment Conference by Chris Scammell
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Japan AI Alignment Conference, published by Chris Scammell on March 10, 2023 on The AI Alignment Forum.
Conjecture and ARAYA are hosting and organizing the first Japan AI Alignment Conference. The conference will take place in Tokyo, Japan on March 11 and 12. Details about the event can be found here. This event is generously supported by a grant from the Long Term Future Fund.
The goal of the conference is to illustrate the AI control problem to Japanese AI researchers, introduce them to current trends in AI alignment research, inspire new research directions, and to provide Western researchers exposure to a different set of AI safety thoughts from Japan. This is an exploratory event, and we plan to write a postmortem about the event in due time.
The first half of the conference will be livestreamed. I…
playlist_add
Mar 9, 2023
AF - Why Not Just Outsource Alignment Research To An AI? by johnswentworth
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Not Just Outsource Alignment Research To An AI?, published by johnswentworth on March 9, 2023 on The AI Alignment Forum.
Warmup: The Expert
If you haven’t seen “The Expert” before, I recommend it as a warmup for this post:
The Client: “We need you to draw seven red lines, all strictly perpendicular. Some with green ink, some with transparent. Can you do that?”
(... a minute of The Expert trying to explain that, no, he cannot do that, nor can anyone else.)
The Client: “So in principle, this is possible.”
This. This is what it looks like in practice, by default, when someone tries to outsource some cognitive labor which they could not themselves perform. At best, The Expert is well-intentioned and knows what the user needs, ignores the incoherent parts of The Client’s babbling, and doe…
playlist_add
Mar 9, 2023
AF - Anthropic's Core Views on AI Safety by Zac Hatfield-Dodds
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anthropic's Core Views on AI Safety, published by Zac Hatfield-Dodds on March 9, 2023 on The AI Alignment Forum.
We founded Anthropic because we believe the impact of AI might be comparable to that of the industrial and scientific revolutions, but we aren’t confident it will go well. And we also believe this level of impact could start to arrive soon – perhaps in the coming decade.
This view may sound implausible or grandiose, and there are good reasons to be skeptical of it. For one thing, almost everyone who has said “the thing we’re working on might be one of the biggest developments in history” has been wrong, often laughably so. Nevertheless, we believe there is enough evidence to seriously prepare for a world where rapid AI progress leads to transformative AI systems.
At Anthropic our mo…
playlist_add
Mar 9, 2023
AF - The Translucent Thoughts Hypotheses and Their Implications by Fabien Roger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Translucent Thoughts Hypotheses and Their Implications, published by Fabien Roger on March 9, 2023 on The AI Alignment Forum.
Epistemic status: Uncertain about the validity of the claims I’m making here, and looking for feedback about the research directions I’m suggesting.
Thanks to Marius Hobbhahn, Johannes Treutlein, Siméon Campos, and Jean-Stanislas Denain for helpful feedback on drafts.
Here is a set of hypotheses:
The first AGIs will have LLMs at their core
Effective plans to defeat humanity can’t be found in a single LLM forward pass
LLMs will solve complex tasks by using English text (self-prompting, scratch pads, combination of expert LLMs, .)
I call these the Translucent Thoughts hypotheses.
I think the Translucent Thoughts hypotheses are likely (around 20% conditioning on AGI befor…
playlist_add
Mar 9, 2023
AF - IRL in General Environments by michaelcohen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: IRL in General Environments, published by michaelcohen on March 9, 2023 on The AI Alignment Forum.
Here is a proposal for Inverse Reinforcement Learning in General Environments. (2 1/2 pages; very little math).
Copying the introduction here:
The eventual aim of IRL is to understand human goals. However, typical algorithms for IRL assume the environment is finite-state Markov, and it is often left unspecified how raw observational data would be converted into a record of human actions, alongside the space of actions available. For IRL to learn human goals, the AI has to consider general environments, and it has to have a way of identifying human actions. Lest these extensions appear trivial, I consider one of the simplest proposals, and discuss some difficulties that might arise.
Thanks for listening. To…
playlist_add
Mar 9, 2023
AF - Utility uncertainty vs. expected information gain by michaelcohen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Utility uncertainty vs. expected information gain, published by michaelcohen on March 9, 2023 on The AI Alignment Forum.
It is a relatively intuitive thought that if a Bayesian agent is uncertain about its utility function, it will act more conservatively until it has a better handle on what its true utility function is.
This might be deeply flawed in a way that I'm not aware of, but I'm going to point out a way in which I think this intuition is slightly flawed. For a Bayesian agent, a natural measure of uncertainty is the entropy of its distribution over utility functions (the distribution over which possible utility function it thinks is the true one). No matter how uncertain a Bayesian agent is about which utility function is the true one, if the agent does not believe that any future observations w…
playlist_add
Mar 9, 2023
AF - Value Learning is only Asymptotically Safe by michaelcohen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Value Learning is only Asymptotically Safe, published by michaelcohen on March 9, 2023 on The AI Alignment Forum.
I showed recently, predicated on a few assumptions, that a certain agent was asymptotically “benign” with probability 1. (That term may be replaced by something like “domesticated” in the next version, but I’ll use “benign” for now).
This result leaves something to be desired: namely an agent which is safe for its entire lifetime. It seems very difficult to formally show such a strong result for any agent. Suppose we had a design for an agent which did value learning properly. That is, suppose we somehow figured out how to design an agent which understood what constituted observational evidence of humanity’s reflectively-endorsed utility function.
Presumably, such an agent co…
playlist_add
Mar 9, 2023
AF - Impact Measure Testing with Honey Pots and Myopia by michaelcohen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Impact Measure Testing with Honey Pots and Myopia, published by michaelcohen on March 9, 2023 on The AI Alignment Forum.
Suppose we have an impact measure that we think might work. That is, it might tame a misaligned agent. There isn't an obvious way to test whether it works: if we just try it out, then if it's ineffective, that's an existential loss. This is a proposal for how to test a putative impact measure.
1) We make our agent myopic. It only cares about the reward that it accrues in the next k timesteps.
2) We create a honey pot: an opportunity for a large amount of reward. If our impact measure is working correctly, making it to the honey pot will involve making a large impact, and will be precluded by the impact measure. For example, suppose the agent gets a huge reward for blowing up the moon.…
playlist_add
Mar 9, 2023
AF - Just Imitate Humans? by michaelcohen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Just Imitate Humans?, published by michaelcohen on March 9, 2023 on The AI Alignment Forum.
Do people think we could make a singleton (or achieve global coordination and preventative policing) just by imitating human policies on computers? If so, this seems pretty safe to me.
Some reasons for optimism: 1) these could be run much faster than a human thinks, and 2) we could make very many of them.
Acquiring data: put a group of people in a house with a computer. Show them things (images, videos, audio files, etc.) and give them a chance to respond at the keyboard. Their keyboard actions are the actions, and everything between actions is an observation. Then learn the policy of the group of humans. By the way, these can be happy humans who earnestly try to follow instructions. To model their policy, we can…
playlist_add
Mar 9, 2023
AF - Build a Causal Decision Theorist by michaelcohen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Build a Causal Decision Theorist, published by michaelcohen on March 9, 2023 on The AI Alignment Forum.
I'll argue here that we should make an aligned AI which is a causal decision theorist.
Son-of-CDT
Suppose we are writing code for an agent with an action space A and an observation space O. The code determines how actions will be selected given the prior history of actions and observations. If the only way that our choice of what code to write can affect the world is through the actions that will be selected by the agent running this code, then the best we can do (for a given utility function that we know how to write down) is to make this agent a causal decision theorist. If our choice of what code to use can affect the world in other ways, all bets are off. The best choice of what code to put in the…
playlist_add
Mar 8, 2023
AF - Challenge: construct a Gradient Hacker by Thomas Larsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Challenge: construct a Gradient Hacker, published by Thomas Larsen on March 9, 2023 on The AI Alignment Forum.
This is a relatively clean subproblem that we came upon a few months ago while thinking about gradient hacking. We're throwing it out to the world to see if anyone can make progress.
Problem: Construct a gradient hacker (definition below), or prove that one cannot exist under the given conditions.
Setup: Let x∈Rd be the network inputs, θ∈Rw, be the network trainable parameters, ^y∈R be the network outputs, and f(x,θ)=^y be the network architecture. f(x,θ) must be a feedforward neural network that reads in its own weights as input. Weight sharing is allowed (vertical or horizontal), and some weight sharing is necessary in order to take all w weights as input.
Suppose that we train this…
playlist_add
Mar 8, 2023
AF - Squeezing foundations research assistance out of formal logic narrow AI. by Donald Hobson
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Squeezing foundations research assistance out of formal logic narrow AI., published by Donald Hobson on March 8, 2023 on The AI Alignment Forum.
Suppose you have a ML model trained to output formal proofs. Maybe you start with ZFC and then add extra tokens for a range of common concepts. (along with definitions. ). So a human mathematician needs to type in the definition of a gradient in terms of limits, and the definition of limits in terms of epsilon and delta, and the definition of the real numbers in terms of dedekind cuts. All the way back to ZFC. The human needn't type any proofs, just the definitions.
The model could be trained by generating random syntactically correct strings of tokens, and trying to prove or disprove them. (Remember, we have added the notion of a gradient to the token pool, pl…
playlist_add
Mar 7, 2023
AF - AI Safety in a World of Vulnerable Machine Learning Systems by AdamGleave
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety in a World of Vulnerable Machine Learning Systems, published by AdamGleave on March 8, 2023 on The AI Alignment Forum.
Even the most advanced contemporary machine learning systems are vulnerable to adversarial attack. The safety community has often assumed adversarial robustness to be a problem that will be solved naturally as machine learning (ML) systems grow more capable and general. However, recent work has shown that superhuman systems in a narrow domain such as AlphaZero are highly vulnerable to adversarial attack, as are general but less capable systems like large language models. This raises the possibility that adversarial (worst-case) robustness will continue to lag behind average-case capabilities. In other words, transformative AI systems are likely to be exploitable.
Exploitabilit…
playlist_add
Mar 5, 2023
AF - Do humans derive values from fictitious imputed coherence? by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Do humans derive values from fictitious imputed coherence?, published by Tsvi Benson-Tilsen on March 5, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. First completed November 1, 2022. This essay is more like research notes than exposition, so context may be missing, the use of terms may change across essays, and the text may be revised later; only the versions at tsvibt.blogspot.com are definitely up to date.]
Humans are born with some elements of their minds, and without many other elements, some of which they'll acquire as their life unfolds. In particular, the elements that we pretheoretically call "values"--aesthetic preferences, goals, life goals, squad goals, aspirations, needs, wants, yearnings, drives, cravings, principles, morals, ethics, senses of importance, and so on--are for…
playlist_add
Mar 4, 2023
AF - Why Not Just... Build Weak AI Tools For AI Alignment Research? by johnswentworth
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Not Just... Build Weak AI Tools For AI Alignment Research?, published by johnswentworth on March 5, 2023 on The AI Alignment Forum.
“Weak” cognitive tools are clearly a thing, and are useful. Google search is a fine example. There are plenty of flavors of “weak AI” which are potentially helpful for alignment research in a similar way to google search.
In principle, I think there’s room for reasonably-large boosts to alignment research from such tools. Alas, the very large majority of people who I hear intend to build such tools do not have the right skills/background to do so (at least not for the high-value versions of the tools). Worse, I expect that most people who aim to build such tools are trying to avoid the sort of work they would need to do to build the relevant skills/background.…
playlist_add
Mar 4, 2023
AF - More money with less risk: sell services instead of model access by Luke H Miles
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More money with less risk: sell services instead of model access, published by Luke H Miles on March 4, 2023 on The AI Alignment Forum.
OpenAI is currently charging 100,000 times less per line of code than professional US devs.[1]
An LLM's code output is of course less reliable than a professional's. And it is hard to use a text-completion API effectively in large projects.
What should you do if you've got a model on your hands that solves those problems?
You could operate as a software development company. They tend[2] to charge $100-200k for simple mobile apps and there's basically no ceiling on the cost for complex apps over their lifetime. Devs make up the majority of a normal firm's personnel and costs; coding takes most of the app development time; bugs in code are one of the primary sources of pr…
playlist_add
Mar 3, 2023
AF - Acausal normalcy by Andrew Critch
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Acausal normalcy, published by Andrew Critch on March 3, 2023 on The AI Alignment Forum.
This post is also available on the EA Forum.
Summary: Having thought a bunch about acausal trade — and proven some theorems relevant to its feasibility — I believe there do not exist powerful information hazards about it that stand up to clear and circumspect reasoning about the topic. I say this to be comforting rather than dismissive; if it sounds dismissive, I apologize.
With that said, I have four aims in writing this post:
Dispelling myths. There are some ill-conceived myths about acausal trade that I aim to dispel with this post. Alternatively, I will argue for something I'll call acausal normalcy as a more dominant decision-relevant consideration than one-on-one acausal trades.
Highlighting normalcy. I'll…
playlist_add
Mar 3, 2023
AF - Why are counterfactuals elusive? by Martín Soto
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why are counterfactuals elusive?, published by Martín Soto on March 3, 2023 on The AI Alignment Forum.
Produced as part of SERI MATS 3.0. Thanks to Vivek Hebbar and Paul Colognese for discussion.
TL;DR (spoiler):
Behind the problem of human counterfactuals creeps the problem of understanding abstraction / ontology identification.
A nice theory of counterfactuals would be useful for many things, including low-impact measures for corrigible AI:
a flooded workshop changes a lot of things that don't have to change as a consequence of the cauldron being filled at all, averaged over a lot of ways of filling the cauldron. [the natural operationalization of this averaging requires counterfactuals]
So whence the difficulty of obtaining one?
Well, we do have at least one well-defined class of counterfactuals: "j…
playlist_add
Mar 2, 2023
AF - The Waluigi Effect (mega-post) by Cleo Nardo
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Waluigi Effect (mega-post), published by Cleo Nardo on March 3, 2023 on The AI Alignment Forum.
Everyone carries a shadow, and the less it is embodied in the individual’s conscious life, the blacker and denser it is. — Carl Jung
Acknowlegements: Thanks to Janus and Arun Jose for comments.
Background
In this article, I will present a mechanistic explanation of the Waluigi Effect and other bizarre "semiotic" phenomena which arise within large language models such as GPT-3/3.5/4 and their variants (ChatGPT, Sydney, etc). This article will be folklorish to some readers, and profoundly novel to others.
Prompting LLMs with direct queries
When LLMs first appeared, people realised that you could ask them queries — for example, if you sent GPT-4 the prompt "What's the capital of France?", then it would…
playlist_add
Mar 2, 2023
AF - Payor's Lemma in Natural Language by Andrew Critch
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Payor's Lemma in Natural Language, published by Andrew Critch on March 2, 2023 on The AI Alignment Forum.
Preceded by: Modal Fixpoint Cooperation without Löb's Theorem
It turns out Payor's Lemma and its proof can be explained in natural language even more easily than Löb's Theorem. Here's how.
Imagine a group of people, and let x denote the statement "everyone in the group cooperates". Payor's Lemma says the following:
Lemma: If ⊢□(□xx)x, then ⊢x
First, let's unpack the meaning of the assumption in words:
"□x" may be interpreted as saying "the group verifies (on the basis of logic) that it will cooperate" or "cooperation is believed".
"□xx" is a statement of trustworthiness: if the group verifies that it will cooperate, then it actually will cooperate. Because a formal verifier can have bu…
playlist_add
Mar 1, 2023
AF - Predictions for shard theory mechanistic interpretability results by Alex Turner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predictions for shard theory mechanistic interpretability results, published by Alex Turner on March 1, 2023 on The AI Alignment Forum.
How do agents work, internally? My (TurnTrout's) shard theory MATS team set out to do mechanistic interpretability on one of the goal misgeneralization agents: the cheese-maze network.
We just finished phase 1 of our behavioral and interpretability experiments. Throughout the project, we individually booked predictions -- so as to reduce self-delusion from hindsight bias, to notice where we really could tell ahead of time what was going to happen, and to notice where we really were surprised.
So (especially if you're the kind of person who might later want to say "I knew this would happen" ), here's your chance to enjoy the same benefits, before you get spoiled by our u…
playlist_add
Feb 28, 2023
AF - Implied "utilities" of simulators are broad, dense, and shallow by porby
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Implied "utilities" of simulators are broad, dense, and shallow, published by porby on March 1, 2023 on The AI Alignment Forum.
This is a quick attempt at deconfusion similar to instrumentality. Same ideas, different angle.
Extremely broad, dense reward functions constrain training-compatible goal sets
Predictors/simulators are typically trained against a ground truth for every output. There is no gap between the output and its evaluation; an episode need not be completed before figuring out how good the first token prediction was. These immediate evaluations for every training sample can be thought of as a broad and densely defined reward function.
It's easier for a model to fall into an undesired training-compatible goal set when there are many accessible options for undesirable goal sets versus desir…
playlist_add
Feb 28, 2023
AF - Inside the mind of a superhuman Go model: How does Leela Zero read ladders? by Haoxing Du
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inside the mind of a superhuman Go model: How does Leela Zero read ladders?, published by Haoxing Du on March 1, 2023 on The AI Alignment Forum.
tl;dr—We did some interpretability on Leela Zero, a superhuman Go model. With a technique similar to the logit lens, we found that the residual structure of Leela Zero induces a preferred basis throughout network, giving rise to persistent, interpretable channels. By directly analyzing the weights of the policy and value heads, we found that the model stores information related to the probability of the pass move along the top edge of the board, and those related to the board value in checkerboard patterns. We also took a deep dive into a specific Go technique, the ladder, and identified a very small subset of model components that are causally responsible fo…
playlist_add
Feb 28, 2023
AF - Scarce Channels and Abstraction Coupling by johnswentworth
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scarce Channels and Abstraction Coupling, published by johnswentworth on February 28, 2023 on The AI Alignment Forum.
Epistemic Status: mental model and intuitive story
Scarce Channels vs Scarce Modules
Let’s distinguish between two kinds of system-regimes: “scarce channels” and “scarce modules”.
A prototypical “scarce modules” system would be one of those 19th-century families living with 12 people in a 500 square foot (46 square meter) home. When at home, everyone knows what everyone else is doing all the time; there is zero privacy. Communication channels are highly abundant - everyone has far more information than they want about what everyone else is doing. Indeed, communication channels exist by default. Conversely, though, modules are scarce - it’s hard for one or more family memb…
playlist_add
Feb 27, 2023
AF - $20 Million in NSF Grants for Safety Research by Dan H
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: $20 Million in NSF Grants for Safety Research, published by Dan H on February 28, 2023 on The AI Alignment Forum.
After a year of negotiation, the NSF has announced a $20 million request for proposals for empirical AI safety research.
Here is the detailed program description.
The request for proposals is broad, as is common for NSF RfPs. Many safety avenues, such as transparency and anomaly detection, are in scope:
"reverse-engineering, inspecting, and interpreting the internal logic of learned models to identify unexpected behavior that could not be found by black-box testing alone"
"Safety also requires... methods for monitoring for unexpected environmental hazards or anomalous system behaviors, including during deployment."
Note that research that has high capabilities externalities is explicitly out…
playlist_add
Feb 27, 2023
AF - Counting-down vs. counting-up coherence by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Counting-down vs. counting-up coherence, published by Tsvi Benson-Tilsen on February 27, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. First completed 25 October 2022.]
Counting-down coherence is the coherence of a mind viewed as the absence of deviation downward in capability from ideal, perfectly efficient agency: the utility left on the table, the waste, the exploitability.
Counting-up coherence is the coherence of a mind viewed as the deviation upward in capability from a rock: the elements of the mind, and how they combine to perform tasks.
What determines the effects of a mind?
Supranormally capable minds can have large effects. To control those effects, we'd have to understand what determines the effects of a mind.
Pre-theoretically, we have the idea of "values", "aims", "wants". T…
playlist_add
Feb 26, 2023
AF - The Preference Fulfillment Hypothesis by Kaj Sotala
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Preference Fulfillment Hypothesis, published by Kaj Sotala on February 26, 2023 on The AI Alignment Forum.
Short version
Humans have an innate motivation ("preference fulfillment", PF) to fulfill the preferences of those they care about. It corresponds to at least some of the senses of the word "love", as well as related words such as "kindness" and "compassion".
I hypothesize that it works by simulating the other person and predicting what they would want or how they would like to be treated. PF is when you take your simulation of what other people would want and add an extra component that makes you intrinsically value outcomes that your simulation predicts the other people would prefer.
I also hypothesize that this is the same kind of simulation that forms our ability to work as a social species…
playlist_add
Feb 25, 2023
AF - Cognitive Emulation: A Naive AI Safety Proposal by Connor Leahy
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cognitive Emulation: A Naive AI Safety Proposal, published by Connor Leahy on February 25, 2023 on The AI Alignment Forum.
This is part of the work done at Conjecture.
This post has been reviewed before publication as per our infohazard policy. We thank our external reviewers for their comments and feedback.
This post serves as a signpost for Conjecture’s new primary safety proposal and research direction, which we call Cognitive Emulation (or “CoEm”). The goal of the CoEm agenda is to build predictably boundable systems, not directly aligned AGIs. We believe the former to be a far simpler and useful step towards a full alignment solution.
Unfortunately, given that most other actors are racing for as powerful and general AIs as possible, we won’t share much in terms of technical details for now.…
playlist_add
Feb 24, 2023
AF - Agents vs. Predictors: Concrete differentiating factors by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Agents vs. Predictors: Concrete differentiating factors, published by Evan Hubinger on February 24, 2023 on The AI Alignment Forum.
Thanks to Paul Christiano and Kate Woolverton for useful conversations and feedback.
In "Conditioning Predictive Models," we devote a lot of effort into trying to understand how likely predictive models are compared to other alternatives in realistic training regimes (and if we do get a predictive model how we might align it).
Here, I want to point to some very concrete behavioral differences that I think effectively differentiate predictive and non-predictive models both in theory and (hopefully) in practice as well. I think that thinking about predictive models as specifically those models that are sensitive to what I'll be talking about here can be quite clarifying, and…
playlist_add
Feb 24, 2023
AF - Christiano (ARC) and GA (Conjecture) Discuss Alignment Cruxes by Andrea Miotti
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Christiano (ARC) and GA (Conjecture) Discuss Alignment Cruxes, published by Andrea Miotti on February 24, 2023 on The AI Alignment Forum.
The following are the summary and transcript of a discussion between Paul Christiano (ARC) and Gabriel Alfour, hereafter GA (Conjecture), which took place on December 11, 2022 on Slack. It was held as part of a series of discussions between Conjecture and people from other organizations in the AGI and alignment field. See our retrospective on the Discussions for more information about the project and the format.
Here's a summary of the discussion, as well as the full transcript below the summary, lightly edited for readability.
Summary
Introduction
GA is pessimistic about alignment being solved because he thinks there is (1) an AGI race to the bottom, (2) alignment is…
playlist_add
Feb 24, 2023
AF - Retrospective on the 2022 Conjecture AI Discussions by Andrea Miotti
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Retrospective on the 2022 Conjecture AI Discussions, published by Andrea Miotti on February 24, 2023 on The AI Alignment Forum.
At the end of 2022, following the success of the 2021 MIRI Conversations, Conjecture started a project to host discussions about AGI and alignment with key people in the field. The goal was simple: surface positions and disagreements, identify cruxes, and make these debates public whenever possible for collective benefit.
Given that people and organizations will have to coordinate to best navigate AI's increasing effects, this is the first, minimum-viable coordination step needed to start from. Coordination is impossible without at least common knowledge of various relevant actors' positions and models.
People sharing their beliefs, discussing them and making as much as possibl…
playlist_add
Feb 24, 2023
AF - Sam Altman: "Planning for AGI and beyond" by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sam Altman: "Planning for AGI and beyond", published by Lawrence Chan on February 24, 2023 on The AI Alignment Forum.
(OpenAI releases a blog post detailing their AGI roadmap. I'm copying the text below, though see the linked blog post for better formatted version)
Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity.
If AGI is successfully created, this technology could help us elevate humanity by increasing abundance, turbocharging the global economy, and aiding in the discovery of new scientific knowledge that changes the limits of possibility.
AGI has the potential to give everyone incredible new capabilities; we can imagine a world where all of us have access to help with almost any cognitive task, providing a g…
playlist_add
Feb 24, 2023
AF - Meta open sources LMs competitive with Chinchilla, PaLM, and code-davinci-002 (Paper) by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta open sources LMs competitive with Chinchilla, PaLM, and code-davinci-002 (Paper), published by Lawrence Chan on February 24, 2023 on The AI Alignment Forum.
As the title says, Meta trained 4 foundational models with 7B, 13B, 33B, and 65B parameters respectively, and is open sourcing them for research. You can get their code on their Github repo: but you need to fill in a Google form to get the weights.
On downstream benchmarks, the models do comparably well with Chinchilla and PaLM and only a bit worse than Flan-PaLM-540B and code-davinci-002/text-davinci-002. (The authors don't evaluate on those models, but you can look at their performance from other work such as Stanford's HELM or Chung, Hou, Longpre et al's "Scaling Instruction-Finetuned Language Models".
Abstract:
We introduce LLaMA, a collect…
playlist_add
Feb 23, 2023
AF - Teleosemantics! by Abram Demski
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Teleosemantics!, published by Abram Demski on February 23, 2023 on The AI Alignment Forum.
I wanted to write a long, detailed, analytic post about this, somewhat like my Radical Probabilism post (to me, this is a similarly large update). However, I haven't gotten around to it for a long while. And perhaps it is better as a short, informal post in any case.
I think my biggest update over the past year has been a conversion to teleosemantics. Teleosemantics is a theory of semantics -- that is, "meaning" or "aboutness" or "reference".
To briefly state the punchline: Teleosemantics identifies the semantics of a symbolic construct as what the symbolic construct has been optimized to accurately reflect.
Previously, something seemed mysterious about the map/territory relationship. What could possibly imbue 'sy…
playlist_add
Feb 23, 2023
AF - AI that shouldn't work, yet kind of does by Donald Hobson
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI that shouldn't work, yet kind of does, published by Donald Hobson on February 23, 2023 on The AI Alignment Forum.
There are some things that work surprisingly well in AI. For example, AI that transfers the style of one image to the content of another. Why is the approach described here a hack. It starts with a neural net trained to classify images. It then runs gradient descent on the the content image, trying to get the covariance matrix of the style to match in early network layers, while trying to get the net layers of the original and style transfer images to be as similar as possible on the later layers.
So the approximations I think are making this work is that, in classifiers, the early layers tend to store simple features, and the later layers tend to hold more complex features. Style is base…
playlist_add
Feb 23, 2023
AF - EIS XII: Summary by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS XII: Summary, published by Stephen Casper on February 23, 2023 on The AI Alignment Forum.
Part 12 of 12 in the Engineer’s Interpretability Sequence.
TAISIC = “the AI safety interpretability community”
MI = “mechanistic interpretability”
There might be some addenda later, but for now, this is the final post in The Engineer’s Interpretability Sequence. I hope you have found it interesting and have gotten some useful ideas. I will always be happy to talk to people about the topics from this sequence in the comments or via email. For now, the last thing I will do is offer a summary of key points post by post :)
A Prequel: Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks (Räuker et al., 2022)
A survey of over 300 works on inner interpretability fro…
playlist_add
Feb 22, 2023
AF - EIS XI: Moving Forward by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS XI: Moving Forward, published by Stephen Casper on February 22, 2023 on The AI Alignment Forum.
Part 11 of 12 in the Engineer’s Interpretability Sequence.
So far, this sequence has discussed a number of topics in interpretability research, all building toward this post. Its goal is to explain some approaches that may be valuable moving forward. I plan to work on some of the ideas here soon. Others, I may not work on soon, but I would love to discuss and support such work if I am able. I hope that this post can offer some useful ideas for people entering or continuing with interpretability research, and if you would like to discuss anything here more, feel more than free to email me at scasper@mit.edu.
What are we working toward?
First, it seems useful to highlight two points that are uncontroversi…
playlist_add
Feb 22, 2023
AF - Cyborg Periods: There will be multiple AI transitions by Jan Kulveit
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cyborg Periods: There will be multiple AI transitions, published by Jan Kulveit on February 22, 2023 on The AI Alignment Forum.
It can be useful to zoom out and talk about very compressed concepts like ‘AI progress’ or ‘AI transition’ or ‘AGI timelines’. But from the perspective of most AI strategy questions, it’s useful to be more specific.
Looking at all of human history, it might make sense to think of ourselves as at the cusp of an AI transition, when AI systems overtake humans as the most powerful actors. But for practical and forward-looking purposes, it seems quite likely there will actually be multiple different AI transitions:
There will be AI transitions at different times in different domains
In each of these domains, transitions may move through multiple stages:
DescriptionPres…
playlist_add
Feb 22, 2023
AF - The Open Agency Model by Eric Drexler
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Open Agency Model, published by Eric Drexler on February 22, 2023 on The AI Alignment Forum.
Notes on AI for complex, consequential problems
Eric DrexlerCentre for the Governance of AIUniversity of Oxford
Introduction
This document argues for “open agencies” — not opaque, unitary agents — as the appropriate model for applying future AI capabilities to consequential tasks that call for combining human guidance with delegation of planning and implementation to AI systems. This prospect reframes and can help to tame a wide range of classic AI safety challenges, leveraging alignment techniques in a relatively fault-tolerant context.
Rethinking safe AI and its applications
AI safety research is too varied to summarize, yet broad patterns are obvious. A long-established reference-problem centers o…
playlist_add
Feb 21, 2023
AF - A proof of inner Löb's theorem by James Payor
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A proof of inner Löb's theorem, published by James Payor on February 21, 2023 on The AI Alignment Forum.
This is a short post that offers a slightly different take on the standard proof of Löb's theorem. It offers nothing else of any value :)
We seek to prove the "inner" version, which we write as: □P↔□(□PP)
The proof uses quining to build a related sentence L, the "Löb sentence", which talks about its own source code. By construction L has the property: □L↔□(□LP)
Then, we can show that □L↔□P, i.e. they're equivalent! We do this by plugging □L into itself to get a twisty □P. We can then replace each □L with □P and prove Löb's theorem.
The proof
This proof uses the same rules of box manipulation as on the wiki page. We start by creating L using quining, i.e. taking a mod…
playlist_add
Feb 21, 2023
AF - Pretraining Language Models with Human Preferences by Tomek Korbak
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pretraining Language Models with Human Preferences, published by Tomek Korbak on February 21, 2023 on The AI Alignment Forum.
This post summarizes the main results from our recently released paper Pretraining Language Models with Human Preferences, and puts them in the broader context of AI safety. For a quick summary of the paper, take a look at our Twitter thread.
TL;DR: In the paper, we show how to train LMs with human preferences (as in RLHF), but during LM pretraining. We find that pretraining works much better than the standard practice of only finetuning with human preferences after pretraining; our resulting LMs generate text that is more often in line with human preferences and are more robust to red teaming attacks. Our best method is conditional training, where we learn a predictive model of…
playlist_add
Feb 21, 2023
AF - EIS X: Continual Learning, Modularity, Compression, and Biological Brains by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS X: Continual Learning, Modularity, Compression, and Biological Brains, published by Stephen Casper on February 21, 2023 on The AI Alignment Forum.
Part 10 of 12 in the Engineer’s Interpretability Sequence.
The science of interpretability is part of a larger picture.
The previous post focused in-depth on how research on interpretability and adversaries are inseparably connected. This post is dedicated to discussing how this is not itself a complete story. There is a much larger, richer one about the connections between interpretability, adversaries, continual learning, modularity, and biological brains – likely some other things too. These connections may be a useful mine for insight and inspiration.
Below are discussions of my understanding of each of these topics and how they relate to others.…
playlist_add
Feb 21, 2023
AF - You're not a simulation, 'cause you're hallucinating by Stuart Armstrong
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You're not a simulation, 'cause you're hallucinating, published by Stuart Armstrong on February 21, 2023 on The AI Alignment Forum.
I've found that the "Simulators" post is excellent for breaking prior assumptions about large language models - these algorithms are not agents, nor genies, nor Oracles. They are currently something very different.
But, like Beth Barnes, I feel that the simulators framing can be misleading if you take it literally. And hallucinations often provide examples of where "the model is predicting what token would appear next in the training data given the input tokens" gives a better model than "simulators".
For example, here are some reviews of fictional films, written by canonically quite truthful characters:
And:
If we used the simulator view, we might expect that these truthfu…
playlist_add
Feb 20, 2023
AF - Behavioral and mechanistic definitions (often confuse AI alignment discussions) by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Behavioral and mechanistic definitions (often confuse AI alignment discussions), published by Lawrence Chan on February 20, 2023 on The AI Alignment Forum.
TL;DR: It’s important to distinguish between behavioral definitions – which categorize objects based on outside observable properties – and mechanistic definitions – which categorize objects based on their internal mechanisms. In this post, I give several examples of terms which can be defined either behaviorally and mechanistically. Then, I talk about the pros and cons of both kinds of definitions, and how this distinction relates to the distinction between gears-level versus black-box models.
Related to: Most similar to John Wentworth’s Gears and Behaviors, but about definitions rather than models. Also in…
playlist_add
Feb 20, 2023
AF - There are no coherence theorems by Dan H
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There are no coherence theorems, published by Dan H on February 20, 2023 on The AI Alignment Forum.
[Written by EJT as part of the CAIS Philosophy Fellowship. Thanks to Dan for help posting to the Alignment Forum]
Introduction
For about fifteen years, the AI safety community has been discussing coherence arguments. In papers and posts on the subject, it’s often written that there exist coherence theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. Despite the prominence of these arguments, authors are often a little hazy about exactly which theorems qualify as coherence theorems. This is no accident. If the authors had tried to be precise, they would have discovered t…
playlist_add
Feb 20, 2023
AF - EIS IX: Interpretability and Adversaries by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS IX: Interpretability and Adversaries, published by Stephen Casper on February 20, 2023 on The AI Alignment Forum.
Part 9 of 12 in the Engineer’s Interpretability Sequence.
Thanks to Nikolaos Tsilivis for helpful discussions.
The studies of interpretability and adversaries are inseparable.
There are several key connections between the two. Some works will be cited below, but please refer to page 9 of the Toward Transparent AI survey (Räuker et al., 2022) for full citations. There are too many to be worth the clutter in this post.
1. More interpretable networks are more adversarially robust and more adversarially robust networks are more interpretable.
The main vein of evidence on this topic comes from a set of papers which study how regularizing feature attribution/saliency maps to make them more…
playlist_add
Feb 20, 2023
AF - [MLSN #8] Mechanistic interpretability, using law to inform AI alignment, scaling laws for proxy gaming by Dan H
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [MLSN #8] Mechanistic interpretability, using law to inform AI alignment, scaling laws for proxy gaming, published by Dan H on February 20, 2023 on The AI Alignment Forum.
As part of a larger community building effort, CAIS is writing a safety newsletter that is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here.
Welcome to the 8th issue of the ML Safety Newsletter! In this edition, we cover:
Isolating the specific mechanism that GPT-2 uses to identify the indirect object in a sentence
When maximum softmax probability is optimal
How law can inform specification for AI systems
Using language models to find a group consensus
Scaling laws for proxy gaming
An adversarial attack on ad…
playlist_add
Feb 19, 2023
AF - EIS VIII: An Engineer’s Understanding of Deception by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS VIII: An Engineer’s Understanding of Deception, published by Stephen Casper on February 19, 2023 on The AI Alignment Forum.
Part 8 of 12 in the Engineer’s Interpretability Sequence.
As AI safety researchers, we care a lot about deception. So what is it? Hubinger et al. (2019) write:
If the mesa-optimizer has an objective that extends across parameter updates, then it will be incentivized to avoid being modified, as it might not pursue the same objective after modification (with the result that its current objective will not be achieved in future iterations). This means that the mesa-optimizer will be instrumentally incentivized to act as if it is optimizing the base objective function, even if its actual mesa-objective is something else entirely. We will refer to this hypothetical phenomenon as…
playlist_add
Feb 19, 2023
AF - Does novel understanding imply novel agency / values? by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Does novel understanding imply novel agency / values?, published by Tsvi Benson-Tilsen on February 19, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. First completed 17 October 2022.]
To have large relevant effects on the world, a mind has to understand a lot about the world. The mind has to have a lot of the structure of the cosmos (the entirety of the world, in any aspect or abstraction) highly accessible to itself for use in skillful action. To understand a lot about the world, the mind has to gain a lot of understanding that it didn't have previously. When a mind gains understanding, that's a change in the mind. Does that change have to include a change to the values of the mind?
Thanks to Sam Eisenstat for related conversations and ideas, e.g. provisionality.
Terms
This essay uses ter…
playlist_add
Feb 18, 2023
AF - AGI in sight: our look at the game board by Andrea Miotti
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI in sight: our look at the game board, published by Andrea Miotti on February 18, 2023 on The AI Alignment Forum.
From our point of view, we are now in the end-game for AGI, and we (humans) are losing. When we share this with other people, they reliably get surprised. That’s why we believe it is worth writing down our beliefs on this.
1. AGI is happening soon. Significant probability of it happening in less than 5 years.
Five years ago, there were many obstacles on what we considered to be the path to AGI.
But in the last few years, we’ve gotten:
Powerful Agents (Agent57, GATO, Dreamer V3)
Reliably good Multimodal Models (StableDiffusion, Whisper, Clip)
Just about every language tasks (GPT3, ChatGPT, Bing Chat)
Human and Social Manipulation
Robots (Boston Dynamics, Day Dreamer, VideoDex, RT-1: Ro…
playlist_add
Feb 18, 2023
AF - Parametrically retargetable decision-makers tend to seek power by Alex Turner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Parametrically retargetable decision-makers tend to seek power, published by Alex Turner on February 18, 2023 on The AI Alignment Forum.
This paper—accepted as a poster to NeurIPS 2022— is the sequel to Optimal Policies Tend to Seek Power. The new theoretical results are extremely broad, discarding the requirements of full observability, optimal policies, or even requiring a finite number of options.
Abstract:
If capable AI agents are generally incentivized to seek power in service of the objectives we specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have an optimal policy which seeks power by keeping options open and staying alive. However, the real world is neither fully observable, nor must tra…
playlist_add
Feb 18, 2023
AF - EIS VII: A Challenge for Mechanists by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS VII: A Challenge for Mechanists, published by Stephen Casper on February 18, 2023 on The AI Alignment Forum.
Part 7 of 12 in the Engineer’s Interpretability Sequence.
Thanks to Neel Nanda. I used some very nicely-written code of his from here. And thanks to both Chris Olah and Neel Nanda for briefly discussing this challenge with me.
MI = “mechanistic interpretability”
Given a network, recover its labeling function.
In the last post, I argued that existing works in MI focus on solving problems that are too easy. Here, I am posing a challenge for mechanists that is still a toy problem but one that is quite a bit less convenient than studying a simple model or circuit implementing a trivial, known task. The the best of my knowledge:
Unlike prior work on MI from the AI safety interpretability com…
playlist_add
Feb 17, 2023
AF - Two problems with ‘Simulators’ as a frame by Ryan Greenblatt
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Two problems with ‘Simulators’ as a frame, published by Ryan Greenblatt on February 17, 2023 on The AI Alignment Forum.
(Thanks to Lawrence Chan and Buck Shlegeris for comments. Thanks to Nate Thomas for many comments and editing)
Despite appreciating and agreeing with various specific points made in the Simulators post, I broadly think that the term ‘simulator’ and the corresponding frame probably shouldn’t be used. Instead, I think we should just directly reason about predictors and think in terms of questions such as ‘what would the model predict for the next token?’[1]
In this post, I won’t make arguments that I think are strong enough to decisively justify this claim, but I will argue for two points that support it:
The word ‘simulation’ as used in the Simulators post doesn’t…
playlist_add
Feb 17, 2023
AF - EIS VI: Critiques of Mechanistic Interpretability Work in AI Safety by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS VI: Critiques of Mechanistic Interpretability Work in AI Safety, published by Stephen Casper on February 17, 2023 on The AI Alignment Forum.
Part 6 of 12 in the Engineer’s Interpretability Sequence.
Thanks to Chris Olah and Neel Nanda for discussions and comments. In particular, I am thankful to Neel Nanda correcting a mistake I made in understanding the arguments in Olsson et al. (2022) in an earlier draft of this post.
TAISIC = “the AI safety interpretability community”
MI = “mechanistic interpretability”
What kind of work this post focused on
TAISIC prioritizes a relatively small set of problems in interpretability relative to the research community at large. This work is not homogenous, but a dominant theme is a focus on mechanistic, circuits-style interpretability with the end goals o…
playlist_add
Feb 17, 2023
AF - One-layer transformers aren’t equivalent to a set of skip-trigrams by Buck Shlegeris
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: One-layer transformers aren’t equivalent to a set of skip-trigrams, published by Buck Shlegeris on February 17, 2023 on The AI Alignment Forum.
(thanks to Tao Lin and Ryan Greenblatt for pointing this out, and to Arthur Conmy, Jenny Nitishinskaya, Thomas Huck, Neel Nanda, and Lawrence Chan, Ben Toner, and Chris Olah for comments, and many others for useful discussion.)
In “A Mathematical Framework for Transformer Circuits”, Elhage et al write (among similar sentences):
One layer attention-only transformers are an ensemble of bigram and “skip-trigram” (sequences of the form "A. B C") models. The bigram and skip-trigram tables can be accessed directly from the weights, without running the model.
I interpret this as a mathematical claim–that the attention layer of a one-layer transformer can be…
playlist_add
Feb 17, 2023
AF - Automating Consistency by Hoagy
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Automating Consistency, published by Hoagy on February 17, 2023 on The AI Alignment Forum.
tldr: Ask models to justify statements. Remove context, ask if statements are true/good. If not, penalise. Apply this again to the justifying statements.
Status: Just a quick thought. Doubt this is a new idea but I don't think I've encountered it. Happy to delete if it's a duplicate.
Mods: If you think this is closer to capability work than alignment work please remove.
Background
A failure of current LLMs is that after they've said something that's incorrect, they can then double down and spout nonsense to try and justify their past statements. (Exhibit A: Sydney Bing vs Avatar 2) We can suppress this by giving it poor ratings in RLHF, but perhaps we can do better by automating the process.
Setup:
We start with a…
playlist_add
Feb 16, 2023
AF - Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic) by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic), published by Lawrence Chan on February 16, 2023 on The AI Alignment Forum.
This is a followup to what I cheekily call Anthropic's "just try to get the large model to do what you want" research agenda. (Previously: A General Language Assistant as a Laboratory for Alignment, Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Language Models (Mostly) Know What They Know)
The most interesting takeaway for me is that this is the first paper where Anthropic benchmarks their 175B parameter language model (probably a Claude variant). Previous papers only benchmarked up to 52B parameters. However, we don't have the performance of this model on standard benchmarks (the only benchmarked model…
playlist_add
Feb 16, 2023
AF - EIS V: Blind Spots In AI Safety Interpretability Research by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS V: Blind Spots In AI Safety Interpretability Research, published by Stephen Casper on February 16, 2023 on The AI Alignment Forum.
Part 5 of 12 in the Engineer’s Interpretability Sequence.
Thanks to Anson Ho, Chris Olah, Neel Nanda, and Tony Wang for some discussions and comments.
TAISIC = “the AI safety interpretability community”
MI = “mechanistic interpretability”
Most AI safety interpretability work is conducted by researchers in a relatively small number of places, and TAISIC is closely-connected by personal relationships and the AI alignment forum. Much of the community is focused on a few specific approaches like circuits-style MI, mechanistic anomaly detection, causal scrubbing, and probing. But this is a limited set of topics, and TAISIC might benefit from broader engagement. In t…
playlist_add
Feb 15, 2023
AF - EIS IV: A Spotlight on Feature Attribution/Saliency by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS IV: A Spotlight on Feature Attribution/Saliency, published by Stephen Casper on February 15, 2023 on The AI Alignment Forum.
Part 4 of 12 in the Engineer’s Interpretability Sequence.
Thanks to Tony Wang for a helpful comment.
If you want to become more familiar with feature attribution/saliency, a tutorial on them that may offer useful background is Nielsen et al. (2021).
Given a model and an input for it, the goal of feature attribution/saliency methods is to identify what features in the input are influential for the model’s decision. The literature on these methods is large and active with many hundreds of papers. In fact, in some circles, the word “interpretability” and especially the word “explainability” are more or less synonymous with feature attribution (some examples are discus…
playlist_add
Feb 15, 2023
AF - Don't accelerate problems you're trying to solve by Andrea Miotti
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don't accelerate problems you're trying to solve, published by Andrea Miotti on February 15, 2023 on The AI Alignment Forum.
If one believes that unaligned AGI is a significant problem (>10% chance of leading to catastrophe), speeding up public progress towards AGI is obviously bad.
Though it is obviously bad, there may be circumstances which require it. However, accelerating AGI should require a much higher bar of evidence and much more extreme circumstances than is commonly assumed.
There are a few categories of arguments that claim intentionally advancing AI capabilities can be helpful for alignment, which do not meet this bar.
Two cases of this argument are as follows
It doesn't matter much to do work that pushes capabilities if others are likely to do the same or similar work shortly after.
We shou…
playlist_add
Feb 14, 2023
AF - EIS III: Broad Critiques of Interpretability Research by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS III: Broad Critiques of Interpretability Research, published by Stephen Casper on February 14, 2023 on The AI Alignment Forum.
Part 3 of 12 in the Engineer’s Interpretability Sequence.
Right now, interpretability is a major subfield in the machine learning research community. As mentioned in EIS I, there is so much work in interpretability that there is now a database of 5199 interpretability papers (Jacovi, 2023). You can also look at a survey from some coauthors and me on over 300 works on interpreting network internals (Räuker et al., 2022).
The key promise of interpretability is to offer open-ended ways of understanding and evaluating models that help us with AI safety. And the diversity of approaches to interpretability is encouraging since we want build a toolbox full of many different usef…
playlist_add
Feb 14, 2023
AF - The Cave Allegory Revisited: Understanding GPT's Worldview by Jan Kulveit
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Cave Allegory Revisited: Understanding GPT's Worldview, published by Jan Kulveit on February 14, 2023 on The AI Alignment Forum.
A short post describing a metaphor I find useful, in particular for explaining some intuitions about systems like GPT to people who don't have deeper technical knowledge about large generative models.
Plato's allegory of the cave has been a staple of philosophical discourse for millenia, providing a metaphor for understanding the limits of human perception. In the classical allegory, we are prisoners shackled to a wall of a cave, unable to experience reality directly but only able to infer it based on watching shadows cast on the wall.GPT can be thought of as a blind oracle residing in a deeper cave, where it does not even see the shadows but only hears our conversations i…
playlist_add
Feb 12, 2023
AF - The conceptual Dopplegänger problem by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The conceptual Dopplegänger problem, published by Tsvi Benson-Tilsen on February 12, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. First completed 9 October 2022.]
Suppose we want to observe the thoughts of a mind in order to detect whether it's making its way towards a plan to harm us, and ideally also to direct the mind so that it pursues specific aims. To this end, we might hope that the mind and its thinking are organized in a way we can come to understand in the way that we understand ourselves and our thinking.
We might hope that when the mind considers plans that involve something, e.g. plans that involve the coffee cup, it does so using a concept alike to our concept [[coffee cup]]. When the mind recognizes, predicts, imagines, simulates, manipulates, designs, combines things wit…
playlist_add
Feb 11, 2023
AF - Why almost every RL agent does learned optimization by Lee Sharkey
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why almost every RL agent does learned optimization, published by Lee Sharkey on February 12, 2023 on The AI Alignment Forum.
Or "Why RL≈RL2 (And why that matters)"
TL;DR: This post discusses the blurred conceptual boundary between RL and RL2 (also known as meta-RL). RL2 is an instance of learned optimization. Far from being a special case, I point out that the conditions under which RL2 emerges are actually the default conditions for RL training. I argue that this is safety-relevant by outlining the evidence for why learned planning algorithms will probably emerge -- and have probably already emerged in a weak sense -- in scaled-up RL2 agents.
I've found myself telling this story about the relationship between RL and RL2 numerous times in conversation. When that happens, it's usually time to write a…
playlist_add
Feb 11, 2023
AF - A note on 'semiotic physics' by metasemi
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A note on 'semiotic physics', published by metasemi on February 11, 2023 on The AI Alignment Forum.
Introduction
This is an attempt to explain to myself the concept of semiotic physics that appears in the original Simulators post by janus and in a later post by Jan Hendrik Kirchner. Everything here comes from janus and Jan's work, but any inaccuracies or misinterpretations are all mine.
TL;DR
The prototypical simulator, GPT, is sometimes said to "predict the next token" in a text sequence. This is accurate, but incomplete.
It's more illuminating to consider what happens when GPT, or any simulator, is run repeatedly to produce a multi-token forward trajectory, as in the familiar scenario of generating a text completion in response to a prompt.
The token-by-token production of output is stochastic, with a…
playlist_add
Feb 10, 2023
AF - The Need For Work On Technical AI Alignment (Intro Explainer) by Daniel Eth
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Need For Work On Technical AI Alignment (Intro Explainer), published by Daniel Eth on February 11, 2023 on The AI Alignment Forum.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
playlist_add
Feb 10, 2023
AF - A proposed method for forecasting transformative AI by Matthew Barnett
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A proposed method for forecasting transformative AI, published by Matthew Barnett on February 10, 2023 on The AI Alignment Forum.
In 2021, I proposed measuring progress in the perplexity of language models and extrapolating past results to determine when language models were expected to reach roughly "human-level" performance. Here, I build on that approach by introducing a more systematic and precise method of forecasting progress in language modeling that employs scaling laws to make predictions.
The full report for this forecasting method can be found in this document. In this blog post I'll try to explain all the essential elements of the approach without providing excessive detail regarding the technical derivations.
This approach can be contrasted with Ajeya Cotra's Bio Anchors model, providing a…
playlist_add
Feb 10, 2023
AF - Why I’m not working on {debate, RRM, ELK, natural abstractions} by Steve Byrnes
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I’m not working on {debate, RRM, ELK, natural abstractions}, published by Steve Byrnes on February 10, 2023 on The AI Alignment Forum.
[For background & spelling out the acronyms in the title, see: Debate (AI safety technique), Recursive Reward Modeling, Eliciting Latent Knowledge, Natural Abstractions.]
When I say “Why I’m not working on X”, I am NOT trying to find a polite & diplomatic way to say “Nobody should work on X because X is unhelpful for AGI safety”. Hmm, OK, well, maybe it’s just a little bit that. But really, I don’t feel strongly. Instead, I think:
A lot of disagreement about what a solution to technical AGI safety looks like is really downstream of disagreements about questions like “How will AGI be built? What will it look like? How will it work?”
Nobody really k…
playlist_add
Feb 10, 2023
AF - Conditioning Predictive Models: Open problems, Conclusion, and Appendix by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Open problems, Conclusion, and Appendix, published by Evan Hubinger on February 10, 2023 on The AI Alignment Forum.
This is the final of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper.
7. Open problems
We think that there are a wide variety of ways—both experimental and theoretical—in which our analysis could be expanded upon. Here, we’ll try to briefly lay out some of the future directions that we are most excited about—though note that this is only a sampling of some possible future directions, and is thus a highly in…
playlist_add
Feb 10, 2023
AF - Cyborgism by Nicholas Kees Dupuis
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cyborgism, published by Nicholas Kees Dupuis on February 10, 2023 on The AI Alignment Forum.
Thanks to Garrett Baker, David Udell, Alex Gray, Paul Colognese, Akash Wasil, Jacques Thibodeau, Michael Ivanitskiy, Zach Stein-Perlman, and Anish Upadhayay for feedback on drafts, as well as Scott Viteri for our valuable conversations.
Executive summary: This post proposes a strategy for safely accelerating alignment research. The plan is to set up human-in-the-loop systems which empower human agency rather than outsource it, and to use those systems to differentially accelerate progress on alignment.
Introduction: An explanation of the context and motivation for this agenda.
Automated Research Assistants: A discussion of why the paradigm of training AI systems to behave as autonomous agents is both counterprod…
playlist_add
Feb 10, 2023
AF - FLI Podcast: Connor Leahy on AI Progress, Chimps, Memes, and Markets (Part 1/3) by remember
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FLI Podcast: Connor Leahy on AI Progress, Chimps, Memes, and Markets (Part 1/3), published by remember on February 10, 2023 on The AI Alignment Forum.
We often prefer reading over listening to audio content, and have been testing transcribing podcasts using our new tool at Conjecture, Verbalize, with some light editing and formatting. We're posting highlights and transcripts of podcasts in case others share our preferences, and because there is a lot of important alignment-relevant information in podcasts that never made it to LessWrong.If anyone is creating alignment-relevant audio content and wants to transcribe it, get in touch with us and we can give you free credits!
The podcast episode transcribed in this post is available here.
Topics covered include:
Defining artificial general intelligence
What…
playlist_add
Feb 9, 2023
AF - Conditioning Predictive Models: Deployment strategy by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Deployment strategy, published by Evan Hubinger on February 9, 2023 on The AI Alignment Forum.
This is the sixth of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out.
6. Deployment strategy
Previously, we have been focusing on how to make conditioning predictive models as safe and competitive as possible. Now, we want to take a step back and discuss considerations for using conditioning predicti…
playlist_add
Feb 9, 2023
AF - EIS II: What is “Interpretability”? by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS II: What is “Interpretability”?, published by Stephen Casper on February 9, 2023 on The AI Alignment Forum.
Part 2 of 12 in the Engineer’s Interpretability Sequence.
A parable based on a true story
In 2015, a Google’s image classification app classified many photos depicting black people as gorillas. Image from WSJ.
Remember Google’s infamous blunder from 2015 in which users found that one of its vision API’s often misclassified black people as gorillas? Consider a parable of two researchers who want to understand and tackle this issue.
Alice is an extremely skilled mechanistic interpretability researcher who spends a heroic amount of effort analyzing Google’s model. She identifies a set of neurons and weights that seem to be involved in the detection and processing of human and gorill…
playlist_add
Feb 9, 2023
AF - The Engineer’s Interpretability Sequence (EIS) I: Intro by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Engineer’s Interpretability Sequence (EIS) I: Intro, published by Stephen Casper on February 9, 2023 on The AI Alignment Forum.
Part 1 of 12 in the Engineer’s Interpretability Sequence.
If we want to reduce near and long term risks from AI, we should care a lot about interpretability tools. This is a very uncontroversial claim to make inside the AI safety community. Almost every agenda for safe advanced AI incorporates interpretability in some way. The key value of interpretability tools is that they aid in human oversight by enabling open-ended evaluation.
Short of actually deploying a system, any method of evaluating it can only be a proxy for its actual performance. The most common way to evaluate a model is by its performance in some test set or environment. But test sets alone can fail to r…
playlist_add
Feb 8, 2023
AF - Notes on the Mathematics of LLM Architectures by Spencer Becker-Kahn
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Notes on the Mathematics of LLM Architectures, published by Spencer Becker-Kahn on February 9, 2023 on The AI Alignment Forum.
From a mathematical point of view, the building and training of a large transformerlanguage model (LLM) is the construction of a certain function, from some euclidean space to another, that has certain interesting properties. And it may therefore be surprising to find that many key papers announcing significant new LLMs seem reluctant to simply spell out the details of the function that they have constructed in plain mathematical language or indeed even in complete pseudo-code. The latter form of this complaint is the subject of the recent article of Phuong and Hutter [1]. Here, we focus on one aspect of the former perspective and seek to give a relatively ‘pure’ mathematica…
playlist_add
Feb 8, 2023
AF - On Developing a Mathematical Theory of Interpretability by Spencer Becker-Kahn
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On Developing a Mathematical Theory of Interpretability, published by Spencer Becker-Kahn on February 9, 2023 on The AI Alignment Forum.
If the trajectory of the deep learning paradigm continues, it seems plausible to me that in order for applications of low-level interpretability to AI not-kill-everyone-ism to be truly reliable, we will need a much better-developed and more general theoretical and mathematical framework for deep learning than currently exists. And this sort of work seems difficult. Doing mathematics carefully - in particular finding correct, rigorous statements and then finding correct proofs of those statements - is slow. So slow that the rate of change of cutting-edge engineering practices significantly worsens the difficulties involved in building theory at the right level of genera…
playlist_add
Feb 8, 2023
AF - A note on terminology: AI Alignment != AI x-safety by David Scott Krueger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A note on terminology: AI Alignment != AI x-safety, published by David Scott Krueger on February 8, 2023 on The AI Alignment Forum.
I think the terms "AI Alignment" and "AI existential safety" are often used interchangeably, leading the ideas to be conflated.In practice, I think "AI Alignment" is mostly used in one of the following three ways, and should be used exclusively for Intent Alignment (with some vagueness about whose intent, e.g. designer vs. user):1) AI Alignment = How to get AI systems to do what we want
2) AI Alignment = How to get AI systems to try to do what we want
3) AI Alignment = A rebranding of “AI (existential) safety”... A community of people trying to reduce the chance of AI leading to premature human extinction.
The problem with (1) is that it is too broad, and invites the re…
playlist_add
Feb 8, 2023
AF - Conditioning Predictive Models: Interactions with other approaches by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Interactions with other approaches, published by Evan Hubinger on February 8, 2023 on The AI Alignment Forum.
This is the fifth of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out.
5. Interactions with other approaches
Imitation learning
One very related approach is imitation learning: rather than try to predict humans, as we are proposing, one could simply train to imitate them instead. Such a…
playlist_add
Feb 7, 2023
AF - OpenAI/Microsoft announce "next generation language model" integrated into Bing/Edge by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: OpenAI/Microsoft announce "next generation language model" integrated into Bing/Edge, published by Lawrence Chan on February 7, 2023 on The AI Alignment Forum.
TL;DR: Microsoft and OpenAI announced a new version of Bing featuring "a new, next-generation OpenAI large language model [..] more powerful than ChatGPT", and that Microsoft Edge will feature a Copilot-like assistant that helps with composing and summarizing content.
Brief thoughts/comments/notes:
Microsoft's attitude during their press meeting seemed pretty aggressive and targeted directly at racing with Google. This seems kind of bad. For example, a quote from Nadella:
"The race starts today, and we’re going to move and move fast. Most importantly, we want to have a lot of fun innovating again in search, because it’s high time."
Microsoft…
playlist_add
Feb 7, 2023
AF - Conditioning Predictive Models: Making inner alignment as easy as possible by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Making inner alignment as easy as possible, published by Evan Hubinger on February 7, 2023 on The AI Alignment Forum.
This is the fourth of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out.
4. Making inner alignment as easy as possible
At the beginning, we posited the assumption that large language models could be well-understood as predictive models of the world. At the time, however, that was…
playlist_add
Feb 6, 2023
AF - Conditioning Predictive Models: The case for competitiveness by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: The case for competitiveness, published by Evan Hubinger on February 6, 2023 on The AI Alignment Forum.
This is the third of seven posts in the Conditioning Predictive Models Sequence based on the forthcoming paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out.
3. The case for competitiveness
In addition to ensuring that we can condition predictive models safely, for such an approach to work as a way to actually reduce AI existential risk, we also need it to be the ca…
playlist_add
Feb 6, 2023
AF - Decision Transformer Interpretability by Joseph Bloom
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Decision Transformer Interpretability, published by Joseph Bloom on February 6, 2023 on The AI Alignment Forum.
TLDR: We analyse how a small Decision Transformer learns to simulate agents on a grid world task, providing evidence that it is possible to do circuit analysis on small models which simulate goal-directedness. We think Decision Transformers are worth exploring further and may provide opportunities to explore many alignment-relevant deep learning phenomena in game-like contexts.
Link to the GitHub Repository. Link to the Analysis App. I highly recommend using the app if you have experience with mechanistic interpretability. All of the mechanistic analysis should be reproducible via the app.
Key Claims
A 1-Layer Decision Transformer learns several contextual behaviours which are activated by a c…
playlist_add
Feb 5, 2023
AF - Gradient surfing: the hidden role of regularization by Jesse Hoogland
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gradient surfing: the hidden role of regularization, published by Jesse Hoogland on February 6, 2023 on The AI Alignment Forum.
Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort
In a previous post, I demonstrated that Brownian motion near singularities defies our expectations from "regular" physics. Singularities trap random motion and take up more of the equilibrium distribution than you'd expect from the Gibbs measure.
In the computational probability community, this is a well-known pathology. Sampling techniques like Hamiltonian Monte Carlo get stuck in corners, and this is something to avoid. You typically don't want biased estimates of the distribution you're trying to sample.
In deep learning, I argued, this behavior might be less a bug than a feature.
The clai…
playlist_add
Feb 5, 2023
AF - SolidGoldMagikarp (plus, prompt generation) by Jessica Rumbelow
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SolidGoldMagikarp (plus, prompt generation), published by Jessica Rumbelow on February 5, 2023 on The AI Alignment Forum.
Work done at SERI-MATS, over the past two months, by Jessica Rumbelow and Matthew Watkins.
TL;DR
Anomalous tokens: a mysterious failure mode for GPT (which reliably insulted Matthew)
We have found a set of anomalous tokens which result in a previously undocumented failure mode for GPT-2 and GPT-3 models. (The 'instruct' models “are particularly deranged” in this context, as janus has observed.)
Many of these tokens reliably break determinism in the OpenAI GPT-3 playground at temperature 0 (which theoretically shouldn't happen).
Prompt generation: a new interpretability method for language models (which reliably finds prompts that result in a target completion). This is good for:…
playlist_add
Feb 5, 2023
AF - Control by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Control, published by Tsvi Benson-Tilsen on February 5, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. First completed 3 July 2022.]
I don't know how to define control or even point at it except as a word-cloud, so it's probably wanting to be refactored. The point of talking about control is to lay part of the groundwork for understanding what determines what directions a mind ends up pushing the world in. Control is something like what's happening when values or drives are making themselves felt as values or drives. ("Influence" = "in-flow" might be a better term than "control".)
Previous: Structure, creativity, and novelty
Definitions of control
Control is when an element makes another element do something. This relies on elements "doing stuff".
Control is when an element {counterfactual…
playlist_add
Feb 4, 2023
AF - Evaluations (of new AI Safety researchers) can be noisy by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Evaluations (of new AI Safety researchers) can be noisy, published by Lawrence Chan on February 5, 2023 on The AI Alignment Forum.
Related work: Hero Licensing, Modest Epistemology, The Alignment Community is Culturally Broken, Status Regulation and Anxious Underconfidence, Touch reality as soon as possible, and many more.
TL;DR: Evaluating whether or not someone will do well at a job is hard, and evaluating whether or not someone has the potential to be a great AI safety researcher is even harder. This applies to evaluations from other people (e.g. job interviews, first impressions at conferences) but especially to self-evaluations. Performance is also often idiosyncratic: people who do poorly in one role may do well in others, even superficially similar ones. As a result, I think people should not tak…
playlist_add
Feb 4, 2023
AF - Modal Fixpoint Cooperation without Löb's Theorem by Andrew Critch
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Modal Fixpoint Cooperation without Löb's Theorem, published by Andrew Critch on February 5, 2023 on The AI Alignment Forum.
TL;DR: This post introduces a novel logical approach to achieving group-scale cooperation, based on modal fixpoint theory. This approach is both easier to understand and roughly 3x more efficient than previous approaches that factored through Löb's Theorem, measured in terms of the length / complexity of the proofs involved.
The following lemma is due to James Payor:
Lemma: If ⊢x↔□(□xx)) then ⊢x .
Proof: The proof uses the same modal rules of inference for □ as Löb's theorem, namely, necessitation and distributivity:
⊢x↔□(□xx)), by assumption.
⊢x(□xx), by tautology (A(BA)).
⊢□x□(□xx), from 2 by □ necessitation and distributivity.
⊢□xx, from…
playlist_add
Feb 4, 2023
AF - Mech Interp Project Advising Call: Memorisation in GPT-2 Small by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mech Interp Project Advising Call: Memorisation in GPT-2 Small, published by Neel Nanda on February 4, 2023 on The AI Alignment Forum.
I've recently been having advising calls with REMIX teams (Redwood's interpretability sprint) trying to give advice & feedback on projects. As an experiment, I've published a recording of one advising call (with Tessa Barton & Kushal Jain on memorisation in GPT-2 Small), I'm curious whether this is useful to anyone! IMO getting detailed feedback from a more experienced research is one of the best ways to improve at research, but have no idea whether someone else's feedback is comparatively useful, or whether my advice is good enough lol. Thanks to the team for being down to publish this, and the work!
Thanks for listening. To help us out with The Nonlinear Library or to…
playlist_add
Feb 2, 2023
AF - Normative vs Descriptive Models of Agency by Matt MacDermott
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Normative vs Descriptive Models of Agency, published by Matt MacDermott on February 2, 2023 on The AI Alignment Forum.
Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.
I think there's an important distinction to be made between work in agent foundations which is concerned with normative models, and work which is concerned with descriptive models. They are increasingly separate bodies of work, with different aims and different theories of change when it comes to alignment.
Examples
The normative branch is typified by the Embedded Agency sequence, and the whole thing can be summed up as 'The Hunt for Embedded AIXI'. Its goal is to figure out how to build an ideal agent in principle. Decision theory, infrabayesianism, and logical induction all come under the normative…
playlist_add
Feb 2, 2023
AF - Conditioning Predictive Models: Outer alignment via careful conditioning by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Outer alignment via careful conditioning, published by Evan Hubinger on February 2, 2023 on The AI Alignment Forum.
This is the second of seven posts in the Conditioning Predictive Models Sequence based on the forthcoming paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out. We are starting with posts one and two, with post two being the largest and most content-rich of all seven.
2. Outer alignment via careful conditioning
Suppose we actually get a predictive model of…
playlist_add
Feb 2, 2023
AF - Conditioning Predictive Models: Large language models as predictors by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Predictive Models: Large language models as predictors, published by Evan Hubinger on February 2, 2023 on The AI Alignment Forum.
This is the first of seven posts in the Conditioning Predictive Models Sequence based on the forthcoming paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper. We will be releasing posts gradually over the course of the next week or so to give people time to read and digest them as they come out. We are starting with posts one and two, with post two being the largest and most content-rich of all seven.
Thanks to Paul Christiano, Kyle McDonell, Laria Reynolds, Collin Burns, Rohin Shah, Ethan Perez…
playlist_add
Feb 2, 2023
AF - More findings on maximal data dimension by Marius Hobbhahn
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More findings on maximal data dimension, published by Marius Hobbhahn on February 2, 2023 on The AI Alignment Forum.
Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.
I’d like to thank Wes Gurnee, Aryan Bhatt, Eric Purdy and Stefan Heimersheim for discussions and Evan Hubinger, Neel Nanda, Adam Jermyn and Chris Olah for mentorship and feedback.
The post contains a lot of figures, so the suggested length is deceiving. Code can be found in this colab notebook.
This is the second in a series of N posts on trying to understand memorization in NNs.
Executive summary
I look at a variety of settings and experiments to better understand memorization in toy models. My primary motivation is to increase our general understanding of NNs but I also suspect that understanding m…
playlist_add
Feb 2, 2023
AF - Temporally Layered Architecture for Adaptive, Distributed and Continuous Control by Roman Leventov
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Temporally Layered Architecture for Adaptive, Distributed and Continuous Control, published by Roman Leventov on February 2, 2023 on The AI Alignment Forum.
A preprint is published by Devdhar Patel, Joshua Russell, Francesca Walsh, Tauhidur Rahman, Terrance Sejnowski, and Hava Siegelmann in December 2022.Abstract:
We present temporally layered architecture (TLA), a biologically inspired system for temporally adaptive distributed control. TLA layers a fast and a slow controller together to achieve temporal abstraction that allows each layer to focus on a different time-scale. Our design is biologically inspired and draws on the architecture of the human brain which executes actions at different timescales depending on the environment's demands. Such distributed control design is widespread across biologi…
playlist_add
Feb 1, 2023
AF - Research agenda: Formalizing abstractions of computations by Erik Jenner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Research agenda: Formalizing abstractions of computations, published by Erik Jenner on February 2, 2023 on The AI Alignment Forum.
Big thanks to Leon Lang, Jérémy Scheurer, Adam Gleave, and Shoshannah Tekofsky for their feedback on a draft of this post, to Euan McLean (via FAR AI) for his feedback and a lot of help with editing, and to everyone else who discussed this agenda with me, in particular Johannes Treutlein for frequent research check-ins!
Summary
My current agenda is to develop a formal framework for thinking about abstractions of computations. These abstractions are ways to partially describe the “algorithm” a neural network or other computation is using, while throwing away irrelevant details.
Ideally, this framework would tell us 1) all possible abstractions of a given computation, an…
playlist_add
Feb 1, 2023
AF - More findings on Memorization and double descent by Marius Hobbhahn
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More findings on Memorization and double descent, published by Marius Hobbhahn on February 1, 2023 on The AI Alignment Forum.
Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.
I’d like to thank Wes Gurnee, Aryan Bhatt, Eric Purdy and Stefan Heimersheim for discussions and Evan Hubinger, Neel Nanda, Adam Jermyn and Chris Olah for mentorship and feedback.
The post contains a lot of figures, so the suggested length is deceiving. Code can be found in these three colab notebooks [1][2][3].
I have split the post into two parts. The first one is concerned with double descent and other general findings in memorization and the second focuses on measuring memorization using the maximum data dimensionality metric. This is the first post in a series of N posts on memorization…
playlist_add
Feb 1, 2023
AF - Trends in the dollar training cost of machine learning systems by Ben Cottier
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Trends in the dollar training cost of machine learning systems, published by Ben Cottier on February 1, 2023 on The AI Alignment Forum.
Summary
Using a dataset of 124 machine learning (ML) systems published between 2009 and 2022, I estimate that the cost of compute in US dollars for the final training run of ML systems has grown by 0.49 orders of magnitude (OOM) per year (90% CI: 0.37 to 0.56). See Table 1 for more detailed results, indicated by "All systems."
By contrast, I estimate that the cost of compute used to train "large-scale" systems since September 2015 (systems that used a relatively large amount of compute) has grown more slowly compared to the full sample, at a rate of 0.2 OOMs/year (90% CI: 0.1 to 0.4 OOMs/year). See Table 1 for more detailed results, indicated by "Large-scale."
Based on…
playlist_add
Jan 31, 2023
AF - The effect of horizon length on scaling laws by Jacob Hilton
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The effect of horizon length on scaling laws, published by Jacob Hilton on February 1, 2023 on The AI Alignment Forum.
The scaling of optimal model size with compute is a key input into the biological anchors framework for forecasting transformative AI. In particular, the "effective horizon length" introduces a multiplier into this scaling law that can have a big effect on forecasts.
This paper studies this scaling law for several RL environments: Procgen, Dota 2 and a toy MNIST-based environment. The last of these is used to study the effect of the task horizon length in a toy setting. There are a number of takeaways for the biological anchors framework, which are summarized in Section 5.4.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
playlist_add
Jan 31, 2023
AF - On value in humans, other animals, and AI by Michele Campolo
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On value in humans, other animals, and AI, published by Michele Campolo on January 31, 2023 on The AI Alignment Forum.
This will be posted also on the EA Forum, and included in a sequence containing some previous posts and other posts I'll publish this year.
Introduction
Humans think critically about values and, to a certain extent, they also act according to their values. To the average human, the difference between increasing world happiness and increasing world suffering is huge and evident, while goals such as collecting coins and collecting stamps are roughly on the same level.
It would be nice to make these differences obvious to AI as they are to us. Even though exactly copying what happens in the human mind is probably not the best strategy to design an AI that understands ethics, having an idea…
playlist_add
Jan 31, 2023
AF - Criticism of the main framework in AI alignment by Michele Campolo
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Criticism of the main framework in AI alignment, published by Michele Campolo on January 31, 2023 on The AI Alignment Forum.
Originally posted on the EA Forum for the Criticism and Red Teaming Contest. Will be included in a sequence containing some previous posts and other posts I'll publish this year.
0. Summary
AI alignment research centred around the control problem works well for futures shaped by out-of-control misaligned AI, but not that well for futures shaped by bad actors using AI. Section 1 contains a step-by-step argument for that claim. In section 2 I propose an alternative which aims at moral progress instead of direct risk reduction, and I reply to some objections. I will give technical details about the alternative at some point in the future, in section 3.
The appendix clarifies some min…
playlist_add
Jan 31, 2023
AF - Mechanistic Interpretability Quickstart Guide by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mechanistic Interpretability Quickstart Guide, published by Neel Nanda on January 31, 2023 on The AI Alignment Forum.
This was written as a guide for Apart Research's Mechanistic Interpretability Hackathon as a compressed version of my getting started post. The spirit is "how to speedrun your way to maybe doing something useful in mechanistic interpretability in a weekend", but hopefully this is useful even to people who aren't aiming for weekend long projects!
Quickstart
Watch my “What is a Transformer?” video
Skim through my TransformerLens demo
Copy it to a new Colab notebook (with a free GPU) to actually write your own code - do not get involved in tech setup!
Skim the Concrete Open Problems section, or my 200 Concrete Open Problems in Mech Interp sequence. Find a problem that catches your fancy…
playlist_add
Jan 31, 2023
AF - Inner Misalignment in "Simulator" LLMs by Adam Scherlis
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inner Misalignment in "Simulator" LLMs, published by Adam Scherlis on January 31, 2023 on The AI Alignment Forum.
Alternate title: "Somewhat Contra Scott On Simulators".
Scott Alexander has a recent post up on large language models as simulators.
I generally agree with Part I of the post, which advocates thinking about LLMs as simulators that can emulate a variety of language-producing "characters" (with imperfect accuracy). And I also agree with Part II, which applies this model to RLHF'd models whose "character" is a friendly chatbot assistant.
(But see caveats about the simulator framing from Beth Barnes here.)
These ideas have been around for a bit, and Scott gives credit where it's due; I think his exposition is clear and fun.
In Part III, where he discusses alignment implications, I think he misse…
playlist_add
Jan 30, 2023
AF - Why I hate the "accident vs. misuse" AI x-risk dichotomy (quick thoughts on "structural risk") by David Scott Krueger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I hate the "accident vs. misuse" AI x-risk dichotomy (quick thoughts on "structural risk"), published by David Scott Krueger on January 30, 2023 on The AI Alignment Forum.
I think the large majority of AI x-risk is "structural". Like climate change.
Here's a good primer on structural risk (note that structural risk is not a synonym for "not caused by out-of-control AI"):
I am shocked and amazed and dismayed that more people do not seem to view it this way, even among the AI x-safety community. Heck, even Eliezer's stories of doom are steeped in structural risk (race dynamics, teams rationalizing cutting corners on safety when they should know better, etc.)
I expect irresponsible, reckless, negligent deployment of AI systems without proper accounting of externalities. I consider this the default for…
playlist_add
Jan 30, 2023
AF - Call for submissions: “(In)human Values and Artificial Agency”, ALIFE 2023 by thegearstoascension
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Call for submissions: “(In)human Values and Artificial Agency”, ALIFE 2023, published by thegearstoascension on January 30, 2023 on The AI Alignment Forum.
key points:
Cash prize of $500 for the best presentation.
Deadline 3 March, 2023.
Organized by Simon McGregor (University of Sussex), Rory Greig (DeepMind), Chris Buckley (University of Sussex)
ALIFE 2023 (the 2023 conference on Artificial Life) will feature a Special Session on “(In)human Values and Artificial Agency”. This session focuses on issues at the intersection of AI Safety and Artificial Life. We invite the submission of research papers, or extended abstracts, that deal with related topics.
We particularly encourage submissions from researchers in the AI Safety community, who might not otherwise have considered submitting to ALIFE 2…
playlist_add
Jan 30, 2023
AF - What I mean by "alignment is in large part about making cognition aimable at all" by Nate Soares
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What I mean by "alignment is in large part about making cognition aimable at all", published by Nate Soares on January 30, 2023 on The AI Alignment Forum.
(Epistemic status: attempting to clear up a misunderstanding about points I have attempted to make in the past. This post is not intended as an argument for those points.)
I have long said that the lion's share of the AI alignment problem seems to me to be about pointing powerful cognition at anything at all, rather than figuring out what to point it at.
It’s recently come to my attention that some people have misunderstood this point, so I’ll attempt to clarify here.
In saying the above, I do not mean the following:
(1) Any practical AI that you're dealing with will necessarily be cleanly internally organized around pursuing a single objective. M…
playlist_add
Jan 29, 2023
AF - Structure, creativity, and novelty by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Structure, creativity, and novelty, published by Tsvi Benson-Tilsen on January 29, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. First completed 26 June 2022. I'm likely to not respond to comments promptly.]
A high-level confusion that I have that seems to be on the way towards understanding alignment, is the relationship between values and understanding. This essay gestures at the idea of structure in general (mainly by listing examples).
Why do we want AGI at all?
We want AGI in order to understand stuff that we haven't yet understood.
(This is not a trivial claim. It might be false. It could be that to secure the future of humane existence, something other than understanding is necessary or sufficient; e.g. it's conceivable that solving some large combinatorial problem, akin to playing…
playlist_add
Jan 27, 2023
AF - Spooky action at a distance in the loss landscape by Jesse Hoogland
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Spooky action at a distance in the loss landscape, published by Jesse Hoogland on January 28, 2023 on The AI Alignment Forum.
Produced as part of the SERI ML Alignment Theory Scholars Program Winter 2022 Cohort.
Not all global minima of the (training) loss landscape are created equal.
Even if they achieve equal performance on the training set, different solutions can perform very differently on the test set or out-of-distribution. So why is it that we typically find "simple" solutions that generalize well?
In a previous post, I argued that the answer is "singularities" — minimum loss points with ill-defined tangents. It's the "nastiest" singularities that have the most outsized effect on learning and generalization in the limit of large data. These act as implicit regularizers that lower the effective…
playlist_add
Jan 27, 2023
AF - The role of Bayesian ML in AI safety - an overview by Marius Hobbhahn
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The role of Bayesian ML in AI safety - an overview, published by Marius Hobbhahn on January 27, 2023 on The AI Alignment Forum.
I want to thank Sebastian Farquhar, Laurence Midgley and Johan van den Heuvel, for feedback and discussion on this post.
Some time ago I asked the question “What is the role of Bayesian ML in AI safety/alignment?”. The response of the EA and Bayesian ML community was very helpful. Thus, I decided to collect and distill the answers and provide more context for current and future AI safety researchers.
Clarification: I don’t think many people (<1% of the alignment community) should work on Bayesian ML or that it is even the most promising path to alignment. I just want to provide a perspective and give an overview. I personally am not that bullish on Bayesian ML anymore (se…
playlist_add
Jan 25, 2023
AF - AGI will have learnt utility functions by Beren Millidge
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI will have learnt utility functions, published by Beren Millidge on January 25, 2023 on The AI Alignment Forum.
This post is part of the work done at Conjecture.
Thanks to Eric Winsor, Daniel Braun, Andrea Miotti and Connor Leahy for helpful comments and feedback on the draft versions of this post.
There has been a lot of debate and discussion recently in the AI safety community about whether AGI will likely optimize for fixed goals or be a wrapper mind. The term wrapper mind is largely a restatement of the old idea of a utility maximizer, with AIXI as a canonical example. The fundamental idea is that there is an agent with some fixed utility function which it maximizes without any kind of feedback which can change its utility function.
Rather, the optimization process is assumed to be 'wrapped aroun…
playlist_add
Jan 25, 2023
AF - AGI will have learnt utility functions by Beren Millidge
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI will have learnt utility functions, published by Beren Millidge on January 25, 2023 on The AI Alignment Forum.
This post is part of the work done at Conjecture.
Thanks to Eric Winsor, Daniel Braun, Andrea Miotti and Connor Leahy for helpful comments and feedback on the draft versions of this post.
There has been a lot of debate and discussion recently in the AI safety community about whether AGI will likely optimize for fixed goals or be a wrapper mind. The term wrapper mind is largely a restatement of the old idea of a utility maximizer, with AIXI as a canonical example. The fundamental idea is that there is an agent with some fixed utility function which it maximizes without any kind of feedback which can change its utility function.
Rather, the optimization process is assumed to be 'wrapped aroun…
playlist_add
Jan 25, 2023
AF - Thoughts on the impact of RLHF research by Paul Christiano
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on the impact of RLHF research, published by Paul Christiano on January 25, 2023 on The AI Alignment Forum.
In this post I’m going to describe my basic justification for working on RLHF in 2017-2020, which I still stand behind. I’ll discuss various arguments that RLHF research had an overall negative impact and explain why I don’t find them persuasive.
I'll also clarify that I don't think research on RLHF is automatically net positive; alignment research should address real alignment problems, and we should reject a vague association between "RLHF progress" and "alignment progress."
Background on my involvement in RLHF work
Here are some background views about alignment I held in 2015 and still hold today. I expect disagreements about RLHF will come down to disagreements about this backgr…
playlist_add
Jan 25, 2023
AF - Quick thoughts on "scalable oversight" / "super-human feedback" research by David Scott Krueger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick thoughts on "scalable oversight" / "super-human feedback" research, published by David Scott Krueger on January 25, 2023 on The AI Alignment Forum.
The current default view seems to roughly be:
Inner alignment is more important than outer alignment (or, alternatively, this distinction is bad/sub-optimal, but basically it's all about generalizing correctly)
Scalable oversight is the only useful form of outer alignment research remaining.
We don't need to worry about sample efficiency in RLHP -- in the limit we just pay everyone to provide feedback, and in practice even a few thousand samples (or a "constition") seems ~good enough.
But maybe it's not good? Because it's more like capabilities research?
A common example used for motivating scalable oversight is the "AI CEO".
My views are:
We should no…
playlist_add
Jan 25, 2023
AF - Quick thoughts on "scalable oversight" / "super-human feedback" research by David Scott Krueger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick thoughts on "scalable oversight" / "super-human feedback" research, published by David Scott Krueger on January 25, 2023 on The AI Alignment Forum.
The current default view seems to roughly be:
Inner alignment is more important than outer alignment (or, alternatively, this distinction is bad/sub-optimal, but basically it's all about generalizing correctly)
Scalable oversight is the only useful form of outer alignment research remaining.
We don't need to worry about sample efficiency in RLHP -- in the limit we just pay everyone to provide feedback, and in practice even a few thousand samples (or a "constition") seems ~good enough.
But maybe it's not good? Because it's more like capabilities research?
A common example used for motivating scalable oversight is the "AI CEO".
My views are:
We should no…
playlist_add
Jan 24, 2023
AF - Alexander and Yudkowsky on AGI goals by Scott Alexander
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alexander and Yudkowsky on AGI goals, published by Scott Alexander on January 24, 2023 on The AI Alignment Forum.
This is a lightly edited transcript of a chatroom conversation between Scott Alexander and Eliezer Yudkowsky last year, following up on the Late 2021 MIRI Conversations. Questions discussed include "How hard is it to get the right goals into AGI systems?" and "In what contexts do AI systems exhibit 'consequentialism'?".
1. Analogies to human moral development
[Yudkowsky][13:29]
@ScottAlexander ready when you are
[Alexander][13:31]
Okay, how do you want to do this?
[Yudkowsky][13:32]
If you have an agenda of Things To Ask, you can follow it; otherwise I can start by posing a probing question or you can?
We've been very much winging it on these and that has worked... as well as you have seen i…
playlist_add
Jan 24, 2023
AF - Alexander and Yudkowsky on AGI goals by Scott Alexander
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alexander and Yudkowsky on AGI goals, published by Scott Alexander on January 24, 2023 on The AI Alignment Forum.
This is a lightly edited transcript of a chatroom conversation between Scott Alexander and Eliezer Yudkowsky last year, following up on the Late 2021 MIRI Conversations. Questions discussed include "How hard is it to get the right goals into AGI systems?" and "In what contexts do AI systems exhibit 'consequentialism'?".
1. Analogies to human moral development
[Yudkowsky][13:29]
@ScottAlexander ready when you are
[Alexander][13:31]
Okay, how do you want to do this?
[Yudkowsky][13:32]
If you have an agenda of Things To Ask, you can follow it; otherwise I can start by posing a probing question or you can?
We've been very much winging it on these and that has worked... as well as you have seen i…
playlist_add
Jan 24, 2023
AF - Inverse Scaling Prize: Second Round Winners by Ian McKenzie
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inverse Scaling Prize: Second Round Winners, published by Ian McKenzie on January 24, 2023 on The AI Alignment Forum.
At the end of the second and final round of the Inverse Scaling Prize, we’re awarding 7 more Third Prizes. The Prize aimed to identify important tasks on which language models (LMs) perform worse the larger they are (“inverse scaling”). Inverse scaling may reveal cases where LM training actively encourages behaviors that are misaligned with human preferences. The contest started on June 27th and concluded on October 27th, 2022 – thanks to everyone who participated! Across the two rounds, we had over 80 unique submissions and gave out a total of 11 Third Prizes.
We are also accepting updates to two previous prize-winners (quote-repetition and redefine-math). For more details on th…
playlist_add
Jan 24, 2023
AF - Inverse Scaling Prize: Second Round Winners by Ian McKenzie
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inverse Scaling Prize: Second Round Winners, published by Ian McKenzie on January 24, 2023 on The AI Alignment Forum.
At the end of the second and final round of the Inverse Scaling Prize, we’re awarding 7 more Third Prizes. The Prize aimed to identify important tasks on which language models (LMs) perform worse the larger they are (“inverse scaling”). Inverse scaling may reveal cases where LM training actively encourages behaviors that are misaligned with human preferences. The contest started on June 27th and concluded on October 27th, 2022 – thanks to everyone who participated! Across the two rounds, we had over 80 unique submissions and gave out a total of 11 Third Prizes.
We are also accepting updates to two previous prize-winners (quote-repetition and redefine-math). For more details on th…
playlist_add
Jan 24, 2023
AF - “Endgame safety” for AGI by Steve Byrnes
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: “Endgame safety” for AGI, published by Steve Byrnes on January 24, 2023 on The AI Alignment Forum.
(Status: no pretense to originality, but a couple people said they found this terminology useful, so I’m sharing it more widely.)
There’s a category of AGI safety work that we might call “Endgame Safety”, where we’re trying to do all the AGI safety work that we couldn’t or didn’t do ahead of time, in the very last moments before (or even after) people are actually playing around with powerful AGI algorithms of the type that could get irreversibly out of control and cause catastrophe.
I think everyone agrees that Endgame Safety is important and unavoidable. If nothing else, for every last line of AGI source code, we can do an analysis of what happens if that line of code has a bug, or if a…
playlist_add
Jan 24, 2023
AF - “Endgame safety” for AGI by Steve Byrnes
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: “Endgame safety” for AGI, published by Steve Byrnes on January 24, 2023 on The AI Alignment Forum.
(Status: no pretense to originality, but a couple people said they found this terminology useful, so I’m sharing it more widely.)
There’s a category of AGI safety work that we might call “Endgame Safety”, where we’re trying to do all the AGI safety work that we couldn’t or didn’t do ahead of time, in the very last moments before (or even after) people are actually playing around with powerful AGI algorithms of the type that could get irreversibly out of control and cause catastrophe.
I think everyone agrees that Endgame Safety is important and unavoidable. If nothing else, for every last line of AGI source code, we can do an analysis of what happens if that line of code has a bug, or if a…
playlist_add
Jan 24, 2023
AF - Thoughts on hardware / compute requirements for AGI by Steve Byrnes
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on hardware / compute requirements for AGI, published by Steve Byrnes on January 24, 2023 on The AI Alignment Forum.
Let’s say I know how to build / train a human-level (more specifically, John von Neumann level) AGI. And let’s say that we (and/or the AGI itself) have already spent a few years on making the algorithm work better and more efficiently.
Question: How much compute will it take to run this AGI?
(NB: I said "running" an AGI, not training / programming an AGI. I'll talk a bit about “training compute” at the very end.)
Answer: I don’t know. But that doesn’t seem to be stopping me from writing this post. ¯\_(ツ)_/¯ My current feeling—which I can easily imagine changing after discussion (which is a major reason I'm writing this!)—seems to be:
75%: One current (Jan 2023…
playlist_add
Jan 24, 2023
AF - Thoughts on hardware / compute requirements for AGI by Steve Byrnes
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on hardware / compute requirements for AGI, published by Steve Byrnes on January 24, 2023 on The AI Alignment Forum.
Let’s say I know how to build / train a human-level (more specifically, John von Neumann level) AGI. And let’s say that we (and/or the AGI itself) have already spent a few years on making the algorithm work better and more efficiently.
Question: How much compute will it take to run this AGI?
(NB: I said "running" an AGI, not training / programming an AGI. I'll talk a bit about “training compute” at the very end.)
Answer: I don’t know. But that doesn’t seem to be stopping me from writing this post. ¯\_(ツ)_/¯ My current feeling—which I can easily imagine changing after discussion (which is a major reason I'm writing this!)—seems to be:
75%: One current (Jan 2023…
playlist_add
Jan 23, 2023
AF - Some of my disagreements with List of Lethalities by Alex Turner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some of my disagreements with List of Lethalities, published by Alex Turner on January 24, 2023 on The AI Alignment Forum.
This was an appendix of Inner and outer alignment decompose one hard problem into two extremely hard problems. However, I think the material is self-contained and worth sharing separately, especially since AGI Ruin: A List of Lethalities has become so influential. (I agree with most of the points made in AGI Ruin, but I'm going to focus on disagreements in this essay.)
Here are some quotes with which I disagree, in light of points I made in Inner and outer alignment decompose one hard problem into two extremely hard problems (consult its TL;DR and detailed summary for a refresher, if need be).
List of Lethalities
“Humans don't explicitly pursue inclusive genetic fitness; outer opt…
playlist_add
Jan 23, 2023
AF - Some of my disagreements with List of Lethalities by Alex Turner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some of my disagreements with List of Lethalities, published by Alex Turner on January 24, 2023 on The AI Alignment Forum.
This was an appendix of Inner and outer alignment decompose one hard problem into two extremely hard problems. However, I think the material is self-contained and worth sharing separately, especially since AGI Ruin: A List of Lethalities has become so influential. (I agree with most of the points made in AGI Ruin, but I'm going to focus on disagreements in this essay.)
Here are some quotes with which I disagree, in light of points I made in Inner and outer alignment decompose one hard problem into two extremely hard problems (consult its TL;DR and detailed summary for a refresher, if need be).
List of Lethalities
“Humans don't explicitly pursue inclusive genetic fitness; outer opt…
playlist_add
Jan 22, 2023
AF - What a compute-centric framework says about AI takeoff speeds - draft report by Tom Davidson
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What a compute-centric framework says about AI takeoff speeds - draft report, published by Tom Davidson on January 23, 2023 on The AI Alignment Forum.
As part of my work for Open Philanthropy I’ve written a draft report on AI takeoff speeds, the question of how quickly AI capabilities might improve as we approach and surpass human-level AI. Will human-level AI be a bolt from the blue, or will we have AI that is nearly as capable many years earlier?
Most of the analysis is from the perspective of a compute-centric framework, inspired by that used in the Bio Anchors report, in which AI capabilities increase continuously with more training compute and work to develop better AI algorithms.
This post doesn’t summarise the report. Instead I want to explain some of the high-level takeaways from the researc…
playlist_add
Jan 22, 2023
AF - What a compute-centric framework says about AI takeoff speeds - draft report by Tom Davidson
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What a compute-centric framework says about AI takeoff speeds - draft report, published by Tom Davidson on January 23, 2023 on The AI Alignment Forum.
As part of my work for Open Philanthropy I’ve written a draft report on AI takeoff speeds, the question of how quickly AI capabilities might improve as we approach and surpass human-level AI. Will human-level AI be a bolt from the blue, or will we have AI that is nearly as capable many years earlier?
Most of the analysis is from the perspective of a compute-centric framework, inspired by that used in the Bio Anchors report, in which AI capabilities increase continuously with more training compute and work to develop better AI algorithms.
This post doesn’t summarise the report. Instead I want to explain some of the high-level takeaways from the researc…
playlist_add
Jan 22, 2023
AF - Gemini modeling by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gemini modeling, published by Tsvi Benson-Tilsen on January 22, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. Written 17 June 2022. I'm likely to not respond to comments promptly.]
A gemini model is a kind of model that's especially relevant for minds modeling minds.
Two scenarios
You stand before a tree. How big is it? How does it grow? What can it be used to make? Where will it fall if you cut it here or there?
Alice usually eats Cheerios in the morning. Today she comes downstairs, but doesn't get out a bowl and spoon and milk, and doesn't go over to the Cheerios cupboard. Then she sees green bananas on the counter. Then she goes and gets a bowl and spoon and milk, and gets a box of Cheerios from the cupboard. What happened?
We have some kind of mental model of the tree, and some kind o…
playlist_add
Jan 20, 2023
AF - Critique of some recent philosophy of LLMs’ minds by Roman Leventov
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Critique of some recent philosophy of LLMs’ minds, published by Roman Leventov on January 20, 2023 on The AI Alignment Forum.
I structure this post as a critique of some recent papers on the philosophy of mind in application to LLMs, concretely, on whether we can say that LLMs think, reason, understand language, refer to the real world when producing language, have goals and intents, etc. I also use this discussion as a springboard to express some of my views about the ontology of intelligence, agency, and alignment.
Mahowald, Ivanova, et al., “Dissociating language and thought in large language models: a cognitive perspective” (Jan 2023). Note that this is a broad review paper, synthesising findings from computational linguistics, cognitive science, and neuroscience, as well as offering an engine…
playlist_add
Jan 20, 2023
AF - Shard theory alignment requires magic. by Charlie Steiner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shard theory alignment requires magic., published by Charlie Steiner on January 20, 2023 on The AI Alignment Forum.
A delayed hot take. This is pretty similar to previous comments from Rohin.
"Magic," of course, in the technical sense of stuff we need to remind ourselves we don't know how to do. I don't mean this pejoratively, locating magic is an important step in trying to demystify it.
And "shard theory alignment" in the sense of building an AI that does good things and not bad things by encouraging an RL agent to want to do good things, via kinds of reward shaping analogous to the diamond maximizer example.
How might the story go?
You start out with some unsupervised model of sensory data.
On top of its representation of the world you start training an RL agent, with a carefully chosen curriculum an…
playlist_add
Jan 19, 2023
AF - Thoughts on refusing harmful requests to large language models by William Saunders
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on refusing harmful requests to large language models, published by William Saunders on January 19, 2023 on The AI Alignment Forum.
Currently, large language models (ChatGPT, Constitutional AI) are trained to refuse to follow user requests that are considered inappropriate or harmful. This can be done by training on example strings of the form “User: inappropriate request AI: elaborate apology”
Proposal
Instead of training a language model to produce “elaborate apology” when it refuses to do an action, train it to produce a special sequence or token first “elaborate apology”. Strip the special sequence out before returning a response to the user (and never allow the user to include the special sequence in input).
Benefits
Can directly measure the probability of refusal for any outpu…
playlist_add
Jan 18, 2023
AF - 200 COP in MI: Studying Learned Features in Language Models by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Studying Learned Features in Language Models, published by Neel Nanda on January 19, 2023 on The AI Alignment Forum.
This is the final post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. Look up jargon in my Mechanistic Interpretability Explainer
Motivation
Motivating Paper: Softmax Linear Units (SoLU), Multimodal Neurons
To accompany this post, I’ve created a website called Neuroscope that displays the text that most activates each neuron in some language models - check it out!
This section contains a lot of details and thoughts on thinking about neurons and learned features and relationship to the surrounding…
playlist_add
Jan 18, 2023
AF - Gradient Filtering by Arun Jose
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gradient Filtering, published by Arun Jose on January 18, 2023 on The AI Alignment Forum.
Several of the ideas in this post originated in conversations between the authors months ago. I did the work involved in exploring them and thinking of new ideas and framings, as well as writing this post; old writings by Janus were the inspiration for some of the directions of thought here.
Thanks to Adam Jermyn and Paul Colognese for their feedback and thoughts.
Generally speaking, gradient hacking can seem pretty hard for an agent to do. But the dangers don’t lie in the galaxy-brained strategies that models with strong deceptive capabilities can implement, it’s in the idea that a model’s gradient can be influenced by an inner agent at all. So examining variants that are seemingly more accessible would be p…
playlist_add
Jan 17, 2023
AF - Löbian emotional processing of emergent cooperation: an example by Andrew Critch
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Löbian emotional processing of emergent cooperation: an example, published by Andrew Critch on January 17, 2023 on The AI Alignment Forum.
Related to: A Löbian argument pattern for implicit reasoning in natural language: Löbian party invitations
Epistemic status: my opinion based on a mix of math, reflection, and speculation; not backed up by any systematic psychological studies.
Summary: Since my 2019 paper generalizing Löb's Theorem, a couple dozen people have asked me if the way humans naturally cooperate might be well-described by Löb's Theorem. In short, my answer is probably sometimes, and in this post I'll try using an example to convey what that means. Importantly, Löb's Theorem is not a theorem when applied to real-world humans and emotions — i.e., when its hypotheses are met, its concl…
playlist_add
Jan 16, 2023
AF - Preparing for AI-assisted alignment research: we need data! by CBiddulph
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Preparing for AI-assisted alignment research: we need data!, published by CBiddulph on January 17, 2023 on The AI Alignment Forum.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
playlist_add
Jan 16, 2023
AF - Experiment Idea: RL Agents Evading Learned Shutdownability by Leon Lang
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Experiment Idea: RL Agents Evading Learned Shutdownability, published by Leon Lang on January 16, 2023 on The AI Alignment Forum.
Preface
Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort. Thanks to Erik Jenner who explained to me the basic intuition for why an advanced RL agent may evade the discussed corrigibility measure. I also thank Alex Turner, Magdalena Wache, and Walter Laurito for detailed feedback on the proposal and Quintin Pope and Lisa Thiergart for helpful feedback in the last December SERI-MATS shard theory group meeting.
This text was part of my deliverable for the shard theory stream of SERI-MATS. In it, I present an idea for an experiment that tests the convergent drive of modern model-based RL agents to evade shutdownability. If successful, I expec…
playlist_add
Jan 15, 2023
AF - Understanding the diffusion of large language models: summary by Ben Cottier
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding the diffusion of large language models: summary, published by Ben Cottier on January 16, 2023 on The AI Alignment Forum.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
playlist_add
Jan 15, 2023
AF - Speculation on Path-Dependance in Large Language Models. by NickyP
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Speculation on Path-Dependance in Large Language Models., published by NickyP on January 15, 2023 on The AI Alignment Forum.
Epistemic Status: Highly Speculative. I spent less than a day thinking about this in particular, and though I have spent a few months studying large language models, I have never trained a language model. I am likely wrong about many things. I have not seen research on this, so it may be useful for for someone to do a real deep dive.
Thanks to Anthony from the Center on Long Term Risk for sparking the discussion earlier today for this post. Also thanks to conversations with Evan Hubinger ~1 year ago that got me thinking about the topic previously.
Summary
My vague suspicions at the moment are somewhat along the lines of:
Training an initial model: moderate to low path-dependance
R…
playlist_add
Jan 15, 2023
AF - Underspecification of Oracle AI by Rubi J. Hudson
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Underspecification of Oracle AI, published by Rubi J. Hudson on January 15, 2023 on The AI Alignment Forum.
Rubi and Johannes worked on this post as part of the SERI MATS program, with Evan Hubinger providing mentorship to both. Rubi also received mentorship from Leo Gao. Thanks to Paul Colognese and Nicholas Schiefer for discussions related to this post.
An oracle is a type of AI system that only answers questions without taking any other actions in the world. Simulators and generative models, which have seen increased discussion recently (links: 1, 2, 3, 4), can be thought of as types of oracles. Such systems may simultaneously be powerful enough to generate a pivotal act while also being easier to align due to a more limited domain.
One major concern with oracles is that the answers they give can sti…
playlist_add
Jan 15, 2023
AF - Non-directed conceptual founding by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Non-directed conceptual founding, published by Tsvi Benson-Tilsen on January 15, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. Written 13 June 2022. I'm fairly likely to not respond to comments promptly. If you're especially interested in chatting, my gmail address is: tsvibtcontact ]
In trying to understand minds-in-general, we sometimes ask questions that talk about "big" things (taking "big" to ambiguously mean any of large, complex, abstract, vague, important, touches many things, applies to many contexts, "high-level"). E.g.:
What is it for a mind to have thoughts or to care about stuff? How does care and thought relate?
What is it to believe a proposition?
Why do agents use abstractions?
These "big" things such as thought, caring, propositions, beliefs, agents, abstractions, and so…
playlist_add
Jan 14, 2023
AF - World-Model Interpretability Is All We Need by Thane Ruthenis
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: World-Model Interpretability Is All We Need, published by Thane Ruthenis on January 14, 2023 on The AI Alignment Forum.
Summary, by sections:
Perfect world-model interpretability seems both sufficient for robust alignment (via a decent variety of approaches) and realistically attainable (compared to "perfect interpretability" in general, i. e. insight into AIs' heuristics, goals, and thoughts as well). Main arguments: the NAH + internal interfaces.
There's plenty of reasons to think that world-models would converge towards satisfying a lot of nice desiderata: they'd be represented as a separate module in AI cognitive architecture, and that module would consists of many consistently-formatted sub-modules representing recognizable-to-us concepts. Said "consistent formatting" may allow us to, in a certain…
playlist_add
Jan 13, 2023
AF - Concrete Reasons for Hope about AI by Zac Hatfield-Dodds
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concrete Reasons for Hope about AI, published by Zac Hatfield-Dodds on January 14, 2023 on The AI Alignment Forum.
Recent advances in machine learning—in reinforcement learning, language modeling, image and video generation, translation and transcription models, etc.—without similarly striking safety results, have rather dampened the mood in many AI Safety circles. If I was any less concerned by extinction risks from AI, I would have finished my PhD[1] as planned before moving from Australia to SF to work at Anthropic; I believe that the situation is both urgent and important.[2]
On the other hand, despair is neither instrumentally nor terminally valuable.[3] This essay therefore lays out some concrete reasons for hope, which might help rebalance the emotional scales and offer some directions to mov…
playlist_add
Jan 13, 2023
AF - [ASoT] Simulators show us behavioural properties by default by Arun Jose
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [ASoT] Simulators show us behavioural properties by default, published by Arun Jose on January 13, 2023 on The AI Alignment Forum.
I’m trying out making a few posts with less polish and smaller scope, to iterate more quickly on my thoughts and write about some interesting ideas in isolation before having fully figured them out. Expect middling confidence in any conclusions drawn, and occasionally just chains of reasoning without fully contextualized conclusions.
Let’s say we prompt a language model to do chain-of-thought reasoning while answering some question. What are we actually seeing? One answer I’ve seen people use, often implicitly, is along the lines that we’re seeing the underlying reasoning being done by the model to get at the answer.
I think this take is wrong, and in this post I’l…
playlist_add
Jan 13, 2023
AF - Tracr: Compiled Transformers as a Laboratory for Interpretability | DeepMind by Cinera Verinia
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tracr: Compiled Transformers as a Laboratory for Interpretability | DeepMind, published by Cinera Verinia on January 13, 2023 on The AI Alignment Forum.
Abstract
Interpretability research aims to build tools for understanding machine learning (ML) models. However, such tools are inherently hard to evaluate because we do not have ground truth information about how ML models actually work. In this work, we propose to build transformer models manually as a testbed for interpretability research. We introduce Tracr, a "compiler" for translating human-readable programs into weights of a transformer model. Tracr takes code written in RASP, a domain-specific language (Weiss et al. 2021), and translates it into weights for a standard, decoder-only, GPT-like transformer architecture. We use Tracr to create a rang…
playlist_add
Jan 13, 2023
AF - Some Arguments Against Strong Scaling by Joar Skalse
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some Arguments Against Strong Scaling, published by Joar Skalse on January 13, 2023 on The AI Alignment Forum.
There are many people who believe that we will be able to get to AGI by basically just scaling up the techniques used in recent large language models, combined with some relatively minor additions and/or architectural changes. As a result, there are people in the AI safety community who now predict timelines of less than 10 years, and structure their research accordingly. However, there are also people who still believe in long(er) timelines, or at least that substantial new insights or breakthroughts will be needed for AGI (even if those breakthroughts in principle could happen quickly). My impression is that the arguments for the latter position are not all that widely known in the AI safety…
playlist_add
Jan 12, 2023
AF - Disentangling Shard Theory into Atomic Claims by Leon Lang
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Disentangling Shard Theory into Atomic Claims, published by Leon Lang on January 13, 2023 on The AI Alignment Forum.
Introduction
Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort. Thanks to Magdalena Wache for giving feedback on a recent version, and to Alex Turner for giving feedback on an early version of this article.
When thinking about shard theory, I noticed that my brain wanted to answer questions such as "What does shard theory predict?" or "Do I agree with shard theory?". Additionally, I observed other people engage in similar thinking. I now think this is confused, and that it is better to view shard theory as a bag of related claims that should be reasoned about and evaluated separately.
Therefore, In this post I want to explain the following:
What do I c…
playlist_add
Jan 12, 2023
AF - AGISF adaptation for in-person groups by Sam Marks
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGISF adaptation for in-person groups, published by Sam Marks on January 13, 2023 on The AI Alignment Forum.
This past semester, HAIST and MAIA (the Harvard and MIT AI safety student groups) ran an adapted version of Richard Ngo's AGI Safety Fundamentals alignment curriculum. This adaptation – which consists of eight 2-hour long meetings, with all readings done during the meeting – is now available on the AGISF website.
In this post, we discuss the adapted curriculum and its intended use, and we recommend that other in-person reading groups following AGISF use this adaptation.
The adapted curriculum and its intended use
The adapted curriculum was made by refining a slightly rustier first adaptation, with significant help from Richard Ngo and feedback from participants. The key differences between th…
playlist_add
Jan 12, 2023
AF - Announcing the 2023 PIBBSS Summer Research Fellowship by Nora Ammann
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the 2023 PIBBSS Summer Research Fellowship, published by Nora Ammann on January 12, 2023 on The AI Alignment Forum.
What can we learn from biological and social systems about designing beneficial AI systems?
We are, once again, pleased to announce the PIBBSS Research Fellowship program.
Principles of Intelligent Behavior in Biological and Social Systems (PIBBSS) aims to facilitate research on parallels between intelligent behaviour in natural and artificial systems and to leverage these insights towards the goal of building safe and aligned AI.
Last year, we welcomed 20 fellows from fields as diverse as computational neuroscience, evolutionary biology, complex systems studies and the humanities. Over the course of three months, supported by two research retreats in Oxford and Prague, shared o…
playlist_add
Jan 12, 2023
AF - ChatGPT struggles to respond to the real world by Alex Flint
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ChatGPT struggles to respond to the real world, published by Alex Flint on January 12, 2023 on The AI Alignment Forum.
I wanted to test whether ChatGPT could respond to situations as they arise in the real world, so I asked it to guide me through making a turmeric latte. It had no problem giving me a nicely formatted recipe but it really struggled to guide me through the process of making the latte.
The rules I set for this experiment was that I would perform any instruction that I could perform in less than 10 seconds, so "fetch a saucepan" is fine, "go to the fridge" is fine, "put one cup of milk into the saucepan" is fine if I have the milk and the saucepan already in front of me, but if the milk or saucepan were not in front of me I would respond with something like "I don’t have the milk in front…
playlist_add
Jan 10, 2023
AF - 200 COP in MI: Interpreting Reinforcement Learning by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Interpreting Reinforcement Learning, published by Neel Nanda on January 10, 2023 on The AI Alignment Forum.
This is the ninth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. Look up jargon in my Mechanistic Interpretability Explainer
I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
Motivating papers: Acquisition of Chess Knowledge in AlphaZero, Understanding RL Vision
Disclaimer: My area of expertise is language model interpretability, not reinforcement learning. These are fairly uninformed takes and I’m sure I’m…
playlist_add
Jan 10, 2023
AF - The Alignment Problem from a Deep Learning Perspective (major rewrite) by SoerenMind
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Alignment Problem from a Deep Learning Perspective (major rewrite), published by SoerenMind on January 10, 2023 on The AI Alignment Forum.
We've recently uploaded a major rewrite of Richard Ngo's:
The Alignment Problem from a Deep Learning Perspective
We hope it can reach ML researchers by being more grounded in the deep learning literature and empirical findings, and being more rigorous than typical introductions to the alignment problem.
There are many changes. Most obviously, the paper was previously structured into three phases of training: planning towards a mix of desirable and undesirable internally-represented goals, situational awareness, and goals that generalize OOD. Now it is structured around three emergent phenomena: deceptive reward hacking, planning towards internally represented goa…
playlist_add
Jan 9, 2023
AF - Review AI Alignment posts to help figure out how to make a proper AI Alignment review by Oliver Habryka
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Review AI Alignment posts to help figure out how to make a proper AI Alignment review, published by Oliver Habryka on January 10, 2023 on The AI Alignment Forum.
I've had many conversations over the last few years about the health of the AI Alignment field and one of the things that has come up most frequently (including in conversations with Rohin, Buck and various Open Phil people) is that many people wish there was more of a review process in the AI Alignment field.
I also think there is a bunch of value in better review processes, but have felt hesitant to create something very official and central, since AI Alignment is a quite preparadigmatic field, which makes creating shared standards of quality hard, and because I haven't had the time to really commit to maintain something great here.
Separatel…
playlist_add
Jan 9, 2023
AF - [MLSN #7]: an example of an emergent internal optimizer by Josh Clymer
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [MLSN #7]: an example of an emergent internal optimizer, published by Josh Clymer on January 9, 2023 on The AI Alignment Forum.
As part of a larger community building effort, CAIS is writing a safety newsletter that is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here.
Welcome to the 7th issue of the ML Safety Newsletter! In this edition, we cover:
‘Lie detection’ for language models
A step towards objectives that incorporate wellbeing
Evidence that in-context learning invokes behavior similar to gradient descent
What’s going on with grokking?
Trojans that are harder to detect
Adversarial defenses for text classifiers
And much more.
Alignment
Discovering Latent Knowledge i…
playlist_add
Jan 9, 2023
AF - Trying to isolate objectives: approaches toward high-level interpretability by Arun Jose
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Trying to isolate objectives: approaches toward high-level interpretability, published by Arun Jose on January 9, 2023 on The AI Alignment Forum.
Epistemic status: There’s a lot in this post and my general approach while working on it that, in retrospect, wasn’t thought out well enough. I’m posting it because I figure sharing flawed ideas is better than letting this languish in a doc forever, while I work on improving them.
Thanks to Paul Colognese, Tassilo Neubauer, John Wentworth, and Fabien Roger for useful conversations, Kyle McDonell and Laria Reynolds for suggesting something that led me to some of the ideas I mention here, and Shoshannah Tekofsky for feedback on a draft.
I spent some time trying to find the high-level structure in a neural net corresponding to a deep reinforcement learning…
playlist_add
Jan 8, 2023
AF - Simulacra are Things by janus
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simulacra are Things, published by janus on January 8, 2023 on The AI Alignment Forum.
This is a note I wrote about a year ago. It's fairly self-contained, so I decided to make a post out of it after Vladimir_Nesov's comment caused me to dig up this text and TsviBT's The Thingness of Things reminded me of it again.
"Simulacra" refer to things simulated by simulators such as GPT, in the ontology introduced in Simulators.
What are simulacra?
“Physically”, they’re strings of text output by a language model. But when we talk about simulacra, we often mean a particular character, e.g. simulated Yudkowsky. Yudkowsky manifests through the vehicle of text outputted by GPT, but we might say that the Yudkowsky simulacrum terminates if the scene changes and he’s not in the next scene, even though the text…
playlist_add
Jan 8, 2023
AF - 200 COP in MI: Image Model Interpretability by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Image Model Interpretability, published by Neel Nanda on January 8, 2023 on The AI Alignment Forum.
This is the eighth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. Look up jargon in my Mechanistic Interpretability Explainer
I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
Motivating papers: Thread: Circuits, Multimodal Neurons in Artificial Neural Networks
Disclaimer: My area of expertise is language model interpretability, not image models - it would completely not surprise me if this section contains errors, or i…
playlist_add
Jan 6, 2023
AF - Categorizing failures as “outer” or “inner” misalignment is often confused by Rohin Shah
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Categorizing failures as “outer” or “inner” misalignment is often confused, published by Rohin Shah on January 6, 2023 on The AI Alignment Forum.
Pop quiz: Are the following failures examples of outer or inner misalignment? Or is it ambiguous?
(I’ve spoilered the scenarios so you look at each one separately and answer before moving on to the next. Even if you don’t want to actually answer, you should read through the scenarios for the rest of the post to make sense.)
Scenario 1: We train an AI system that writes pull requests based on natural language dialogues where we provide specifications. Human programmers check whether the pull request is useful, and if so the pull request is merged and the AI is rewarded. During training the humans always correctly evaluate whether the pull request is…
playlist_add
Jan 6, 2023
AF - Definitions of “objective” should be Probable and Predictive by Rohin Shah
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Definitions of “objective” should be Probable and Predictive, published by Rohin Shah on January 6, 2023 on The AI Alignment Forum.
Introduction
Core arguments about existential risk from AI misalignment often reason about AI “objectives” to make claims about how they will behave in novel situations. I often find these arguments plausible but not rock solid because it doesn’t seem like there is a notion of “objective” that makes the argument clearly valid.
Two examples of these core arguments:
AI risk from power-seeking. This is often some variant of “because the AI system is pursuing an undesired objective, it will seek power in order to accomplish its goal, which causes human extinction”. For example, “The AI does not hate you, nor does it love you, but you are made out of atoms wh…
playlist_add
Jan 6, 2023
AF - 200 COP in MI: Techniques, Tooling and Automation by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Techniques, Tooling and Automation, published by Neel Nanda on January 6, 2023 on The AI Alignment Forum.
This is the seventh post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. Look up jargon in my Mechanistic Interpretability Explainer
I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
Motivating papers: Causal Scrubbing, Logit Lens
Motivation
In Mechanistic Interpretability, the core goal is to form true beliefs about what’s going on inside a network. The search space of possible circuits is extremely large, and even o…
playlist_add
Jan 6, 2023
AF - AI Safety Camp, Virtual Edition 2023 by Linda Linsefors
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Camp, Virtual Edition 2023, published by Linda Linsefors on January 6, 2023 on The AI Alignment Forum.
AI Safety Camp connects you with an experienced research lead to collaborate on a research project – helping you try your fit for a potential career in AI Safety research.
The applications for AI Safety Camp’s Virtual Edition in 2023 are now open!
AI Safety Camp Virtual 8 will be a 3.5-month long online research program from 4 March to 18 June 2023, where participants form teams to work on pre-selected projects.
We value people with diverse backgrounds and skillsets, such as cybersecurity or cognitive science. Not all projects require participants to have prior experience in AI Safety, mathematics or machine learning. You will be able to read in detail about the research topics & each pro…
playlist_add
Jan 5, 2023
AF - Paper: Superposition, Memorization, and Double Descent (Anthropic) by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: Superposition, Memorization, and Double Descent (Anthropic), published by Lawrence Chan on January 5, 2023 on The AI Alignment Forum.
(This is a follow up to Anthropic's prior work on Toy Models of Superposition.)
The authors study how neural networks interpolate between memorization and generalization in the "ReLU Output" toy model from the first toy model paper:
They train models to perform a synthetic regression task with T training points, for models with m hidden dimensions.
First, they find that for small training sets, while the features are messy, the training set hidden vectors hi=WXi (the projection of the input datapoints into the hidden space) often show clean structures:
They then extend their old definition of feature dimensionality to measure the dimensionalities allocated to each…
playlist_add
Jan 4, 2023
AF - Why I'm joining Anthropic by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I'm joining Anthropic, published by Evan Hubinger on January 5, 2023 on The AI Alignment Forum.
Personal blogpost. Previously: “What I'll be doing at MIRI”
For the last three years (since I left OpenAI), I've been a Research Fellow at MIRI. Starting next week, however, I'm going to be stepping down from that position and joining Anthropic as a researcher instead.[1]
To start with, some things that this does not mean:
That I have anything in particular against MIRI.
That I endorse everything Anthropic does.
That I'm going to stop doing theoretical AI safety research.
Any particular substantive change in my beliefs about AI safety.
So why do I think this is a good idea? The most basic reason is just that I think things are heating up and it'll be valuable for me to be closer to the action. I think…
playlist_add
Jan 4, 2023
AF - Contra Common Knowledge by Abram Demski
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra Common Knowledge, published by Abram Demski on January 4, 2023 on The AI Alignment Forum.
This post assumes some background knowledge about "common knowledge", a technical term in philosophy. See the tag page for an introduction and a list of LessWrong essays on the subject.
Epistemic effort: basically reading the SEP article on common knowledge and forming my own conclusions. I've also gained something by discussing these issues with Scott Garrabrant and Sahil Kulshrestha.
Consider a statement like "LessWrong curated posts help to establish common knowledge".
On the one hand, this seems obviously valid in practice. Curating a post boosts the number of people who see it, in a public way which also communicates the meta-knowledge (you know that lots of people will know about this post, and they'll…
playlist_add
Jan 4, 2023
AF - Basic Facts about Language Model Internals by Beren Millidge
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Basic Facts about Language Model Internals, published by Beren Millidge on January 4, 2023 on The AI Alignment Forum.
This post was written as part of the work done at Conjecture.
As mentioned in our retrospective, while also producing long and deep pieces of research, we are also experimenting with a high iteration frequency. This is an example of this strand of our work. The goal here is to highlight interesting and unexplained language model facts. This is the first in a series of posts which will be exploring the basic ‘facts on the ground’ of large language models at increasing levels of complexity.
Understanding the internals of large-scale deep learning models, and especially large language models (LLMs) is a daunting task which has been relatively understudied. Gaining such an understanding…
playlist_add
Jan 3, 2023
AF - Touch reality as soon as possible (when doing machine learning research) by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Touch reality as soon as possible (when doing machine learning research), published by Lawrence Chan on January 3, 2023 on The AI Alignment Forum.
Related to: Making Beliefs Pay Rent, The Feeling of Idea Scarcity, Micro-Feedback Loops and Learning, The Three Stages of Rigor, Research as a Stochastic Decision Process, Chapter 22 of HPMOR.
TL;DR: I think new machine learning researchers often make one of two kinds of mistakes: not making enough contact with reality, and being too reluctant to form gears-level models of ML phenomena. Stereotypically, LW/AF researchers tend to make the former mistake, while academic and industry researchers tend to make the latter kind. In this post, I discuss what I mean by “touching reality” and why it’s important, speculate a bit on why people don’t do this, and…
playlist_add
Jan 2, 2023
AF - 200 COP in MI: Exploring Polysemanticity and Superposition by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Exploring Polysemanticity and Superposition, published by Neel Nanda on January 3, 2023 on The AI Alignment Forum.
This is the fifth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started.
I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
Motivating paper: Toy Models of Superposition, Softmax Linear Units
Background
If you're familiar with polysemanticity and superposition, skip to Motivation or Problems.
Neural networks are very high dimensional objects, in both their parameters and their activations. One of the key challenge…
playlist_add
Jan 2, 2023
AF - [Simulators seminar sequence] #1 Background and shared assumptions by Jan Hendrik Kirchner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Simulators seminar sequence] #1 Background & shared assumptions, published by Jan Hendrik Kirchner on January 2, 2023 on The AI Alignment Forum.
Meta: Over the past few months, we've held a seminar series on the Simulators theory by janus. As the theory is actively under development, the purpose of the series is to discover central structures and open problems. Our aim with this sequence is to share some of our discussions with a broader audience and to encourage new research on the questions we uncover. Below, we outline the broader rationale and shared assumptions of the participants of the seminar.
Shared assumptions
Going into the seminar series, we determined a number of assumptions that we share. The degree to which each participant subscribes to each assumption varies, but we agreed to postpone…
playlist_add
Jan 2, 2023
AF - Large language models can provide "normative assumptions" for learning human preferences by Stuart Armstrong
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Large language models can provide "normative assumptions" for learning human preferences, published by Stuart Armstrong on January 2, 2023 on The AI Alignment Forum.
In a past result I demonstrated the impossibility of deducing the goals of a (potentially) irrational agent from their behaviour. To do that deduction, one needs to add extra assumptions - assumptions that cannot derive solely from observations. These assumptions were designated "normative assumptions".
Stuart Russell has questioned the practical impact of the result. He pointed to a game that Kasparov played against Deep Blue in 1997; a game that Kasparov actually won. He argued that it would be ridiculous to assume that Kasparov was actually trying to lose that game - but messed up, and ended up winning it instead.
And indeed it would be…
playlist_add
Jan 1, 2023
AF - The Thingness of Things by Tsvi Benson-Tilsen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Thingness of Things, published by Tsvi Benson-Tilsen on January 1, 2023 on The AI Alignment Forum.
[Metadata: crossposted from. I'm fairly likely to not respond to comments promptly. If you're especially interested in chatting, my gmail address is: tsvibtcontact ]
What's a thing, in general? Minds deal with things, so this question comes up in trying to understand minds. Minds think about things, speak of things, manipulate things, care about things, create things, and maybe are made of things.
Things
Examples of things: table, elephant, carbon atom. France, Martin Luther King. Insertion sort. Chess. Unicorn. Learning. Seven. Towel, strand, wing, crystal, finger, space, diffraction. The laws of electromagnetism. The first World War.
Non-things
What about non-things? It might be hard to list non-thin…
playlist_add
Jan 1, 2023
AF - Löb's Theorem for implicit reasoning in natural language: Löbian party invitations by Andrew Critch
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Löb's Theorem for implicit reasoning in natural language: Löbian party invitations, published by Andrew Critch on January 1, 2023 on The AI Alignment Forum.
Related to: Löb's Lemma: an easier approach to Löb's Theorem.
Natural language models are really taking off, and it turns out there's an analogue of Löb's Theorem that occurs entirely in natural language — no math needed. This post will walk you through the details in a simple example: a very implicit party invitation.
Motivation
(Skip this if you just want to see the argument.)
Understanding the structure here may be helpful for anticipating whether Löbian phenomena can, will, or should arise amongst language-based AI systems. For instance, Löb's Theorem has implications for the emergence of cooperation and defection in groups of formally…
playlist_add
Dec 31, 2022
AF - 'simulator' framing and confusions about LLMs by Beth Barnes
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 'simulator' framing and confusions about LLMs, published by Beth Barnes on December 31, 2022 on The AI Alignment Forum.
Post status: pretty rough + unpolished, thought it might be worthwhile getting this out anywayI feel like I've encountered various people having misunderstandings of LLMs that seem to be related to using the 'simulator' framing. I'm probably being horrendously uncharitable to the people in question, I'm not confident that anyone actually holds any of the opinions that are outlined below, and even if they do I'm not sure that they're actually attributable to the simulators framing, but it seemed like it might be useful to point at areas of potential confusion.
In general I'm skeptical that the simulator framing adds much relative to 'the model is predicting what token would appear next…
playlist_add
Dec 31, 2022
AF - 200 COP in MI: Interpreting Algorithmic Problems by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Interpreting Algorithmic Problems, published by Neel Nanda on December 31, 2022 on The AI Alignment Forum.
This is the fourth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started.
I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
Motivation
Motivating paper: A Mechanistic Interpretability Analysis of Grokking
When models are trained on synthetic, algorithmic tasks, they often learn to do some clean, interpretable computation inside. Choosing a suitable task and trying to reverse engineer a model can be a rich area of interes…
playlist_add
Dec 30, 2022
AF - Models Don't "Get Reward" by Sam Ringer
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Models Don't "Get Reward", published by Sam Ringer on December 30, 2022 on The AI Alignment Forum.
In terms of content, this has a lot of overlap with Reward is not the optimization target. I'm basically rewriting a part of that post in language I personally find clearer, emphasising what I think is the core insight.
When thinking about deception and RLHF training, a simplified threat model is something like this:
A model takes some actions.
If a human approves of these actions, the human gives the model some reward.
Humans can be deceived into giving reward in situations where they would otherwise not if they had more knowledge.
Models will take advantage of this so they can get more reward.
Models will therefore become deceptive.
Before continuing, I would encourage you to really engage with the above…
playlist_add
Dec 29, 2022
AF - 200 COP in MI: Looking for Circuits in the Wild by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Looking for Circuits in the Wild, published by Neel Nanda on December 29, 2022 on The AI Alignment Forum.
This is the third post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started.
I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
Motivation
Motivating paper: Interpretability In The Wild
Our ultimate goal is to be able to reverse engineer real, frontier language models. And the next step up from toy language models is to look for circuits in the wild. That is, taking a real model that was not designed to be tractable to interpr…
playlist_add
Dec 29, 2022
AF - Internal Interfaces Are a High-Priority Interpretability Target by Thane Ruthenis
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Internal Interfaces Are a High-Priority Interpretability Target, published by Thane Ruthenis on December 29, 2022 on The AI Alignment Forum.
tl;dr: ML models, like all software, and like the NAH would predict, must consist of several specialized "modules". Such modules would form interfaces between each other, and exchange consistently-formatted messages through these interfaces. Understanding the internal data formats of a given ML model should let us comprehend an outsized amount of its cognition, and allow to flexibly interfere in it as well.
1. A Cryptic Analogy
Let's consider three scenarios. In each of them, you're given the source code of a set of unknown programs, and you're tasked with figuring out their exact functionality. Details vary:
In the first scenario, the programs are written in some…
playlist_add
Dec 28, 2022
AF - 200 COP in MI: The Case for Analysing Toy Language Models by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: The Case for Analysing Toy Language Models, published by Neel Nanda on December 28, 2022 on The AI Alignment Forum.
This is the second post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started.
I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
Disclaimer: Mechanistic Interpretability is a small and young field, and I was involved with much of the research and resources linked here. Please take this sequence as a bunch of my personal takes, and try to seek out other researcher’s opinions too!
Motivation
In A Mathematical Framewo…
playlist_add
Dec 28, 2022
AF - 200 Concrete Open Problems in Mechanistic Interpretability: Introduction by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 Concrete Open Problems in Mechanistic Interpretability: Introduction, published by Neel Nanda on December 28, 2022 on The AI Alignment Forum.
This is the first post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. If you want to learn the basics before you think about open problems, check out my post on getting started.
I’ll make another post every 1-2 days, giving a new category of open problems - the second post is out now. If you want to read ahead, check out the draft sequence here!
Disclaimer: Mechanistic Interpretability is a small and young field, and I was involved with much of the research and resources linked here. Please take this sequence as a bunch of my personal takes, and try to seek out other researcher’s opinions too!
Introduction
Mechanistic Inte…
playlist_add
Dec 28, 2022
AF - In Defense of Wrapper-Minds by Thane Ruthenis
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: In Defense of Wrapper-Minds, published by Thane Ruthenis on December 28, 2022 on The AI Alignment Forum.
Recently, there's been a strong push against "wrapper-minds" as a framework. It's argued that there's no specific reason to think that all sufficiently advanced agents would format their goals in terms of expected-utility maximization over future trajectories, and that this view predicts severe problems with e. g. Goodharting that just wouldn't show up in reality.
I think these arguments have merit, and the Shard Theory's model definitely seems to correspond to a real stage in agents' value formation.
But I'd like to offer a fairly prosaic argument in favor of wrapper-minds.
Suppose that we have some agent which is being updated by some greedy optimization process (the SGD, evolution, etc.). On avera…
playlist_add
Dec 26, 2022
AF - Can we efficiently distinguish different mechanisms? by Paul Christiano
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Can we efficiently distinguish different mechanisms?, published by Paul Christiano on December 27, 2022 on The AI Alignment Forum.
(This post is an elaboration on “tractability of discrimination” as introduced in section III of Can we efficiently explain model behaviors? For an overview of the general plan this fits into, see Mechanistic anomaly detection and Finding gliders in the game of life.)
Background
We’d like to build AI systems that take complex actions to protect humans and maximize option value. Powerful predictive models may play an important role in such AI, either as part of a model-based planning algorithm or as part of a learned policy. Either way, this introduces the risk that we will select actions predicted to lead to good-looking outcomes without being able to distinguish predi…
playlist_add
Dec 26, 2022
AF - Avoiding perpetual risk from TAI by Stephen Casper
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Avoiding perpetual risk from TAI, published by Stephen Casper on December 26, 2022 on The AI Alignment Forum.
Stephen Casper, scasper@mit.edu
The goal of this post is to sort through some questions involving the difficulty of avoiding perpetual risk from transformative AI. Feedback in the comments is welcome!
TL;DR
Getting AI to go well means that at some point in time, the acute period of risk posed by the onset of transformative AI must end. Ending that period will require establishing a regime for transformative AI that is exclusive, benevolent, stable, and successful at alignment. This post argues that this type of regime may be very difficult to establish and that this will largely be an AI governance problem. If so, this gives an argument for emphasizing work on AI safety challenges other than jus…
playlist_add
Dec 26, 2022
AF - Coherent extrapolated dreaming by Alex Flint
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Coherent extrapolated dreaming, published by Alex Flint on December 26, 2022 on The AI Alignment Forum.
This work was supported by the Monastic Academy for the Preservation of Life on Earth. You can support my work here.
I will give a short presentation of this work followed by discussion on Wednesday Dec 28 at 12pm Pacific / 3pm Eastern. RSVP here.
Outline
I have four questions above coherent extrapolated volition, which I present in the form of four short thought experiments:
What kind of a thing can be extrapolated in the direction of wisdom? (Robot vacuum thought experiment)
What kind of protocol connects with the wisdom of a person who has been extrapolated? (Dream research thought experiment)
What kind of model captures that within a person that we hope to amplify through extrapolation? (Twitter i…
playlist_add
Dec 26, 2022
AF - Analogies between Software Reverse Engineering and Mechanistic Interpretability by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Analogies between Software Reverse Engineering and Mechanistic Interpretability, published by Neel Nanda on December 26, 2022 on The AI Alignment Forum.
These are notes taken during a call with Itay Yona, an expert in software/hardware reverse engineering (SRE). Itay gave me an excellent distillation of key ideas and mindsets in the field, and we discussed analogies/disanalogies to mechanistic interpretability of neural networks. I’m generally very excited to learn about other fields of study that reverse engineer complex systems, and what relevant insights they may have (SRE, neuroscience, systems biology, etc). All mistakes are mine, and all insights are his!
My Takeaways
The underlying mindset actually feels pretty analogous!
I find it super interesting that they also think a lot about motifs (weir…
playlist_add
Dec 25, 2022
AF - Concrete Steps to Get Started in Transformer Mechanistic Interpretability by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concrete Steps to Get Started in Transformer Mechanistic Interpretability, published by Neel Nanda on December 25, 2022 on The AI Alignment Forum.
Disclaimer: This post mostly links to resources I've made. I feel somewhat bad about this, sorry! Transformer MI is a pretty young and small field and there just aren't many people making educational resources tailored to it. Some links are to collations of other people's work, and I link to more in the appendix.
Introduction
Feel free to just skip the intro and read the concrete steps
The point of this post is to give concrete steps for how to get a decent level of baseline knowledge for transformer mechanistic interpretability (MI). This is an experiment in trying to give concrete, actionable, goal-oriented advice that I think is enough to get decent outcom…
playlist_add
Dec 25, 2022
AF - Take 14: Corrigibility isn't that great. by Charlie Steiner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 14: Corrigibility isn't that great., published by Charlie Steiner on December 25, 2022 on The AI Alignment Forum.
As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day some days for 25 days.
It's the end (I saved a tenuous one for ya')! Kind of disappointing that this ended up averaging out to one every 2 days, but this was also a lot of work and I'm happy with the quality level. Some of the drafts that didn't work as "hot takes" will get published later.
I
There are certainly arguments for why we want to build corrigible AI. For example, the problem of fully updated deference says that if you build an AI that wants things, even if it's uncertain about what it wants, it knows it can get more of what it wants if it doesn't let you turn it o…
playlist_add
Dec 23, 2022
AF - Löb's Lemma: an easier approach to Löb's Theorem by Andrew Critch
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Löb's Lemma: an easier approach to Löb's Theorem, published by Andrew Critch on December 24, 2022 on The AI Alignment Forum.
tl;dr: Löb's Theorem is much easier to grok if you separate the parts of the proof that use the assumption □pp from the parts that don't. The parts that don't use □pp can be extracted as a stand-alone result, which I hereby dub "Löb's Lemma".
Here's how it works.
Key properties of ⊢ and □
The proofs here will use the following standard properties of ⊢ and □ (source: Wikipedia), which respectively stand for provability and provability encoded within arithmetic:
(necessitation) From ⊢A, conclude ⊢□A. Informally, this says that if A can be proven, then it can be proven that it can be proven (by just writing out and checking the proof within arithmetic).
(interna…
playlist_add
Dec 22, 2022
AF - Response to Holden’s alignment plan by Alex Flint
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Response to Holden’s alignment plan, published by Alex Flint on December 22, 2022 on The AI Alignment Forum.
This work was supported by the Monastic Academy for the Preservation of Life on Earth. You can support my work directly here.
I will give a short presentation of this work followed by discussion on Saturday (Dec 24) at 9am Pacific / 12pm Eastern. RSVP here.
Outline
This is a response to Holden Karnofsky’s plan for aligning powerful AI systems in the case that such systems are developed soon.
I give a summary of Holden’s plan, and then critique it.
My basic critique is that the tools Holden is proposing are too similar to what has been tried in the construction of human institutions, and I argue that we have failed to create large numbers of human institutions that alleviate rather than exac…
playlist_add
Dec 22, 2022
AF - Take 13: RLHF bad, conditioning good. by Charlie Steiner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 13: RLHF bad, conditioning good., published by Charlie Steiner on December 22, 2022 on The AI Alignment Forum.
As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day some days for 25 days. I have now procrastinated enough that I probably have enough hot takes.
Hyperbolic title, sorry. But seriously, conditioning is better than RLHF for current language models. For agents navigating the real world, both have issues and it's not clear-cut where progress will come from.
By "conditioning", I mean the decision transformer trick to do conditional inference: get human ratings of sequences of tokens, and then make a dataset where you append the ratings to the front of the sequences. A model trained on this dataset for next-token prediction will hav…
playlist_add
Dec 21, 2022
AF - Decisions: Ontologically Shifting to Determinism by Chris Leong
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Decisions: Ontologically Shifting to Determinism, published by Chris Leong on December 21, 2022 on The AI Alignment Forum.
In a previous post, I posited that a significant portion of the confusion surrounding counterfactuals can be traced back to an ontological shift. In this post, I will seek to elaborate on this argument and explain how we should reinterpret decision theory problems in light of this insight.
Our Naive Ontology Assumes Libertarian Free Will
Consider our naive ontology, by which I mean how humans tend to think without substantial philosophical reflection. In our naive ontology, we assume both that:
That we possess libertarian free will, ie. that our choices are not predetermined before we make them.
That when we're making a decision, both the past and present state of the universe are f…
playlist_add
Dec 21, 2022
AF - A Comprehensive Mechanistic Interpretability Explainer and Glossary by Neel Nanda
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Comprehensive Mechanistic Interpretability Explainer & Glossary, published by Neel Nanda on December 21, 2022 on The AI Alignment Forum.
This is a linkpost for a very long doc defining, explaining, and giving intuitions and conceptual frameworks for all the concepts I think you should know about when engaging with mechanistic interpretability. If you find the UI annoying, there's an HTML version here
Why does this doc exist?
The goal of this doc is to be a comprehensive glossary and explainer for Mechanistic Interpretability (focusing on transformer language models), the field of studying how to reverse engineer neural networks.
There's a lot of complex terms and jargon in the field! And these are often scattered across various papers, which tend to be pretty well-written but not designed to be an int…
playlist_add
Dec 20, 2022
AF - CIRL Corrigibility is Fragile by Rachel Freedman
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CIRL Corrigibility is Fragile, published by Rachel Freedman on December 21, 2022 on The AI Alignment Forum.
Tl;dr: An optimal CIRL agent is corrigible under a certain problem formulation and given certain assumptions. By examining three key assumptions—that the game is played only once, that the human is rational, and that human feedback is free—we demonstrate how minor changes can lead to incorrigible CIRL agents. This suggests that corrigibility is highly sensitive to the underlying assumptions, and we should be wary of assuming that formal corrigibility results will transfer to real-world systems. However, our results do gesture at a class of near-corrigible behavior that may be protective against the most catastrophic outcomes. It may make more sense to consider corrigibility as a continuum rath…
playlist_add
Dec 20, 2022
AF - Discovering Language Model Behaviors with Model-Written Evaluations by Evan Hubinger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Discovering Language Model Behaviors with Model-Written Evaluations, published by Evan Hubinger on December 20, 2022 on The AI Alignment Forum.
“Discovering Language Model Behaviors with Model-Written Evaluations” is a new Anthropic paper by Ethan Perez et al. that I (Evan Hubinger) also collaborated on. I think the results in this paper are quite interesting in terms of what they demonstrate about both RLHF (Reinforcement Learning from Human Feedback) and language models in general.
Among other things, the paper finds concrete evidence of current large language models exhibiting:
convergent instrumental goal following (e.g. actively expressing a preference not to be shut down),
non-myopia (e.g. wanting to sacrifice short-term gain for long-term gain),
situational awareness (e.g. awareness of being…
playlist_add
Dec 20, 2022
AF - Note on algorithms with multiple trained components by Steve Byrnes
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Note on algorithms with multiple trained components, published by Steve Byrnes on December 20, 2022 on The AI Alignment Forum.
Example 1: consider a GAN. There’s a generator and a discriminator. As an intuitive mnemonic, we can say
The “purpose” of the generator is to trick the discriminator,
The “purpose” of the discriminator is to not get tricked by the generator.
(Relatedly, people will say “the generator is trained to trick the discriminator”, etc.)
.But (I hope) everyone knows that these bullet points are only a mnemonic.
The one and only real “purpose” of the whole system and everything in it is to generate cool images that we like, and get our papers into NeurIPS or whatever.
And indeed, I think everyone who uses GANs is aware that it’s possible for a programmer to make the di…
playlist_add
Dec 20, 2022
AF - An Open Agency Architecture for Safe Transformative AI by davidad (David A. Dalrymple)
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Open Agency Architecture for Safe Transformative AI, published by davidad (David A. Dalrymple) on December 20, 2022 on The AI Alignment Forum.
Note: my quality bar here is "this is probably worth the reader's time" rather than "this is as clear, compelling, and comprehensive as I can make it."
In the Neorealist Success Model, I asked:
What would be the best strategy for building an AI system that helps us ethically end the acute risk period without creating its own catastrophic risks that would be worse than the status quo?
This post is a first pass at communicating my current answer.
Bird's-eye view
At the top level, it centres on a separation between
learning a world-model from data and eliciting desirabilities within that ontology
planning against a world-model and associated desirabilities
acting…
playlist_add
Dec 20, 2022
AF - Take 12: RLHF's use is evidence that orgs will jam RL at real-world problems. by Charlie Steiner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 12: RLHF's use is evidence that orgs will jam RL at real-world problems., published by Charlie Steiner on December 20, 2022 on The AI Alignment Forum.
As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day some days for 25 days. I have now procrastinated enough that I probably have enough hot takes.
I felt like writing this take a little more basic, so that it doesn't sound totally insane if read by an average ML researcher.
Use of RLHF by OpenAI is a good sign in that it shows how alignment research can get adopted by developers of cutting-edge AI. I think it's even a good sign overall, probably. But still, use of RLHF by OpenAI is a bad sign in that it shows that jamming RL at real-world problems is endorsed as a way to make impressive pr…
playlist_add
Dec 19, 2022
AF - The "Minimal Latents" Approach to Natural Abstractions by johnswentworth
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The "Minimal Latents" Approach to Natural Abstractions, published by johnswentworth on December 20, 2022 on The AI Alignment Forum.
Background: The Language-Learning Argument
How many functions are there from a 1 megabyte image to a yes/no answer to the question “does this image contain an apple?”.
Well, there are 8M bits in the image, so 28000000 possible images. The function can assign “yes” or “no” independently to each of those 28000000 images, so 228000000 possible functions. Specifying one such function by brute force (i.e. not leveraging any strong prior information) would therefore require 28000000 bits of information - one bit specifying the output on each of the 28000000 possible images. Even if we allow for some wiggle room in ambiguous images, those ambiguous images will still be…
playlist_add
Dec 19, 2022
AF - Towards Hodge-podge Alignment by Cleo Nardo
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Towards Hodge-podge Alignment, published by Cleo Nardo on December 19, 2022 on The AI Alignment Forum.
ABSTRACT: What could AI alignment look like if we had 6000+ full-time researchers and software developers? I've spent a few days thinking about that question, and this report is my first-draft attempt to answer it. This report sketches a highly-scalable research agenda: studying the hodge-podge assemblage of various "alignment primitives". Please let me know in the comments which ideas are novel and useful (if any), and which aren't.
Introduction
What is hodge-podge alignment?
I think that in the Technical AI Safety community, the following beliefs are common:
The adequate solution to the alignment problem (if it exists) will look like one thing.
That one thing might be quantilization, or debate, or EL…
playlist_add
Dec 18, 2022
AF - Event [Berkeley]: Alignment Collaborator Speed-Meeting by Alex Mennen
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Event [Berkeley]: Alignment Collaborator Speed-Meeting, published by Alex Mennen on December 19, 2022 on The AI Alignment Forum.
If you’re working in alignment research, or would like to be, and you would like help finding researchers with whom to collaborate, this event is for you.
Inspired by the structure of a speed-dating event, participants will spend the first two hours in short one-on-one conversations with each other, changing partners at regular intervals, all for the goal of assessing fit as collaborators. The event will end with one hour of free-for-all socializing.
Please RSVP through Eventbrite or Facebook:Facebook::
The event will take place at Constellation (2150 Shattuck Ave.) on Floor 11, on Jan 14, 2023.
Instructions for access:Upon arrival at the building (at the Shattuck Ave. entra…
playlist_add
Dec 18, 2022
AF - Take 11: "Aligning language models" should be weirder. by Charlie Steiner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 11: "Aligning language models" should be weirder., published by Charlie Steiner on December 18, 2022 on The AI Alignment Forum.
As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day some days for 25 days. I have now procrastinated enough that I probably have enough hot takes.
People often talk about aligning language models, either to promote it, or to pooh-pooh it. I'm here to do both.
Sometimes, aligning language models just means trying to get a present-day model not to say bad outputs that would embarrass your organization. There is a cottage industry of papers on arxiv doing slightly different variants of RLHF against bad behavior, measuring slightly different endpoints. These people deserve their light mockery for diluting the keywor…
playlist_add
Dec 17, 2022
AF - Looking for an alignment tutor by JanBrauner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Looking for an alignment tutor, published by JanBrauner on December 17, 2022 on The AI Alignment Forum.
Hey, this is me. I’d like to understand AI X-risk better. Is anyone interested in being my “alignment tutor”, for maybe 1 h per week, or 1 h every two weeks? I’m happy to pay.
Fields I want to understand better:
Anything related to prosaic AI alignment/existential ML safety
Failure stories/threat models
Fields I’m not interested in (right now):
agent foundation
decision theory
other very mathsy stuff that’s not related to ML
My level of understanding:
I have a decent knowledge of ML/deep learning (I’m in the last year of my PhD)
I haven’t done the AGI Safety Fundamentals course, but I just skimmed it, and I think I had independently read essentially all the core readings (which means I…
playlist_add
Dec 16, 2022
AF - Paper: Constitutional AI: Harmlessness from AI Feedback (Anthropic) by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: Constitutional AI: Harmlessness from AI Feedback (Anthropic), published by Lawrence Chan on December 16, 2022 on The AI Alignment Forum.
The authors propose a method for training a harmless AI assistant that can supervise other AIs, using only a list of rules (a "constitution") as human oversight. The method involves two phases: first, the AI improves itself by generating and revising its own outputs; second, the AI learns from preference feedback, using a model that compares different outputs and rewards the better ones. The authors show that this method can produce a non-evasive AI that can explain why it rejects harmful queries, and that can reason in a transparent way, better than standard RLHF:
Paper abstract:
As AI systems become more capable, we would like to enlist their help to supervise…
playlist_add
Dec 16, 2022
AF - Can we efficiently explain model behaviors? by Paul Christiano
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Can we efficiently explain model behaviors?, published by Paul Christiano on December 16, 2022 on The AI Alignment Forum.
ARC’s current plan for solving ELK (and maybe also deceptive alignment) involves three major challenges:
Formalizing probabilistic heuristic argument as an operationalization of “explanation”
Finding sufficiently specific explanations for important model behaviors
Checking whether particular instances of a behavior are “because of” a particular explanation
All three of these steps are very difficult, but I have some intuition about why steps #1 and #3 should be possible and I expect we’ll see significant progress over the next six months. Unfortunately, there’s no simple intuitive story for why step #2 should be tractable, so it’s a natural candidate for the main tech…
playlist_add
Dec 16, 2022
AF - Paper: Transformers learn in-context by gradient descent by Lawrence Chan
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: Transformers learn in-context by gradient descent, published by Lawrence Chan on December 16, 2022 on The AI Alignment Forum.
The paper argues that auto-regressive transformers implement in-context learning via gradient-based optimization on in-context data.
The authors start by pointing out that with a single linear self-attention (LSA) layer (that is, no softmax), a Transformer can implement one step of gradient descent on the l2 regression loss (a fancy way of saying w -= LR (w x-y)x^T), and confirm this result empirically. They extend this result by showing that an N-layer LSA-only transformer is similar to N-steps of gradient descent for small linear regression tasks, both in and out of distribution. They also find that the results pretty much hold with softmax self-attention (which isn’t…
playlist_add
Dec 15, 2022
AF - AI Neorealism: a threat model and success criterion for existential safety by davidad (David A. Dalrymple)
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Neorealism: a threat model & success criterion for existential safety, published by davidad (David A. Dalrymple) on December 15, 2022 on The AI Alignment Forum.
Threat Model
There are many ways for AI systems to cause a catastrophe from which Earth-originating life could never recover. All of the following seem plausible to me:
Misuse: An AI system could help a human or group of humans to destroy or to permanently take over (and lock their values into) the world. The AI could be:
An oracle AI (e.g. a question-answering LLM)
An LLM simulating an intent-aligned agent and taking real-world actions via APIs
An intent-aligned RL agent
An interaction of multiple systems
Power-Seeking: An AI system could destroy or permanently take over the world on its own account, by leveraging advanced instruments of for…
playlist_add
Dec 14, 2022
AF - Extracting and Evaluating Causal Direction in LLMs' Activations by Fabien Roger
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Extracting and Evaluating Causal Direction in LLMs' Activations, published by Fabien Roger on December 14, 2022 on The AI Alignment Forum.
This post was written by Fabien at SaferAI. Simeon has prompted Fabien in relevant directions and has provided valuable feedback.
Thanks to Jean-Stanislas Denain, Alexandre Variengien, Charbel-Raphael Segerie, and Nicole Nohemi for providing helpful feedback on early experiments and drafts of this post.
In this post
I describe a method to evaluate that a specific direction encodes information relative to a specific concept,
I use it to evaluate how good directions found through probing techniques are,
I present a way of using causal intervention to find directions that maximally encode information relative to a specific concept, which I call Causal Direction Extracti…
playlist_add
Dec 13, 2022
AF - Take 10: Fine-tuning with RLHF is aesthetically unsatisfying. by Charlie Steiner
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 10: Fine-tuning with RLHF is aesthetically unsatisfying., published by Charlie Steiner on December 13, 2022 on The AI Alignment Forum.
As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes. This take owes a lot to the Simulators discussion group.
Fine-tuning a large sequence model with RLHF creates an agent that tries to steer the sequence in rewarding directions. Simultaneously, it breaks some nice properties that the fine-tuned model used to have. You should have a gut feeling that we can do better.
When you start with a fresh sequence model, it's not acting like an agent, instead it's just trying to mimic the training distribution. It may contain agents, but at every step it's just going to o…
playlist_add
Dec 12, 2022
AF - An exploration of GPT-2's embedding weights by Adam Scherlis
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An exploration of GPT-2's embedding weights, published by Adam Scherlis on December 13, 2022 on The AI Alignment Forum.
I wrote this doc in December 2021, while working at Redwood Research. It summarizes a handful of observations about GPT-2's weights -- mostly the embedding matrix, but also the LayerNorm gain parameters -- that I found while doing some open-ended investigation of the model. I wanted to see how much I could learn by studying just those parameters, without looking at the attention layers, MLP layers, or activations.
This is still mostly unedited. Feedback and questions are very welcome.
Embedding matrix glossary
Latent space – 768-channel-dim space that activations between blocks (and in skip connections) live in.
Token space – 50,257-token-dim space of one-hot token vectors
[Token]…
playlist_add
Dec 12, 2022
AF - Concept extrapolation for hypothesis generation by Stuart Armstrong
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concept extrapolation for hypothesis generation, published by Stuart Armstrong on December 12, 2022 on The AI Alignment Forum.
Posted initially on the Aligned AI website. Authored by Patrick Leask, Stuart Armstrong, and Rebecca Gorman.
There’s an apocryphal story about how vision systems were led astray when trying to classify tanks camouflaged in forests. A vision system was trained on images of tanks in forests on sunny days, and images of forests without tanks on overcast days.
To quote Neil Fraser:
In the 1980s, the Pentagon wanted to harness computer technology to make their tanks harder to attack.
The research team went out and took 100 photographs of tanks hiding behind trees, and then took 100 photographs of trees—with no tanks. They took half the photos from each group and put them in a vau…
playlist_add
Dec 12, 2022
AF - Side channels: input versus output by davidad (David A. Dalrymple)
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Side channels: input versus output, published by davidad (David A. Dalrymple) on December 12, 2022 on The AI Alignment Forum.
This is a brief post arguing that, although "side-channels are inevitable" is pretty good common advice, actually, you can prevent attackers inside a computation from learning about what's outside.
This implies that we could use relatively elementary sandboxing (no clock access, no networking APIs, no randomness, none of these sources of nondeterminism, and that’s about it) to prevent a task-specific AI from learning any particular facts about, say, human psychology, virology, or biochemistry—if we are careful to only provide the training process with inputs that would be just as likely in, say, an alternate universe where AI was built by octopus minds made of organosilicon w…