Customize
Rationality+Rationality+World Modeling+World Modeling+AIAIWorld OptimizationWorld OptimizationPracticalPracticalCommunityCommunity
Personal Blog+
simeon_c12368
17
Idea: Daniel Kokotajlo probably lost quite a bit of money by not signing an OpenAI NDA before leaving, which I consider a public service at this point. Could some of the funders of the AI safety landscape give some money or social reward for this? I guess reimbursing everything Daniel lost might be a bit too much for funders but providing some money, both to reward the act and incentivize future safety people to not sign NDAs would have a very high value. 
Raemon40
0
New concept for my "qualia-first calibration" app idea that I just crystallized. The following are all the same "type": 1. "this feels 10% likely" 2. "this feels 90% likely" 3. "this feels exciting!" 4. "this feels confusing :(" 5. "this is coding related" 6. "this is gaming related" All of them are a thing you can track: "when I observe this, my predictions turn out to come true N% of the time". Numerical-probabilities are merely a special case (tho it still gets additional tooling, since they're easier to visualize graphs and calculate brier scores for) And then a major goal of the app is to come up with good UI to help you visualize and compare results for the "non-numeric-qualia". Depending on circumstances, it might seem way more important to your prior "this feels confusing" than "this feels 90% likely". (I'm guessing there is some actual conceptual/mathy work that would need doing to build the mature version of this)
Interest groups without an organizer. This is a product idea that solves a large coordination problem. With billion people, there could be a huge number of groups of people sharing multiple interests. But currently, the number of valuable groups of people is limited by a) the number of organizers and b) the number of people you meet via a random walk. Some progress has been made on (b) with better search, but it is difficult to make (a) go up because of human tendencies - most people are lurkers - and the incentive to focus on one area to stand out. So what is the idea? Cluster people by interests and then suggest the group to all members. If people know that the others know that there is interest, the chance of the group coming together gets much higher.
A list of some contrarian takes I have: * People are currently predictably too worried about misuse risks * What people really mean by "open source" vs "closed source" labs is actually "responsible" vs "irresponsible" labs, which is not affected by regulations targeting open source model deployment. * Neuroscience as an outer alignment[1] strategy is embarrassingly underrated. * Better information security at labs is not clearly a good thing, and if we're worried about great power conflict, probably a bad thing. * Much research on deception (Anthropic's recent work, trojans, jailbreaks, etc) is not targeting "real" instrumentally convergent deception reasoning, but learned heuristics. Not bad in itself, but IMO this places heavy asterisks on the results they can get. * ML robustness research (like FAR Labs' Go stuff) does not help with alignment, and helps moderately for capabilities. * The field of ML is a bad field to take epistemic lessons from. Note I don't talk about the results from ML. * ARC's MAD seems doomed to fail. * People in alignment put too much faith in the general factor g. It exists, and is powerful, but is not all-consuming or all-predicting. People are often very smart, but lack social skills, or agency, or strategic awareness, etc. And vice-versa. They can also be very smart in a particular area, but dumb in other areas. This is relevant for hiring & deference, but less for object-level alignment. * People are too swayed by rhetoric in general, and alignment, rationality, & EA too, but in different ways, and admittedly to a lesser extent than the general population. People should fight against this more than they seem to (which is not really at all, except for the most overt of cases). For example, I see nobody saying they don't change their minds on account of Scott Alexander because he's too powerful a rhetorician. Ditto for Eliezer, since he is also a great rhetorician. In contrast, Robin Hanson is a famously terrible rhetorician, so people should listen to him more. * There is a technocratic tendency in strategic thinking around alignment (I think partially inherited from OpenPhil, but also smart people are likely just more likely to think this way) which biases people towards more simple & brittle top-down models without recognizing how brittle those models are. ---------------------------------------- 1. A non-exact term ↩︎
koratkar1-6
0
Feeding on conflict and being in fights is something you can learn. It’s a great source of energy if you’re feeling low.

Popular Comments

Recent Discussion

To some, it might seem like a strange question. If you think of being college-educated as a marker of class (or personhood), the fact that I don't have a degree at age of thirty-six (!!) probably looks like a scandalous anomaly, which it would be only natural for me to want to remediate at the earliest opportunity.

I deeply resent that entire worldview—not because I've rejected education, properly understood. On the contrary. The study of literature, history, mathematics, science—these things are among the noblest pursuits in life, sources of highest pleasure and deepest meaning. It's precisely because I value education so much that I can't stand to see it conflated with school and its culture of bureaucratic servitude where no one cares what you know and no one...

cata20

I still don't get why you are even considering finishing the degree, even though you clearly tried to explain it to me. Taking eight college classes is a lot of work actually? "Why not" doesn't really seem to cover it. How is doing a "terrible" commute several times per week for two semesters and spending many hours per week a low cost?

You sort of imply that someone is judging you for not having the degree but you didn't give any examples of actually being judged.

If you really really want to prove to yourself that you can do it, or if you really want to learn more math (I agree that taking college courses seems like a fine way to learn more math) then I understand, but based on your post it's not clear to me.

2Viliam
School degree may not be a strong signal, but it is legible. If I don't know math, I have no idea whether your math articles make sense, or you're a crackpot. If I don't know programming, I have no idea whether your commits are good. But everyone knows what "I have a degree" means. So basically, school degree is better for "impressing a lot of people a little bit", while the things you did are better for "impressing a few people a lot". Neither is strictly superior to the other. School typically tries to teach you a lot of things. You could learn any of them much better on your own, but it is unlikely that you would learn all of them, because there is too much knowledge out there. University-educated people will probably judge the knowledge they learned at university as elementary, so from their perspective, you have many gaps in elementary knowledge, which seems bad, even if you have deep knowledge in something else. And there will always be the question: "if you are smart enough to succeed at school, why didn't you?" So... if getting the degree is cheap, obviously go for it.
2Dave Orr
I went back to finish college as an adult, and my main surprise was how much fun it was. It probably depends on what classes you have left, but I took every AI class offered and learned a ton that is still relevant to my work today, 20 years later. Even the general classes were fun -- it turns out it's easy to be an excellent student if you're used to working a full work week, and being a good student is way more pleasant and less stressful than being a bad one, or at least it was for me. I'm not sure what you should do necessarily, but given that you're thinking about this less as useful for anyone in particular and more for other reasons, fun might be a good goal. As it happened I think the credential ended up useful too, but it was a top school so more valuable than many.
2cousin_it
I think once you're past a certain basic level in math, it's feasible to continue learning pretty much by yourself, just download problem sets and go through them. But it's a bit lonely. Going to classes lets you meet other people who are into the same thing as you! Shared love for something is what makes communities happen, I got this idea in Jim Butcher of all places. And the piece of paper itself is also quite nice, it can come in handy unexpectedly, and getting it now is probably easier than getting it later. So on the whole I'd lean toward getting the degree. About the philosophical stuff, I think "the world and/or my life will be over soon anyway" is kind of a nasty idea, because it makes you feel like nothing's worth doing. That's no way for a human being to be! You're not a potato! Hence it's better to act on the assumption that neither the world nor your life will be over anytime soon.

For all things that become much less interesting as time passes. This applies to newsletters and things covering current events. This tag can be useful for filtering them out.

Co-Authors: @Rocket, @Ryan Kidd, @LauraVaughan, @McKennaFitzgerald, @Christian Smith, @Juan Gil, @Henry Sleight

The ML Alignment & Theory Scholars program (MATS) is an education and research mentorship program for researchers entering the field of AI safety. This winter, we held the fifth iteration of the MATS program, in which 63 scholars received mentorship from 20 research mentors. In this post, we motivate and explain the elements of the program, evaluate our impact, and identify areas for improving future programs.

Summary

Key details about the Winter Program:

  • The four main changes we made after our Summer program were:
  • Educational attainment of MATS scholars:
    • 48% of scholars
...

Note that number of scholars is a much more important metric than number of mentors when it comes to evaluating MATS resources, as scholar per mentors varies a bunch (eg over winter I had 10 scholars, which is much more than most mentors). Harder to evaluate from the outside though!

2Ryan Kidd
This is potentially exciting news! You should definitely visit the LISA office, where many MATS extension program scholars are currently located.
2Sheikh Abdur Raheem Ali
I’m a LISA member already!
6Ryan Kidd
Last program, 44% of scholar research was on interpretability, 18% on evals/demos, 17% on oversight/control, etc. In summer, we intend for 35% of scholar research to be on interpretability, 17% on evals/demos, 27% on oversight/control, etc., based on our available mentor pool and research priorities. Interpretability will still be the largest research track and still has the greatest interest from potential mentors and applicants. The plot below shows the research interests of 1331 MATS applicants and 54 potential mentors who have applied for our Summer 2024 or Winter 2024-25 Programs.
Raemon20

I agree with this overall point, although I think "trade secrets" in the domain of AI can be relevant for people having surprising timelines views that they can't talk about.

1Orborde
Mostly for @habryka's sake: it sounds like you are likely describing your unvested equity, or possibly equity that gets clawed back on quitting. Neither of which is (usually) tied to signing an NDA on the way out the door - they'd both be lost simply due to quitting. The usual arrangement is some extra severance payment tied to signing something on your way out the door, and that's usually way less than the unvested equity.
2habryka
My current best guess is that actually cashing out the vested equity is tied to an NDA, but I am really not confident. OpenAI has a bunch of really weird equity arrangements.
4wassname
One is the change to the charter to allow the company to work with the military. https://news.ycombinator.com/item?id=39020778

Feeding on conflict and being in fights is something you can learn. It’s a great source of energy if you’re feeling low.

Something I'd like to try at LessOnline is to somehow iterate on the "Public Doublecrux" format. I'm not sure if I'll end up focusing on it, but here are some ideas.

Public Doublecrux is a more truthseeking oriented version of Public Debate. The goal of a debate is to change your opponent's mind or the public's mind. The goal of a doublecrux is more like "work with your partner to figure out if you should change your mind, and vice versa."

Reasons to want to do public doublecrux include:

  • It helps showcase subtle mental moves that are hard to write down explicitly (i.e. tacit knowledge transfer.
  • There's still something good and exciting about seeing high profile smart people talk about ideas. Having some variant of that format seems good for LessOnline.
...
10cousin_it
Last year I had an idea for a debate protocol which got pretty highly upvoted.
2Raemon
Ah yeah, that actually seems like maybe a good format given that the event-in-question I'm preparing for is "a blogging festival". There is trouble with (one of my goals) being "make something that makes for an interesting in-person-event" (we sorta made our jobs hard by framing an in-person-event around blogging, although I think something like "get two attendees to do this sort of debate framework beforehand, and then maybe have an interviewer/facilitator have a "takeaways discussion panel" might be good) Copying the text here for convenience:

Cool. If you go with it, I'd be super interested to know how it went, and lmk if you need any help or elaboration on the idea.

To get the best posts emailed to you, create an account! (2-3 posts per week, selected by the LessWrong moderation team.)
Log In Reset Password
...or continue with

I have no experience with data science, but D&D Sci seems fun and I would like to improve and get better at it. Where can/should I start?

Fellow not-at-all-a-data-scientist-but-wait-actually-that-sounds-fun here! I don’t know more about it than you do, but I’m glad you asked, since I hope to also benefit from the answers you’ll get :-)

2Answer by Jay Bailey
To provide the obvious advice first: * Attempt a puzzle. * If you didn't get the answer, check the comments of those who did. * Ask yourself how you could have thought of that, or what common principle that answer has. (e.g, I should check for X and Y) * Repeat. I assume you have some programming experience here - if not, that seems like a prerequisite to learn. Or maybe you can get away with using LLM's to write the Python for you.
1FinalFormal2
That sounds like a pretty good basic method- I do have some (minimal) programming experience, but I didn't use it for D&D Sci, I literally just opened the data in Excel and tried looking at it and manipulating it that way. I don't know where I would start as far as using code to try and synthesize info from the dataset. I'll definitely look into what other people did though.

In 2018, I was a bright-eyed grad student who was freaking out about AI alignment. I guess I'm still a bright-eyed grad student freaking out about AI alignment, but that's beside the point. 

I wanted to help, and so I started levelling up. While I'd read Nate Soares's self-teaching posts, there were a few key lessons I'd either failed to internalize or failed to consider at all. I think that implementing these might have doubled the benefit I drew from my studies. 

I can't usefully write a letter to my past self, so let me write a letter to you instead, keeping in mind that good advice for past-me may not be good advice for you.

Make Sure You Remember The Content

TL;DR: use a spaced repetition system like Anki. Put...

Sting10

It is worth noting that Mathpix only allows 10 free snips per month. Of course, they do not tell you this until you have installed the program and created an account. 

Readers must be 15+ 

This is a story about AI and existential risk.

22.

20 years old, lives-alone-with-her-dad Victoria Brittleson was stood on the scales, (She’d shucked out of her clothes for accuracy). 

The pointer on the gauge swung, like the merciless swing of the grim reaper’s scythe, then shivered to a conclusion. 

161 pounds.

Up from last week.

She rubbed her forehead, then got off the scales (the pointer flopped back to starting position), her most recent notebook, (she had been documenting her weight once a week, every week for the past four years). 

She was chubby.

Overweight.

But overweight was just one step from becoming obese, and obese people cost $260 billion in medical bills for Americans, in the year 2016. 

She was becoming a burden.

She was becoming unlovable.

She was becoming a leak in the bottom...

1datawitch
I would like to read the next chapter! I don't understand what happened at the end -- why was the AI written erotica trailing off into spelling mistakes?

Sorry that's my mistake. The erotica was written by a human being (I should have clarified), so it's horrible for no particular reason. 

But I'm very glad you're enjoying it! I'll try to release the third chapter sometime tomorrow. 

LessOnline Festival

Ticket prices increase in 3 days

Join us May 31st to June 2nd, at Lighthaven, Berkeley CA