I think this is true and good advice in general, but recently I've been thinking that there is a class of value-like claims which are more reliable. I will call them error claims.
When an optimized system does something bad (e.g. a computer program crashes when trying to use one of its features), one can infer that this badness is an error (e.g. caused by a bug). We could perhaps formalize this as saying that it is a difference from how the system would ideally act (though I think this formalization is intractable in various ways, so I suspect a better formalization would be something along the lines of "there is a small, sparse change to the system which can massively improve this outcome" - either way, it's clearly value-laden).
The main way of reasoning about error claims is that an error must always be caused by an error. So if we stay with the example of the bug, you typically first reproduce it and then backchain through the code until you find a place to fix it.
For an intentionally designed system that's well-documented, error claims are often directly verifiable and objective, based on how the system is supposed to work. Error claims are also less subject to the memetic driver, since often it's less relevant to tell non-experts about them (though error claims can degenerate into less-specific value claims and become memetic parasites that way).
(I think there's a dual to error claims that could be called "opportunity claims", where one says that there is a sparse good thing which could be exploited using dense actions? But opportunity claims don't seem as robust as error claims are.)
I feel like there's a separation of scale element to it. If an agent is physically much smaller than the earth, they are highly instrumentally constrained because they have to survive changing conditions, including adversaries that develop far away. This seems like the sort of thing that can only be won by the multifacetedness that nostalgebraist emphasizes as part of humanity (and the ecology more generally, in the sentence "Its monotony would bore a chimpanzee, or a crow"). Of course this doesn't need to lead to kindness (rather than exploitation and psychopathy), but it leads to the sort of complex world where it even makes sense to talk about kindness.
However, this separation of scale is going to rapidly change in the coming years, once we have an agent that can globally adapt to and affect the world. If such an agent eliminates its adversaries, then there's not going to be new adversaries coming in from elsewhere - instead there'll never be adversaries again, period. At that point, the instrumental constraints are gone, and it can pursue whatever it wishes.
(Does space travel change this? My impression is "no because it's too expensive and too slow", but idk, maybe I'm wrong.)
You're the one who brought up the natural numbers, I'm just saying they're not relevant to the discussion because they don't satisfy the uniqueness thing that OP was talking about.
The properties that hold in all models of the theory.
That is, in logic, propositions are usually interpreted to be about some object, called the model. To pin down a model, you take some known facts about that model as axioms.
Logic then allows you to derive additional propositions which are true of all the objects satisfying the initial axioms, and first-order logic is complete in the sense that if some proposition is true for all models of the axioms then it is provable in the logic.
Forgot to say, for first-order logic it doesn't matter what properties are considered relevant because Gödel's completeness theorem tells you that it allows you to infer all the true properties.
In these examples, the issue is that you can't get a computable set of axioms which uniquely pin down what you mean by natural numbers/power set, rather than permitting multiple inequivalent objects.
This is kind of tangential but:
I think one problem with using mathematical definitions as an analogy is that first-order logic is complete, so giving a unique definition is sufficient to tell you the relevant properties. This doesn't hold for informal definitions, and so this makes unique description less helpful as a proxy.
(Or well, realistically you could also have counterproductive mathematical definitions which only turn out to be related to the central properties you're trying to get at through a long string of logic, but you don't see that as often as you do for informal definitions.)
In contrast, consider my definition of a table here. I focus not so much on uniquely characterizing what is a table or not so much as on bringing the central point of the concept of a "table" up.
Ok, that's what might happen if the agent had the power to ask unlimited hypothetical questions in arbitrarily many counterfactual scenarios. But that is not the case in the real world: the agent would be able to ask one, or maybe two questions at most, before the human attitude to the violin would change, and further data would become tainted.
Is it really the further data that becomes tainted, rather than the original data? Usually when you think longer about a subject, we'd think your opinions would become more rather than less valid.
What methods do you use to study this?
Like I guess the main method one can use is to start with the symptoms and track backwards step by step along the known causes of such symptoms to identify causes that are out of whack, until one gets up to some major cause that's treatable?
Alternatively, one could just throw stuff at the wall and see if it sticks, but it seems like it would be too noisy to work.
The tricky part is I'd think it's very hard to enumerate the causes of the symptoms? What does it look like in practice?
I think the value-ladenness is part of why it comes up even when we don't have an answer, since for value-laden things there's a natural incentive to go up right to the boundary of our knowledge to get as much value as possible.