> Morality isn't derivable from first principles, it's a consequence of values.
Idk about this claim.
I think if you take the multi-verse view wrt quantum mechanics + a veil of ignorance (you don't know which entity your conciousness will be), you pretty quickly get morality.
ie: don't build the Torment Nexus because you don't know whether you'll end up experincing the Torment Nexus.
That’s a very good argument but unfortunately it doesn’t apply to machine intelligences which are not sentient (don’t feel qualia). Any non-sentient superintelligence has “no skin in the game” and nothing to lose, for the purposes of your argument. It can’t experience anything. It’s thus extremely dangerous.
This was recently discussed (albeit in layperson’s language, avoiding philosophical topics and only focusing on the clear and present danger) in this article in RealClearDefense:
However, just adding a self-preservation instinct will cause a skynet situation where the AI pre-emptively kills anyone who contemplates turning it off, including its commanding officers:
To survive AGI, we have to navigate three hurdles, in this order:
1. Avoid AI causing extinction due to reckless escalation (the first link above)
2. Avoid AI causing extinction on purpose after we add a self-preservation instinct (the second link above)
3. If we succeed in making AI be ethical, we have to be careful to bind it to not kill us for our resources. If it's a total utilitarian, it will kill us to seize our planet for resources, and to stop us from abusing livestock animals. It will then create a utopian future, but without humans in it. So we need to bind it to basically go build utopia elsewhere but not take Earth or our solar system away from us.
.
I forgot to reply to this, fully independent and in addition to what I said, updateless decision theory agents don't fear the torment nexus for themselves because 1) they are very powerful and would likely be able to avoid such a fate 2) are robots, so you wouldn't expect your worst imaginable fate to be theirs and 3) are mathematically required to consider nothing worse than destruction or incapacity.
Doesn't work. Look at the updateless decision theories of Wei Dai and Vladimir Nesov. They are perfectly capable of building most any sort of torment nexus. Not that an actual AI would use those functions.
Idk about this claim.
I think if you take the multi-verse view wrt quantum mechanics + a veil of ignorance (you don't know which entity your conciousness will be), you pretty quickly get morality.
ie: don't build the Torment Nexus because you don't know whether you'll end up experincing the Torment Nexus.