I agree there are a lot of things outside the computer that are a lot more difficult to reverse, but I think that we are maybe conflating things a bit. Most of us just need the code and data magic. We aren't all trying to automate doing the dishes or vacuuming the floors just yet.
But an AI assistant can do so much more damage in a short space of time.
It probably won't go wrong, but when it does go wrong you will feel immense pain.
I will keep low productivity in exchange for never having to deal with the fallout.