I might be wrong but mathematically we would call a 2-dimensional object a matrix and a 3- or higher-dimensional object a tensor.

Now I guess since the described auto grad algorithm works for arbitrarily high dimensional objects it makes sense to call these objects tensors.

Tensors are objects that combine in a certain way and act on vectors in a certain way, they roughly represent a collection of linear transformations. One way to represent a tensor is a matrix after fixing a basis.

Machine learning people use "tensor" to just mean an N-dimensional array of numbers. The term is divorced from its meaning in Physics and Mathematics, which caused me some confusion when I started looked at machine learning papers coming from physics.

Again, doesn't seem so divorced: "N-dimensional array of numbers" pretty much works for Mathematics from my understanding.