The 'rm' command should complete pretty quickly. Actually freeing up space takes time since a delete is subject to delayed allocation. The default transaction commit time is 30 seconds. And if there are snapshots or reflink copies, a backref walk is needed before extents can be freed.
It can take multiple seconds (like 5-10) on my fileserver to delete just one ~10GB file.