The 'rm' command should complete pretty quickly. Actually freeing up space takes time since a delete is subject to delayed allocation. The default transaction commit time is 30 seconds. And if there are snapshots or reflink copies, a backref walk is needed before extents can be freed.