There's a middle road here. Serve the assets via django, but have a reverse proxy sitting in front of your web server which caches all the subsequent requests for the same resource. (That's why getting the headers right is so important)