When using X-Delete-After, or other expiration headers, swift will properly return 404 after an object has expired, but the listings will still return the expired objects. Per https://bugs.launchpad.net/swift/+bug/1069849 this is intended behaviour and the swift-object-expirer service is necessary for the listings to stop returning expired objects.
New users of expiration include the new data transfers between analytics and production, along with upcoming work on expiring thumbnails. In general implementations will have to handle 404 anyways, but my reading suggests that without swift-object-expirer the expired objects are never actually deleted, the api layer simply checks the object and reports 404 for existing-but-expired objects.