Scalable empirical testing of media file playback utilizes test hooks in each media player to support simulated human interaction and playback monitoring. A media crawler catalogs media files accumulated in a media file database to create a wordlist. One or more scalable instances of media tester accesses the wordlist to select items of work linked to media files. Work items and/or operating modes of media tester specify test parameters such as performance profiles or further define testing such as specifying repetitious playback on one or more media players. Media files are downloaded to and played by a scalable number of media players. Playback performance is monitored, analyzed and reported. Failure reports are accompanied by instructions to reproduce failures and cross-references to content or source code in media files. Failures can be audited by additional work items for follow-up testing.