A computer-implemented technique is described herein for monitoring a web application that runs on plural computing machines provided by a network-accessible system. The technique involves determining a set of application instances that are assessed as healthy by the system, generating addresses for those application instances, and then testing the application instances in the set using their respective addresses. Overall, the technique can be considered to employ an “outside-in” strategy insofar as it interacts with the system from the vantage point of an end user.