Invalid application names will need to be manually cleaned up by the user, and as such this is a bc-break. Closes #3054