Since flags can be applied automatically when data meets certain criteria, we need the ability to restrict users from being able add or remove them manually. Otherwise, users can remove a flag that should remain on the record because it's appropriately applied, which can really mess up data extraction, and causes confusion.
Because we don't have audit history, it's impossible to know why a flag was added or removed without examining each record. If we could limit user's manual access, we could at least rely on the flags being applied based on the system's set criteria.