Why don't verbs always agree with the subject in the perfect tense?

The simple explanation is this: it depends on which auxiliary is being used ("avoir" or "être"). When using "être", the verb always agrees with the subject. For example: "Il est tombé." versus "Elle est tombée".
When using "avoir", the verb agrees with the object of the sentence (but only if it is placed before the auxiliary, in this case "avoir"). Example 1: "Ils ont mangé une lasagne." Notice that the verb "manger" will only agree with lasagne if it is placed before "avoir".Example 2: "Ils l'ont mangée." Here, "une lasagne" has been placed before the verb by replacing it with an object pronoun. Notice however that in both of these examples, the verb never agrees with the subject of the sentence "ils".