The first section of the Palácio was built in 1449, on the top of the ancient fortified Roman and Moorish walls, called "Cerca do Alcáçova" and "Cerca Moura", by agregating three towers : a rectangular keep at the western extremity, a corner tower to the north built upon Roman foundations, and a pentagonal moorish tower of the VIIth century to the west, which built upon a forty meter high rock, makes up an integral part of the "Cerca Moura".

In 1640, the family enlarged the building, adding the majestic East terrace and five façades in a classical style, giving the Palácio its present look. In 1720 and 1730 the family commissionned two Portuguese master tile-makers, Manuel Santos and Valentim de Almeida, of a unique collection of Portuguese tiles. The 59 panels of more than 30,000 tiles are to be found throughout the building and are one of the Palácio’s distinguishing features.