The Oratory of San Desiderio is all that remains of the Benedictine monastery church, consecrated in 1084. In 1440 Pope Eugene IV abolished the monastery and the building was gifted to the Compagnia dei Rossi or Disciplinati dei Servi di Maria, who transformed it into a structure able to offer aid to pilgrims and travellers. In 1516, the place was again used as a Clarisse monastery and remained active until Pietro Leopoldo and the Bishop of Pistoia Scipione de' Ricci had it abolished (1780-91). Thereafter, Cavalier Giulio Amati bought the property, but then sold it to three private citizens. In the 20th century, the building was turned into a timber warehouse, but in 1910 Alessandrina Gelli, Rospigliosi's widow, bequeathed it to the State, which then handed it over to the Ministry of National Education in 1938, so that it could be preserved and made open to the public.
The structure presents itself as a plain rectangular hall with a gabled roof, where the 16th-century fresco by Sebastiano Vini is prominent on the end side. This fresco, entitled "The Crucifixion of San Desiderio and the Ten Thousand Martyrs", actually depicts Acacio and the ten thousand martyrs, who, led by him, converted to Christianity and were crucified on Mount Ararat in Armenia on the orders of the Romans. In the work by Sebastiano Vini, elements from different backgrounds combine, including Venetian colourism, northern narrative layouts and a strongly rooted devotion. On the walls of the structure are two detached frescoes depicting Sant'Agnese and Maria Maddalena, both attributed to the Pistoiese school of the final years of the 14th century. Lastly, prior to 1786, the hall housed an inlaid wooden chorus, made in 1519 by Piero Mati, father of Giovanni, author together with Bartolomeo della Residenza, preserved in the Sala Maggiore of the Palazzo Comunale.