The first records of the castle date back to the first half of the thirteenth century. At that time, it was in the possession of the Ortenburgs, the most powerful noble family in Upper Carniola and Carinthia. In 1256 Friderik made Waldenberg Castle the new centre of his estate, while Kamen Castle in the village of Begunje, and the developing market town of Radovljica were also parts of the estate.

Lipnica estate comprised the farms and the developing iron-forging villages of Kropa, Kamna Gorica, and Kolnica in the Lipnica Valley. The state passed into the hands of the Cilli in 1420, with the Habsburgs inheriting it from them in 1456. The Habsburgs leased the estate to the nobility.

The waning of the feudal system and the rise of the town of Radovljica resulted in the decline of Kamen Castle and Lipnica Castle. Furthermore, the latter was severely damaged in the 1509 earthquake and, later, by fire. Lipnica Castle's leaseholders did not rebuild it, and so it remains only a silent witness to its onetime splendour.

There is a legend that surrounds the death of Friderik III, the last count of Habsburg. His wife, Margaret Teck, is said to have poisoned him here with a poisoned apple. As a punishment, the mischievous countess was transformed into a disgusting snake with a golden crown on its head, which still slithers around the castle today. However, history contradicts this legend, as Margaret Teck passed away before her husband in 1408.

Another legend speaks of a hidden tunnel that links Lipnica Castle with the town of Radovljica under the Sava, but this story has not yet been confirmed by archaeological research.