The history of Stafford Castle dates from the period following the Norman Conquest in 1066. Lands held by the defeated Saxons were shared among the followers of William the Conqueror. As a result, the estates to the west of Stafford, formerly held by Earl Edwin were awarded to Robert de Tosny, a member of a powerful Norman family.

On the east side of his large estate Robert built a castle overlooking the town of Stafford -the castle which we know today as Stafford Castle. In the traditions of the time, it would have been a 'motte and bailey' castle, constructed of earth and timber. There would have been a mound (or motte) with a ditch around it, and a courtyard (or bailey) also surrounded by a bank and ditch. In some instances a castle might have two baileys; this happened at Stafford. Earth from the ditches was used to make the motte and the bank around the outside of the bailey. Palisades of timber would have defended the castle.

Over the next two hundred years Stafford Castle suffered a gradual period of decline, but all was to be changed by the career of Ralph Stafford who succeeded to his father's estates in 1323. Ralph's friendship with Edward III, his leading role as a soldier during the Hundred Years' War, and the inheritances gained through two marriages brought him great wealth and influence. In 1348 he was chosen as one of the founder members of the Order of the Garter, and in 1351 he was created the first Earl of Stafford.

It was in 1348 that Ralph contracted with John of Bicester to build a stone castle “on the motte in the manor of Stafford". The castle was to have “towers, halls, chambers, spiral staircases, windows, doors and gates". The resulting castle was rectangular in shape, with octagonal towers on each corner, and survived, with the addition of a fifth tower, until the Civil War nearly three hundred years later.

By the seventeenth century, however, the structure of the castle was, in the words of a contemporary diarist, “somewhat ruined”. But it was still occupied by Lady Isabel Stafford, who supported the Royalist cause in the Civil War. The town of Stafford fell to the Parliamentarians in 1643, and after a brief period of resistance the castle also fell. The local Parliamentary Committee shortly afterwards ordered that “Stafford Castle be forthwith demolished”.

In 1783 the Stafford Estate was inherited by Sir William Jerningham, a Norfolk gentleman. He uncovered the foundations of the medieval keep, and following his death his son, Edward Jerningham, was instrumental in 1813 in rebuilding the keep in a Gothic manner on the original medieval foundations "after the style of Edward III”.

During the twentieth century the keep fell into decay, and was donated to Stafford Borough Council by Lord Stafford in 1961. Subsequently a programme of archaeological and historic research has been carried out, and in May 1991 the Visitor Centre was opened.