The reason the Digital Panopticon project is able to trace the lives of convicts in such detail is the growing tendency of British local government to collect information about the lives and personal characteristics of criminals.  From the late eighteenth century, and culminating in the registers mandated by the Habitual Criminals Act of 1869, information about places of origin, occupation, physical features, literacy, religion, previous offences, and behaviour while in prison was increasingly collected, leading to entries in the Habitual Criminals Register such as the following:

TNA, MEPO 6/5 (1893), reproduced from David Hawkings, Criminal Ancestors: A Guide to the Historical Criminal Records in England and Wales (Stroud, 2009), p. 164.

We can trace the origins of these registers back to the detailed evidence about accused and convicted criminals kept by Justice of the Peace John Fielding in London from the 1750s to the 1770s (which were destroyed in the 1780 Gordon Riots), and the Criminal Registers, kept first by the City of London and then by the Home Office from 1791, which recorded information about those committed to Newgate Prison in London.

For example, the last line of this table is about Benjamin Edmondson, committed in September 1791, ‘aged 16, born in Rosemary Lane, 5 feet 10 [inches tall], sallow complexion, short brown hair, by trade a rope maker’.  Accused of stealing a pocket handkerchief, he was convicted at the Old Bailey and transported for seven years.

It is information like this which makes the Digital Panopticon project possible; and the absence of such information explains why it is impossible to carry out a similar project for an earlier time period.

But if we are to understand the information collected, understand its limitations and use it effectively, it’s important to know why these details were collected in the first place.  The most straightforward and compelling explanation is the desire to take advantage of the increasing power of the state in order to control prisoners. This was what the philosopher Jeremy Bentham intended when he invented the ‘panopticon’ as a form of imprisonment in 1791, and this term was picked up by the philosopher Michel Foucault as a metaphor for the tendency of modern disciplinary societies (or states) to seek total control over their populations.  But when we examine when and by whom this information was collected, and what it was used for, this explanation is not fully convincing.  Too much of this information was collected by low-level officials, with no apparent purpose in mind, and not actually used.

For example, while John Fielding’s primary purpose in collecting information at Bow Street was to keep a record of previous and suspected offenders so that he could make arrests when crimes took place, some of the information collected does not appear to have used.  He appears to have thought that information about previous convictions should influence sentencing decisions, but there is no evidence that the evidence was actually used for that purpose.  Similarly, the information he collected about place of birth, trade and age, and even more remarkably about prisoners’ handwriting abilities, does not appear to have had a clear purpose.

Similarly, while much of the information in the Criminal Registers may have been used to detect recidivists, as in the case of Charlotte Walker in 1798 (sentenced to transportation following numerous previous trials at the Old Bailey), and in sentencing and pardoning decisions (though once again there is little evidence that the information was actually used for this purpose), why did they record information about age and place of birth?  Interestingly, in one of the few instances of record-keeping becoming less systematic, the Home Office stopped collecting much of this information in 1802, perhaps because it wasn’t being used.  (The amount of information collected in the Criminal Registers only expanded again in 1834.)

And why did the clerks decide to keep the information in the Criminal Registers in tabular form? (This is one of the first examples of this method of recording evidence.)  Perhaps this was to encourage clerks to fill in every piece of information required, though as you can see from the example above this did not always happen—often clerks could not be bothered, and there were many blanks.  A tabular form makes statistics easier to compile, but there is no indication that they were used for this purpose (the first official criminal statistics were not compiled until 1805-10, and they only counted crimes, sentences and punishments, not personal characteristics).

When imprisonment became a primary means of punishment in the nineteenth century, prison records began to keep detailed records of all those committed, including not only physical descriptions, past convictions, and records of behaviour in prison, but also in some cases marital status (of men as well as women), number of children, religion, and whether prisoners could read or write.  In the controlled environment of a prison it was easy to collect such information, but once again the purpose of collecting some of this information is unclear.  Information about literacy may have been used to determine the type of instruction the prisoner would receive in prison, but such information started to be kept in the early 1830s, a decade before such instruction was actually introduced.

It is remarkable, therefore, how much information was collected without any clear idea of how it was going to be used.  The intentions of those who created these records are often not clear to us, but perhaps few of the creators knew either. What we may have here is a classic instance of ‘bureaucratic creep’.   Sometimes the form of the records was dictated by statute, and some information had to be kept for very practical reasons, but for the most part record keeping systems were devised by local officials (such as the Sheriffs of London or prison governors), who guessed what types of information might be useful or needed.  A culture of information gathering evolved, in an apparent attempt to understand the criminal and shape punishments, but without a clear sense of how specific types of information would be used.  Perhaps Foucault was right—Britain was becoming a disciplinary society in terms of the amount of information collected—but, unlike the activities of national security agencies today, the state lacked the power (and perhaps the will) to act on such information, let alone compel its collection on a regular basis.  Significantly, as Chris Williams has recently demonstrated, in 1874. a few years following the creation of the Habitual Criminals Register, it ‘was found to be a failure’.