Category Archives: Epistemologies

The Challenge of Visualising 100,000 Convict Lives

The Digital Panopticon project is linking together a wide variety of criminal justice, genealogical, and biometric records to trace thousands of convict lives from birth to death.  Each story will start with a birth date anywhere from the mid eighteenth century to the mid nineteenth century, and will include a variety of events including convictions for minor offences, one or more Old Bailey trials and punishments, possible subsequent convictions, marriage, children, census records, and death.  We are calling these life archives, though many will only present fragments of lives, depending on the amount of evidence available.  One such fragment we have already assembled is that of John Davis, born in about 1817, convicted of stealing some clothes and other items from a dwelling house in 1836, incarcerated for a month on the hulks, and transported on the ship Moffatt to New South Wales, where he arrived several months later.

John Davis

Life Archive for John Davis

How do we summarise 100,000 stories like this?  How can we find common patterns among all the individual narratives?  The project is exploring a variety of visualisation techniques in order to summarise this evidence without, as much as possible, obscuring the complexity of the individual stories.  We have already used visualisations to assess levels of missing evidence and detect errors in the Old Bailey Proceedings (Men as Wives: Visualising Errors in the Old Bailey Proceedings Data and Seeing Things Differently: Visualising Patterns of Data from the Old Bailey Proceedings), and to identify patterns in individual datasets (Transportation Under the Macroscope); and Open Data and the Digital Panopticon). But how do we use visualisations to document relations between datasets?

There is a bewildering array of visualisation formats available, as this Google Images screenshot indicates. Which one should we choose?

Which visualisation?!

Which visualisation?!

The choice obviously depends on the nature of the information to be displayed. Our most successful record linkage so far is between the records of sentences (from the Old Bailey Proceedings) and the records of punishments experienced (primarily execution, transportation, and imprisonment).  You may be surprised to read that there was a considerable discrepancy between the punishments judges dictated to convicts in the Old Bailey courtroom and the actual punishments they received.  Following their sentences, many convicts received reduced punishments as a result of pardons, other decisions taken by penal officials, and ill health or death.

Most useful to us for representing these patterns are Sankey diagrams, which depict flows in many to many relationships. Individual lines trace individual journeys, but where the same paths are followed by many people they are brought together as thicker lines, the thickness of the line denoting the volume of the flow.

Old Bailey sentences vs actual penal outcomes, 1790-99

Old Bailey sentences vs actual penal outcomes, 1790-99

For example, this diagram traces the convicts’ experiences in the 1790s, focusing on the two main sentences of that decade, death and transportation.  We can see from this that only a proportion (28%) of those sentenced to death were actually executed, with many others being transported (following a conditional pardon), and a few experiencing other outcomes such as going into service in the army or navy (during the French wars) or death.  Only around two-thirds of those sentenced to transportation, similarly, were actually transported, with the remained ending up in the hulks (and then presumably discharged after a period), or having a small number of other outcomes.

The advantage of presenting the information in this way—as opposed to, for example, a table—is that it is readily understandable without obscuring the variety of the possible outcomes.  Moreover, the patterns which stand out pose questions for further research, such as how and why did so many potential transportees manage to evade this punishment–and what determined which punishments they actually received?  These are issues we are currently investigating.

But what happens when the variables become more complex, and the number of stages prisoners might go through multiplies?  This is the problem we are working on now.  As noted, our multiple datasets include information about a variety of different types of events in convict lives.  Sankey diagrams should be able to help, as they can show multiple paths through several stages, which is what we want to do with convict lives.  Each life history can be a line in a Sankey diagram, which, when 1000s of lives are included, would reveal general patterns.  But how do we manage the large number of events, taking place at different times?  A problem here is that we want to introduce a time element to the variables (the actual dates of events), which makes it too complicated for a normal Sankey diagram.

There is no off-the-peg solution to this problem.  But here is a crude mock up using Excel of what we hope to achieve.  Eventually we will develop visualisations like this using D3, a JavaScript library for producing data visualisations.

Twenty-four convict lives from birth to punishment

Twenty-four convict lives from birth to punishment

This is based on twenty-four convict lives where we currently have eight or more records, including their birth, previous conviction (if any), Old Bailey conviction, and punishment (periods of incarceration in the hulks or a prison and subsequent release, or transportation, or execution).

It is hard to draw conclusions from the rather inelegant presentation, but you can start to see some interesting patterns.  A flat line means little time elapsed, while a steep line connotes a longer period.  We can see how many convicts had previous convictions, and how these often occurred years before the Old Bailey conviction which led to the punishment displayed.  In terms of punishment, we can see significant changes over time in the nineteenth century: crudely a shift from incarceration in the hulks followed by transportation; to prisons followed by transportation; to prisons leading to a prison licence.  What will happen when we replicate this format with tens of thousands of cases?  Will patterns become clearer, or will it just be a mess?

Convict lives by age at which events occurred

Convict lives by age at which events occurred

In fact, this visualization is in some respects already too complicated to interpret easily.  If we remove the date variable and just use the age at which events occurred, it simplifies things.  Here different patterns emerge: the wide age range of previous convictions (many first convictions took place at a young age), the wide age range of those convicted at the Old Bailey; the relatively short time gaps between conviction and commitment on the hulks, and between incarceration on the hulks and transportation (usually); the longer times spent in prison before transportation or licence; and the older ages of those sentenced to prison.

Obviously, this is work in progress, and we have a lot more work to do to create accessible and fine-tuned visualisations providing these types of information, while including thousands more cases. We hope that what we come up with will be of use not only to this project, but also to researchers in other fields who want to create visual representations of vast amounts of complex data in accessible formats.

The evolution of record-keeping as a means of understanding criminality, 1780-1860

Bob Shoemakers’ keynote address from the Penal History in a Digital Age conference in Tasmania, June 2016, focused on the project’s Epistemologies research theme. He asked: Why did they keep such detailed records about criminals?

What makes the Digital Panopticon project possible is the fact that in nineteenth-century Britain and Australia detailed records were kept for the first time about the personal characteristics of convicted criminals.  This information, particularly ages, makes the sophisticated record linkage possible which allows us to compile such substantial life archives for so many convicts.

This was not always the case.  In the eighteenth century, and earlier, we very rarely know anything more about convicts than their names and offences; the one exception is convicts who were executed, whose lives were written up in the wonderfully rich, but religiously framed, Ordinary of Newgate’s Accounts

In contrast, records containing personal information about convicts proliferated after 1780. By 1860 a vast amount of personal data was collected: in addition to their names, offences, verdicts, and sentences, details were recorded about ages, places of birth, occupations, marital status, number of children, and parentage; descriptions of their physical appearance (height, weight, eye and hair colour, ‘build’, marks, and tattoos); whether they could read and write, their education, religion; and previous convictions, character, and behaviour in prison.

Overview of developments in record keeping

Overview of developments in record keeping about criminals, 1778-1842

The table provides a brief overview of changes in record-keeping between 1780 and 1860.  It can be summarised in three main stages:

The first major innovations, in the late eighteenth century, were the recording of ages in the registers of the hulks (following the introduction of this punishment after the cessation of transportation to the American colonies), and the creation of the criminal registers in 1791 by the sheriffs of London and Middlesex.

Home Office Criminal Register, 1791 (HO 26)

Home Office Criminal Register, 1791 (HO 26)

The criminal registers recorded, largely in tabular form, information about the ages, places of birth, occupations, and physical descriptions for each person accused of a crime and committed to Newgate Prison.

Lavishly embellished front page of 1791 Criminal Register

Lavishly embellished front page of 1791 Criminal Register

The elaborate title page in florid handwriting, which proudly promised ‘a particular description of each offender’, indicates the pride taken by the clerk in its creation.

The second stage came with the opening of the first national penitentiary, Millbank Prison, in 1816, whose registers also included information about prisoners’ mental state (their character, behaviour, and religion) and family circumstances (marital status for both sexes, number of children for female prisoners).

Millbank Prison Register

Millbank Prison Register, c.1820

The third stage, in the 1830s and 1840s, saw important new developments in existing record series and the development of new record keeping practices by prison chaplains.  Information was collected about education (literacy skills and degree of previous instruction, particularly in prison chaplain records), and family and friends, and, in the registers of the newly built Pentonville Prison, prisoner weights.

Why did these changes occur? Scholars have offered various theoretical perspectives.  These innovations can be seen as an aspect of the evolution and growth in power of the modern nation state; as part of a society-wide cultural/political initiative to obtain knowledge as a form of control; or as part of growing cultural importance given to ‘facts’. While all these explanations are relevant, we would like to offer a fourth, more crime-focused, reason: that this information collection was the result of a growing desire to understand the criminal. Before developing this argument, what are the limitations of the alternative explanations?

Record-keeping as part of the evolution of the modern nation state works well for Australia, which was run as a military colony for the first few decades (essentially as a prison without walls), so the ‘state’ could easily claim near-absolute power.  But in England a focus on the growing power of the state at a national level, on Parliament and the Home Office, doesn’t explain why so many of the innovations in the collection of information occurred at the local level, by local officials, quasi-officials, and independent social investigators such as Henry Mayhew. The criminal registers, for example, were invented by the sheriffs of London.

And while the creation of the Millbank prison registers was dictated by the statute which authorised the building of the prison, the Rules and Regulations for the prison established by the prison committee mandated the collection of additional information, and the actual registers kept by prison officers went even further.  If anything, this story is about a growth in local state power, not national power, and even then it was often not formally sanctioned by local leaders.

Under the 1823 Gaol Act, prison chaplains were required to produce an annual written ‘statement’ on the condition of the prisoners under their charge and, from 1840, national prison regulations required them to keep a ‘character book’.  There is no evidence that these records were expected to be detailed.  However, in their efforts to reform prisoners (and facilitated by the requirement that they visit prisoners on a regular basis), some chaplains went far beyond these requirements and collected extensive qualitative and quantitative evidence.

John Clay Prison Register

John Clay’s Prison Register Template

John Clay, chaplain to the Preston House of Correction adopted a set of registers in 1839 for recording a vast range of information about those committed to the jail. Although he was the apotheosis of information-gathering chaplains, Clay was not alone: at least twenty other chaplains between 1820 and 1860 also produced detailed reports on their prisoners.

These information gathering activities by local officials were not always welcomed by their superiors.  While some local magistrates and central government officials applauded them, others raised objections, and some reports were even suppressed. So, this doesn’t look like a straightforward example of the growth of national state power.

One could instead see the growth of record keeping as a wider phenomenon, perhaps a society-wide development, in which there was a coming together of local and national institutions to obtain knowledge about the governed, as a means of shaping their conduct.  This is clearly a significant part of the story, particularly in the context of the apparent increasing ungovernability of society from the late 18th century, in the wake of riot, revolution, and rising rates of crime, but the wide range of individuals involved and scale of the information collected makes it difficult to describe this phenomenon solely as a political exercise.

Moreover, we should not exaggerate either the desire of the state to collect this information, or the amount of power this information gave to those in authority. In a perverse way the collection of this information gave some agency to the accused and the criminal, since they were often the only persons in the room who knew their ages, places of birth, occupations, and religion.  At least some prisoners enjoyed their own form of power, reinventing aspects of their identity to, for example, avoid harsher punishments for those with previous convictions or secure more comfortable prison conditions. In 1856 the Governor of Cold Bath Fields prison complained about the large number of convicts who entered his prison with the surname of Smith, thereby creating new identities for themselves so that their previous convictions would not be noticed.

If a political explanation doesn’t work, perhaps we can adopt a more culturally focused one. Bearing in mind that increasing amounts of information were collected at this time about a wider range of issues than just crime and disorder, we could see innovations in record keeping as a result of the growing weight placed on facts, and statistics based on those ‘facts’, throughout society at this time.  As anyone who has spent time in British archives knows, the collection of information on a wide range of topics did expand dramatically at this time, but this doesn’t explain why criminals were some of the first subjects of these new record-keeping practices, alongside two other groups: military recruits and aliens.

So these existing explanations don’t fully explain the how, when and what of the increasing collection of personal information about criminals.  Our explanation is this: these changes were the result of a new moral and empirically-driven desire to better understand the criminal and the causes of crime (and not just control or punish him).

There were, of course, some more mundane and practical reasons why information about criminals was collected: to assist in arresting suspects and apprehending escapees from prisons, and to assist those responsible for reaching sentencing and pardoning decisions, and determining penal regimes for those committed to prison.  But even if we accept that these explanations have some purchase, they do not apply to many types of information collected, nor do they explain why such information was not collected earlier.

Instead, this record-keeping points to the existence of a diffuse and varied, but nonetheless significant information gathering culture among local officials. There was clearly a desire on the part of these officials to understand their custodial subjects and their crimes, but since there was no consensus about the causes of crime in this period there was no agreement about the types of information that needed to be collected, and record-keeping practices varied.

John Clay Statistics

John Clay’s Statistics on Education and Offending

Above is John Clay’s table associating degrees of education with various causes of crime, including drinking, idleness, poverty, etc.  Not being able to read is associated with idleness and ‘confirmed bad habits’ (whatever that means), while being able to read and write is associated with drinking as a cause of crime (a warning we should all perhaps heed).

Rather than merely an example of the growth of state power, innovations in criminal record keeping in the nineteenth century reflect a growing desire among those responsible for dealing with crime at the local level to understand their subjects, and to develop new methods of preventing crime and new methods of reforming offenders.

Conference Notice: Juvenile Justice in Europe: Past, Present and Future

Juvenile Justice in Europe: Past, Present and Future

University of Liverpool, 26-27 May 2016

The conference/symposium is being organized and hosted by the International Criminological Research Unit (ICRU) at the University of Liverpool in association with the British Society of Criminology (Youth Criminology/Youth Justice Network – BSC YC/YJN) and the European Society of Criminology (Thematic Working Group on Juvenile Justice – ESC TWGJJ).

It will address a range of pressing questions relating to the historical origins, contemporary manifestations and future prospects for juvenile justice at a time when Europe is witnessing major social, economic and political challenges and transformations.


2016 marks the 200th anniversary of the first major inquiry into ‘juvenile delinquency’. How has the history of juvenile justice evolved across Europe and how might the past help us to understand the present and signal the future?


What do we know about contemporary juvenile crime trends in Europe and how are nation states responding? Is punitiveness and intolerance eclipsing child welfare and pedagogical imperatives, or is ‘child friendly justice’ holding firm? How might we best understand both the convergent and the divergent patterning of juvenile justice in a changing and reformulating Europe? What impacts are sweeping austerity measures, together with increasing mobilities and migrations, imposing?


What might the future hold for juvenile justice in Europe? How might researchers, policymakers and practitioners shape the future?

It is a crucial time for juvenile justice in Europe and the conference/symposium will comprise a series of plenary presentations delivered by some of Europe’s leading researchers in their respective fields. It will also facilitate ample opportunities for discussion, debate and delegate participation in order to address such questions alongside other past, present and future challenges.

Further details can be found here

UOL ICRUlogo,(3)-246x136 ESC,logo bsc_criminology-283x65

Building Bentham’s Panopticon

This post describes a project that myself and a colleague from the Architecture department at the University of Liverpool, Dr Nick Webb, are currently working on–Building Bentham’s Panopticon– which is creating a 3D model of the Panopticon prison viewed through virtual reality software, Oculus Rift.


Bentham’s Panopticon was imagined as the ‘ideal’ prison; it was designed as a circular building with prisoners’ cells arranged around the outer wall and dominated by an inspection tower. From the tower the prison inspector would be able to gaze upon the prisoners at all times. The central inspection principle, Bentham argued, would result in ‘morals reformed, health preserved, industry invigorated, instruction diffused, public burdens lightened…all by a simple idea in architecture’ (Bentham, 1787).

Due to its escalating cost, his designs were never put in to practice. But the recent digitization of Bentham’s plans by Transcribe Bentham, alongside advances in virtual reality software, means that we now have the opportunity to digitally construct the Panopticon and venture inside.


This small element of the wider Digital Panopticon project seeks to explore how we can use digital technology to examine and recreate alternative ways of seeing and experiencing, in a particular space and place-the Panopticon prison- had it been built. Through the use of 3D modelling and virtual reality technology, we can recreate the perspective, positioning and movements- through sight lines, walking routes, and height and weight records- of the gaolers and prisoners who could have potentially been imprisoned within the walls of the Panopticon.

In doing so, this project takes its inspiration from Tim Hitchcock, who is currently modelling the Old Bailey courtroom, and contends that by, ‘building something in three dimensions, with space, physical form and performance, along with new forms of analysis of text; can change how we understand the experience of imprisonment; allow a more fully empathetic engagement with offenders; along with a better understanding of how their experience impacted on the exercise of power and authority’.[1]

Building Bentham’s Panopticon rests upon two lines of enquiry. Firstly, it seeks to rebuild and re-examine the idealized construction of prison discipline at its most ideological- to examine the beginning of the separate, silent system and the development of modern prison reform through architecture. But it also seeks to contribute to a history from below and examine how, by adding in height and weight records of offenders, we can rebuild the perspectives, movements, and thereby explore the potential for transgression that could have occurred within a prison like the Panopticon.

We are about halfway through our research, and are yet to add in biometric data of prisoners taken from the Digital Panopticon project. Yet, in building the model using SketchUp, we have already begun to discover important findings.


The use of 3D modelling has been essential to visualising Bentham’s process and building the interior of the Panopticon. Bentham’s plans, letters and writings about the Panopticon represent a conversation- between himself, architects, managers, and politicians- that include a series of changes to the design of the building and its regime. We are very early on in our findings, but constructing the Panopticon using 3D software, SketchUp, has demonstrated the significance of using this technology to investigate different lines of historical enquiry. Bentham’s never completed the design for the Panopticon, and the debate continued from the 1780s to 1820s. However, plans exist from 1787 and 1791 and these designs are the source from which we have built the 3D models.  However, the interior was never fully decided upon due to conflicts between, amongst others, Bentham, John Howard, and William Pitt the Younger.[2]As Nick Webb has argued previously, ‘This is important, as inferences have to be made due to representational source data such as architectural drawings almost always being incomplete’.[3] Therefore, it is necessary to delve in to primary and secondary resources to explore the context, and fill in the gaps in an informed way. For example, Bentham initially wanted the Panopticon to be made out of glass and cast iron. ‘Architecturally’, according to Janet Semple, ‘the Panopticon foreshadows Paxton’s Crystal Palace rather than Pentonville’.[4] However, despite technological innovation in glass manufacture in the late eighteenth century, the building materials were never decided upon, so Nick and I decided to use London stock brick as this was the most commonly used material in London at the close of the eighteenth century.

Panopticon build

The models take the form of an idealised, architectural plan, and our current focus is to examine how a series of changes and compromises in the design, seen through the application of 3D modelling, demonstrate the political ideas behind the introduction of the separate, silent system and solitary confinement, but also the relative positions and viewpoints of the different historical actors, in this case, the gaoler, chaplain, and inmates.


What currently interests us at the moment is lines of vision and mobilities as, for Bentham and Foucault, panopticism as a principle is about the power of the gaze- of observation, regulation and power. But I would argue that Foucault and Bentham both had simplistic arguments when it comes to this aspect. In terms of sight lines, or what people can see when stood or walking through a particular point in space, this study builds upon the work of Philip Steadman (UCL). Steadman sketched out two dimensional axonometric drawings of the Panopticon, but with the use of 3D, we are able to build the interior of the Panopticon and therefore provide a space in which the viewer can walk around the prison and inhabit the potential routes of the gaolers, chaplain, and offenders. Steadman draws upon architectural research to plot the totality of what can be seen from a fixed position- also known as an ‘isovist’. (Steadman, 2012: 16).


In Steadman’s image here, you’ll see that the shaded area shows the warder’s isovist. The warder must circulate continuously to watch all the prisoners on his floor. But Steadman’s method, while highlighting the problems in Bentham’s design, is set from a fixed point. Our study builds on this in two crucial ways: firstly, we are able, through the use of Oculus Rift and Virtual Reality Software, to recreate the viewpoints and sight perspective of the gaoler, chaplain, visitors, and offenders, and secondly, we are able to move beyond fixed isovist points to follow the potential mobilities of both gaoler and offender had they been incarcerated in the Panopticon.


So Bentham designed the process of observation to be one way; that is, that the governor, gaolers, other prison staff, and prison visitors to be able to observe the convicts, but that the convicts could only look upon the inspectors gallery. This was, in essence, the central inspection principle. The idea was that every prisoner should be under constant apprehension that he might be observed, night and day, even if no-one was actually looking in his direction at that very moment. He would thus be constantly fearful of being discovered in any misdemeanour.

Screen shot 2012-10-25 at 10.58.43

The Panopticon was a disciplinary technique for making a new social individual; a social laboratory where new subjects were made. Under Bentham’s design, the inmate doesn’t know when they are being watched, and assumes that they are under surveillance at any time. Therefore the prisoner is the subject of observation and power – and this is power through observation. By learning to internalize system of discipline, to watch himself or herself, the Panopticon, theoretically at least, aimed to produce reform through the regulation of the self. The aim of this kind of discipline was, according to Foucault, to turn inmates into quiet, orderly, tractable, malleable subjects or what he provocatively calls ‘Docile Bodies’. As Foucault stated, ‘solitude is the primary condition of total submission’ (Foucault, 1975: 237). Each individual, in his place, is securely confined to a cell from which he is seen from the front by the supervisor; but the side walls prevent him from coming into contact with his companions.

He is seen, but he does not see; he is the object of information, never a subject in communication (Foucault, 1975: 201).The prisoner is therefore, the object of power rather than an agent of power – ‘the object of information’ – never a ‘subject in communication’.


And it is this very notion- the power of the gaze and the power relations that manifest through looking- that Building Bentham’s Panopticon seeks to investigate. The use of 3D and Virtual Reality technology, allows us to put Foucault’s theory, and Bentham’s designs, to the test.


NB Please note that the models are incomplete at present, so may contain errors and inconsistencies.

[1] T. Hitchcock, ‘Voices of Authority: Towards a history from below in patchwork’, [Accessed 22 April 2016].

[2] J. Semple (1993), Bentham’s Prison: A Study of the Panopticon Penitentiary (Oxford University Press: Oxford).

[3]  N. Webb & A. Brown (2016). Digital re-analysis of lost architecture and the particular case of Lutyens׳ Liverpool Metropolitan Cathedral. Frontiers of Architectural Research.

[4] J. Semple (1993), Bentham’s Prison: A Study of the Panopticon Penitentiary (Oxford University Press: Oxford), pp. 116-117.

Open Data and the Digital Panopticon

Of all historical periods and subjects, crime and justice in eighteenth- and nineteenth-century London is the most extensively digitised. Through the digitisation of countless numbers of court records, transportation registers, prison archives, trial reports, criminal biographies, last dying speeches and newspapers (amongst many other things), we can access a wealth of information about crime, policing and punishment in the metropolis, and about the fates of the offenders tried there, all at the click of a mouse.

To our great benefit, much of this data is openly available, a product of the dogged efforts of public bodies, academics, data developers, volunteers and enthusiasts; often (but certainly not always) supported by public funding. In the process it has opened up seemingly boundless possibilities for research.

Indeed, without several of these open datasets the Digital Panopticon could not be realised. In our efforts to trace the life courses and subsequent offending histories of London convicts transported to Australia or imprisoned in Britain in the late eighteenth and nineteenth centuries, we will be reliant on a number of open datasets such as the British Convict Transportation Registers and Female Prison Licences.

It seems timely, therefore, on Open Data Day, to celebrate these fantastic, freely-accessible resources, and to highlight just a couple of ways in which they will be useful to us on the Digital Panopticon. Taking place on 21 February 2015, Open Data Day will involve a series of events and gatherings which seek to develop support for, and to encourage, the adoption of open data policies by the world’s local, regional and national governments.

I have talked in a previous post about the ways in which visualisations of the openly-available British Convict Transportation Registers database can be used to put transportation under the ‘macroscope’ – to chart the complex patterns and interactions of penal transportation in their entirety, spanning the breadth of Australia and the length of a century, taking in the lives of tens of thousands of individuals along the way.

In this post I briefly want to highlight another open dataset which will be at the heart of the project – the prison licence records of females incarcerated in British jails in the nineteenth century, held by the National Archives (under the catalogue reference PCOM 4), the metadata for which is openly available on the Archive’s online catalogue.

The licences almost without exception record the age of the offender on conviction, a potentially useful piece of information for us on the Digital Panopticon in terms of record linkage. But, as with our other datasets, we want to know how accurately ages were recorded, and again in the case of the female licences by visualising the data it suggests some interesting things for us to think about.

Not least, it again reveals the tendency towards age heaping in the recording of ages at round numbers such as 20, 30 and 40, suggesting that recorded ages were regularly rounded up or down rather than representing the true age of the offender. If ages were recorded accurately, we would expect to see a smooth distribution of recorded ages. As seen in the graph below, however, this was far from the case in the recording of female prisoner ages in the nineteenth century, with spikes at the ages of 20, 30, 40 and 50, and dips at the ages 29, 31, 39, 41.

Age on Conviction as Recorded in the PCOM4 Female Prison Licences

Age on Conviction as Recorded in the PCOM4 Female Prison Licences

Does this mean, therefore, that we should disregard recorded ages as entirely inaccurate? Not necessarily – as the graph below demonstrates, when we compare the distribution of ages across different sets of records, it suggests that recorded ages were perhaps broadly reflective of age patterns. The distribution of offender ages is typically younger in the Old Bailey Proceedings (OBP) and in the Convict Indents (CIN – the records of those transported to Australia) compared to that of females imprisoned in Britain (PCOM4) – certainly what we would expect, given the nature of criminal justice policy at the time.

Ages of Female Offenders as Recorded across each Dataset

These are just a couple of ways in which the Digital Panopticon will be drawing upon the wealth of open data available to criminal justice historians. We are indebted to the hard work of all those who have contributed to the creation and dissemination of this embarrassment of riches which, in combination with the powerful digital technologies now at our fingertips, is opening up a whole new realm of research opportunities.


Why collect personal information about criminals if it is not going to be used?

The reason the Digital Panopticon project is able to trace the lives of convicts in such detail is the growing tendency of British local government to collect information about the lives and personal characteristics of criminals.  From the late eighteenth century, and culminating in the registers mandated by the Habitual Criminals Act of 1869, information about places of origin, occupation, physical features, literacy, religion, previous offences, and behaviour while in prison was increasingly collected, leading to entries in the Habitual Criminals Register such as the following:


TNA, MEPO 6/5 (1893), reproduced from David Hawkings, Criminal Ancestors: A Guide to the Historical Criminal Records in England and Wales (Stroud, 2009), p. 164.

We can trace the origins of these registers back to the detailed evidence about accused and convicted criminals kept by Justice of the Peace John Fielding in London from the 1750s to the 1770s (which were destroyed in the 1780 Gordon Riots), and the Criminal Registers, kept first by the City of London and then by the Home Office from 1791, which recorded information about those committed to Newgate Prison in London.


Home Office, Criminal Registers of Prisoners in Middlesex and the City, September 1791-1792, London Lives, 1690-1800, NAHOCR700050004 (, version 1.1, 12 December 2014), National Archives, Ms HO 26/1

For example, the last line of this table is about Benjamin Edmondson, committed in September 1791, ‘aged 16, born in Rosemary Lane, 5 feet 10 [inches tall], sallow complexion, short brown hair, by trade a rope maker’.  Accused of stealing a pocket handkerchief, he was convicted at the Old Bailey and transported for seven years.

It is information like this which makes the Digital Panopticon project possible; and the absence of such information explains why it is impossible to carry out a similar project for an earlier time period.

But if we are to understand the information collected, understand its limitations and use it effectively, it’s important to know why these details were collected in the first place.  The most straightforward and compelling explanation is the desire to take advantage of the increasing power of the state in order to control prisoners. This was what the philosopher Jeremy Bentham intended when he invented the ‘panopticon’ as a form of imprisonment in 1791, and this term was picked up by the philosopher Michel Foucault as a metaphor for the tendency of modern disciplinary societies (or states) to seek total control over their populations.  But when we examine when and by whom this information was collected, and what it was used for, this explanation is not fully convincing.  Too much of this information was collected by low-level officials, with no apparent purpose in mind, and not actually used.

For example, while John Fielding’s primary purpose in collecting information at Bow Street was to keep a record of previous and suspected offenders so that he could make arrests when crimes took place, some of the information collected does not appear to have used.  He appears to have thought that information about previous convictions should influence sentencing decisions, but there is no evidence that the evidence was actually used for that purpose.  Similarly, the information he collected about place of birth, trade and age, and even more remarkably about prisoners’ handwriting abilities, does not appear to have had a clear purpose.

Similarly, while much of the information in the Criminal Registers may have been used to detect recidivists, as in the case of Charlotte Walker in 1798 (sentenced to transportation following numerous previous trials at the Old Bailey), and in sentencing and pardoning decisions (though once again there is little evidence that the information was actually used for this purpose), why did they record information about age and place of birth?  Interestingly, in one of the few instances of record-keeping becoming less systematic, the Home Office stopped collecting much of this information in 1802, perhaps because it wasn’t being used.  (The amount of information collected in the Criminal Registers only expanded again in 1834.)

And why did the clerks decide to keep the information in the Criminal Registers in tabular form? (This is one of the first examples of this method of recording evidence.)  Perhaps this was to encourage clerks to fill in every piece of information required, though as you can see from the example above this did not always happen—often clerks could not be bothered, and there were many blanks.  A tabular form makes statistics easier to compile, but there is no indication that they were used for this purpose (the first official criminal statistics were not compiled until 1805-10, and they only counted crimes, sentences and punishments, not personal characteristics).

When imprisonment became a primary means of punishment in the nineteenth century, prison records began to keep detailed records of all those committed, including not only physical descriptions, past convictions, and records of behaviour in prison, but also in some cases marital status (of men as well as women), number of children, religion, and whether prisoners could read or write.  In the controlled environment of a prison it was easy to collect such information, but once again the purpose of collecting some of this information is unclear.  Information about literacy may have been used to determine the type of instruction the prisoner would receive in prison, but such information started to be kept in the early 1830s, a decade before such instruction was actually introduced.

It is remarkable, therefore, how much information was collected without any clear idea of how it was going to be used.  The intentions of those who created these records are often not clear to us, but perhaps few of the creators knew either. What we may have here is a classic instance of ‘bureaucratic creep’.   Sometimes the form of the records was dictated by statute, and some information had to be kept for very practical reasons, but for the most part record keeping systems were devised by local officials (such as the Sheriffs of London or prison governors), who guessed what types of information might be useful or needed.  A culture of information gathering evolved, in an apparent attempt to understand the criminal and shape punishments, but without a clear sense of how specific types of information would be used.  Perhaps Foucault was right—Britain was becoming a disciplinary society in terms of the amount of information collected—but, unlike the activities of national security agencies today, the state lacked the power (and perhaps the will) to act on such information, let alone compel its collection on a regular basis.  Significantly, as Chris Williams has recently demonstrated, in 1874. a few years following the creation of the Habitual Criminals Register, it ‘was found to be a failure’.

Record Linkage Workshop Report, Part 2

The second half of the workshop was devoted to work in progress from the Digital Panopticon –summaries of which have already appeared (or will soon be appearing) on this blog, so watch this space! As such, I’ll say less about these papers than those from Session 1.

Jamie McLaughlin — ‘How to Disappear Completely: Linking Transportation Records in the Digital Panopticon

Jamie McLaughlin presented some of the insights gained from our recent (and still very early) explorations in linking records of the trial and transportation of convicts in eighteenth- and nineteenth-century London. Uncertainty ‘plagues the records’, and Jamie discussed some of the ways in which we have tried to maximize the quality of the name matches made across the records, such as the use of spelling and date variances, creating control scenarios, and the use of variant lists over general algorithms, all ultimately with an eye on computational performance — an issue which we cannot simply disregard, however much our desire for ‘perfect’ matching techniques. In short, we need to find an optimal, complementary balance of automated and manual work, allowing computers and humans to each do what they’re good at — an ideal strategy reflected in the case of the ‘robot butler’.

Lucy Williams — ‘What’s in a name? Convicts, Context and Multiple Record Linkage

Lucy Williams talked about her recent work in manually checking the automated linkage process undertaken by Jamie, particularly in identifying why good matches have failed to be made. One reason for this is simple name variance — variable spellings of the same surname are notoriously prevalent in eighteenth- and nineteenth-century records. Nor is the data from one record set (such as the Old Bailey Proceedings) carried over consistently to other records. But there is also the problem of “John Smith” — how do we prise apart and correctly link individuals tried at the same session of the Old Bailey who have the same name, spelled in exactly the same way? We can keep adding in information from other sources in order to try and verify these kind of multiple name matches, but that isn’t necessarily always the answer, particularly in terms of automated processes. Adding in all the John Smiths from the census, for instance, can simply lead to even more links. The crucial question for us then is, at what point do we draw a line under things and stop adding in contextual data?

Record Linkage Workshop Report, Part 1

In the first half of this workshop on record linkage we had three fantastic papers from guest speakers who were invited to talk about their own experiences of conducting record linkage in historical research. Each speaker offered a different perspective on the subject, allowing us to think about a wide range of issues relating to record linkage and generating ideas which will be extremely useful to us on the Digital Panopticon.

Jeremy Boulton — ‘Place, Mobility and Class Barriers: The Perils and Possibilities of Nominal Linkage in the Metropolis’

Jeremy Boulton of the University of Newcastle got the event off to a fantastic start with a fascinating and though-provoking window into his self-confessed ‘gruesome fascination’ with nominal record linkage. Reflecting on his experiences as part of the Pauper Lives in Georgian London and Manchester project, Jeremy spoke about the broader methodological (rather than strictly technical) issues associated with record linkage, highlighting both the benefits, but also the inherent dangers, of linking individuals across multiple historical records.

On the one hand, when carried out successfully, nominal record linkage can be an effective means by which to check the accuracy of our historical records. Whilst perfect accuracy is beyond attainment in historical record linkage (as E. A. Wrigley said many years ago, and which still holds true today), nevertheless the creation and collation of successful links allows us to identify the (otherwise imperceptible) lies and concealments of the people being record.

On the other hand, of course, the difficulties associated with nominal record linkage makes the successful creation of links (and thus exposing the ‘fiction in the archives’) a problematic task. Transcription errors (by both the original scribes and present-day transcribers) will defeat even the most sophisticated linkage methodologies, and confirming information can’t always be obtained.

In the latter part of his paper, Jeremy presented an absorbing case-study of the nominal record linkage of Godfrey Sykes, widely documented in sources such as pollbooks, newspapers, the London electoral database and charity subscriber registers — an apparently respectable Georgian businessman who, it turns out from further digging into the historical sources, fathered four bastards with a woman named Ann Farmer.

Gill Newton — ‘Urban Record Linkage before 1754’

Next, Gill Newton of the University of Cambridge shifted the focus onto the nuts and bolts of record linkage — a paper rich in technical detail which provided the audience with a valuable toolkit for undertaking record linkage, even for the particularly challenging context of creating re-constituted families from eighteenth-century London.

Starting with an informative background on the contents of an eighteenth-century parish register and what is meant by a re-constituted family, Gill then noted some of the key challenges which face any researcher looking to undertake urban record linkage. These include a high level of population turnover; rapid growth from migration; blurred parish and administrative boundaries; and a high risk of mistaken identities. There are, however, advantages to linking urban records, such as more detailed registers; a more diverse name base; the ability to sample viably; and the further information generated by civic administration.

Gill then treated us to a fascinating discussion of name distribution in eighteenth-century parish registers. Forenames were heavily bunched around the most common names (John, Mary, Elizabeth etc.). By contrast, whilst some surnames constituted a large proportion of the whole (such as Smith), the distribution of surnames had a much longer ‘tail’ compared to forenames. Moreover, there were stark differences in the patterns of name distribution between rural England and London.

Finally, Gill highlighted some of the most important tools for undertaking nominal record linkage, including phonetic matching and surname dictionary examples, as well as the principles of algorithmic record linkage. She offered some extremely useful tips on how to maximize the quality of the linkages created, emphasising that successful matching requires careful attention and a rigorous methodology — in other words, the cautionary mantra with record linkage should be: ‘garbage in, garbage out’.

Ciara Breathnach — ‘Irish Records Linkage 1864–1913: Big, Macro and Micro Data’

In the final paper of this first session, Ciara Breathnach from the University of Limerick talked about the approach and some of the findings from the Irish Record Linkage 1864–1913 project, on which she is the principal investigator. Funded by the Irish Research Council, and developed in partnership with the Digital Repository of Ireland, University of Limerick and Insight at NUI Galway, the project aims to provide a comprehensive map of infant and maternal mortality for Dublin from 1864 to 1913. The project will reconstruct family units and create longitudinal histories by linking records of Birth, Marriage and Death, which together include millions of name instances.

Starting with an overview of the Irish Record Linkage project, Ciara then discussed some of the forces which served to shape the recording of census and civil data in nineteenth-century Ireland, before moving on to discuss some of the differing definitions of ‘Big Data’, a term about which there is seemingly little agreement.

Ciara also provided useful information on the ontologies utilised by the Irish Record Linkage project, describing the ways in which the data has been analysed and linked, noting the necessity (in the case of such extensive numbers of available records) to sample in order to make such a project feasible.

Finally, through a case-study of Achill in Dublin Ciara presented a glimpse of the significant findings already generated by the Irish Record Linkage project. By mapping infant deaths in the parish in the 1890s, Ciara revealed the nature of the relationship between child mortality and the geography of local health care (in the form of doctors and nurses) in late nineteenth-century Ireland. As Ciara concluded, it is through these kind of detailed micro-level studies, produced by record linkage at the macro level, that we can gain a better understanding of the past.

We are very grateful to all three speakers for providing us with so much food for thought, and so many ideas to follow up!

BCHS4 presentation: Visualising Digital Panopticon Data


The Digital Panopticon will assemble a larger collection of datasets than any other crime history project to date (including, amongst many others, the Old Bailey Proceedings, convict transportation registers and prison records), covering hundreds of thousands of individuals. To effectively bring together this information to reconstruct the lives of offenders, we need to develop a detailed understanding of our datasets – of what information is and isn’t recorded on offenders, and how this varied both over time and across different sets of records. Traditional methods of data analysis and representation such as manual counting and tables are inadequate to this end. This paper instead highlights the power of digital technologies in identifying previously unrecognised (and otherwise unrecognisable) patterns. The techniques of data visualisation in particular have been invaluable in uncovering how extensively, and in what manner, information on offender age, occupation and crime location was recorded within our sources. By using digital technologies to step back from our datasets, and see them in their entirety, we can develop a much fuller and more systematic understanding of the sources we are working with.


Seeing things Differently: Visualising Data on Crime and Punishment

Event: Record Linkage Workshop, Sheffield, 4 November 2014

We’re delighted to be able to announce our second project workshop.

It’s another afternoon workshop, this time in Sheffield, and the subject is Record Linkage (part of the Epistemologies research theme). We’re particularly interested in the challenges and rewards of applying automated (and semi-automated) nominal record linkage to very large-scale historical datasets, with all their variability, fuzziness and uncertainties; our work on the project starts from these questions:

How can we improve current record-linkage processes to maximise both the number of individuals linked across different datasets and the amount of information obtained about each individual? What is the minimum amount of contextual information needed in order to conduct successful large-scale record linkage of data pertaining to specific individuals?

In addition to presentations about our work from project team members, we have three guest speakers who will bring extensive experience of historical record linkage projects:

We think this will add up to a stimulating programme and discussion that will be of interest to many historians who need to link information about large numbers of individuals and using data that is continually growing in diversity and scale.

Download: Workshop Programme/Flyer (pdf).

Workshop Information

When: 2-5.30pm, Tuesday 4 November 2014
Where Humanities Research Institute, Gell Street, Sheffield

Attendance is free but numbers may be limited so you will need to register in advance: email Sharon Howard (