Deep Learning at the National Archives
In September 2014, I walked into the National Archives and Records Administration in College Park, Maryland, with a mandate to work on crowdsourcing tools. Twelve months later, I walked out having built something entirely different: a working prototype that used deep learning to automatically extract metadata from historical documents.
This is the story of that pivot, what I learned about bringing emerging technology into traditional institutions, and why I believe neural networks will transform how we preserve and access our nation’s history.
The Scale of the Problem
Most people know the National Archives as the home of the Declaration of Independence and the Constitution. What they don’t realize is that NARA holds approximately 12 billion pages of textual records, 41 million photographs, hundreds of millions of feet of film, and a growing mountain of electronic records.
Only a small percentage of these holdings are digitized. At current processing rates, it would take well over a century to work through the backlog. Meanwhile, documents are physically degrading. Film is deteriorating. Magnetic tape is demagnetizing. History is literally crumbling while it waits in boxes.
The traditional archival workflow is methodical and meticulous. Records are accessioned, arranged, described, preserved, and eventually, maybe, digitized and made accessible online. Each step requires trained archivists applying professional judgment. It’s careful work, and it produces high-quality results. But it cannot scale to meet the volume.
Why Crowdsourcing Wasn’t the Answer (Yet)
My co-fellow Ashley Jablow and I were originally tasked with exploring crowdsourcing, enlisting the public to help tag, transcribe, and index records. It’s a proven approach; organizations like the Smithsonian and FamilySearch have built successful volunteer programs.
But after two months of intensive research, 31 interviews, site tours, and countless coffees with archivists across the agency, we reached a different conclusion. NARA wasn’t structurally or technologically ready for successful crowdsourcing at scale. The internal systems didn’t talk to each other. Workflows were fragmented. There were significant cultural and organizational barriers to overcome first.
We also kept hearing the same refrain from staff: technology at NARA felt like an obstacle, not a help. Systems were outdated. Automation was essentially nonexistent. Archivists were spending their expertise on manual data entry rather than the interpretive work they were trained for.
What if we could change that equation?
The Pivot to Deep Learning
In late 2014, I started exploring a different question: what if machines could handle the initial, tedious stages of document processing? Not replacing archivists, but augmenting them. Taking the grunt work off their plates so they could focus on the nuanced, contextual analysis that actually requires human expertise.
The timing was good. Deep learning had just achieved breakthrough results in computer vision. In 2012, AlexNet had stunned the research community by crushing the ImageNet competition with a convolutional neural network, cutting the error rate nearly in half. By 2014, Google’s GoogLeNet had driven error rates down to 6.67%, approaching human-level performance on image classification.
More relevantly, Google had just published a paper showing they could use neural networks to transcribe house numbers from Street View images with over 96% accuracy. They processed every address in France in under an hour. That got my attention. If neural networks could read messy, inconsistent street numbers captured from moving vehicles, what could they do with scanned documents?
Building the Prototype
The technical landscape in 2014 was challenging. TensorFlow didn’t exist yet. Neither did Keras. The main options were Caffe (released December 2013), Theano (a Python library with a steep learning curve), and Torch7 (which required knowing Lua). There were no high-level APIs, no pre-trained models you could fine-tune in an afternoon.
I built the prototype using Caffe for the neural network components, combined with traditional OCR tools and natural language processing for text analysis. The system had several stages:
Document Ingestion: Convert any file format into a standardized image format while preserving the original document’s appearance and formatting.
Text Extraction: Apply optical character recognition to convert images into machine-readable text. For printed documents, this worked reasonably well. Handwritten documents remained a significant challenge.
Content Analysis: Here’s where the deep learning came in. The system analyzed extracted text to identify:
- Named entities (people, places, organizations, dates)
- Keywords and topics
- Document categories and types
- Taxonomic classifications
Image Analysis: This was where the prototype really showed its potential. NARA holds 41 million photographs, and describing them manually is even more labor-intensive than processing text. We used convolutional neural networks to identify objects, scenes, people, and activities in photographs. A photo that an archivist might describe as “military personnel, circa 1944” could be automatically tagged with: soldiers, uniforms, jeep, tents, forest, rifles, and dozens of other objects visible in the frame. We tested the system on portions of NARA’s photographic holdings, and the results were promising. Objects that would never make it into a manual description became searchable.
Search Interface: All extracted metadata fed into an index that enabled search across the analyzed records, search that went far beyond the manually-created archival descriptions.
The key insight was that neural networks could dramatically expand the metadata attached to any single record. An archivist might describe a collection at the box level or folder level. The automated system could analyze every individual document and photograph, extracting dozens of potential access points: names, dates, topics, objects, scenes, faces. Information that would never have been captured manually.
Demonstrating the Possible
By summer 2015, the prototype was ready to demonstrate internally. We showed it to teams across the agency: archivists, the digitization division, the Office of Innovation. We presented to the Deputy Archivist and ultimately to David Ferriero, the Archivist of the United States, and the Executive Leadership Team.
The reactions were mixed.
Some people immediately grasped the potential. They could see how this technology might finally offer a path through the backlog problem. They understood that we weren’t proposing to replace archivists but to give them superpowers, letting machines handle initial processing so humans could focus on higher-value work.
Others were skeptical, even concerned. Archival practice has developed over decades for good reasons. Metadata quality matters. Context matters. A machine that misidentifies a document or extracts incorrect information could mislead researchers. The professional standards archivists uphold aren’t arbitrary bureaucracy; they’re essential to maintaining the integrity of the historical record.
And some, frankly, saw it as a threat. If machines could do this work, what would happen to archival positions? These concerns weren’t unfounded. Technology-driven displacement is a real phenomenon. Though I’d argue that augmentation, not replacement, was always the goal.
What I Learned
Traditional institutions move slowly for reasons. NARA isn’t a startup. It can’t “move fast and break things” because the things in question are irreplaceable historical records. The caution I encountered wasn’t obstruction; it was institutional responsibility. Any technology deployed at NARA needs to be reliable, maintainable, and aligned with archival principles.
Technology is the easy part. Building the prototype took months. Getting organizational buy-in would take years. The real barriers weren’t technical. They were cultural, budgetary, and political. Someone with the right vision and authority will need to champion this kind of transformation.
“Good enough” beats “perfect” for initial access. This was controversial, but I believe it firmly: getting documents online quickly with 80% accurate automated metadata is better than keeping them in boxes for decades while waiting for resources to process them “properly.” Initial automated processing could be followed by human review and correction. The public would have access years or decades sooner.
The training data problem is solvable. Deep learning requires large datasets. NARA has something most organizations don’t: decades of human-generated archival descriptions that could serve as training data. The Citizen Archivist program produces human transcriptions that could train handwriting recognition models. The ingredients exist.
What Comes Next
I’m not naive about the timeline. Government technology moves slowly. Budgets are constrained. NARA’s funding has been essentially flat in real dollars for decades. The agency is struggling to maintain basic operations, let alone invest in advanced AI research.
But the technology will only improve. Neural networks are getting better, faster, and more accessible every year. What required a PhD and a GPU cluster in 2014 will be available to any developer with a laptop in a few years. The question isn’t whether these techniques will transform archives. It’s when, and whether American archives will lead or follow.
My hope is that NARA will continue exploring these approaches. The prototype I built was a proof of concept, not a production system. But it demonstrated that the core idea works: machines can extract meaningful information from historical documents at scale.
Imagine a future where every document in the National Archives is digitized, indexed, and searchable. Where a researcher looking for mentions of their great-grandmother doesn’t need to know which record group to search. They can simply type a name and find every instance across billions of pages. Where connections between documents that no archivist had time to identify emerge automatically through computational analysis.
That future is technically possible today. The only question is whether we’ll build it.
I served as a Presidential Innovation Fellow at the National Archives from September 2014 to September 2015. I’m grateful to Ashley Jablow, my co-fellow; Pamela Wright and the Office of Innovation team; and the hundreds of NARA staff who shared their time and insights with us.