The corpus consists of approximately 8 million PDFs collected from the internet and it will improve security.

JPL data scientist Tim Allison said that PDFs were used everywhere and are essential for contracts, legal documents, 3D engineering designs, and many other purposes.

“Unfortunately, they are complex and can be compromised to hide malicious code or render different information for different users maliciously,” he said.

"To confront these and other challenges from PDFs, a large sample of real-world PDFs needs to be collected from the internet to create a shared, freely available resource for software experts."

Building the corpus was no easy task. As a starting point, Allison's team used Common Crawl, an open-source, public repository of web-crawl data, to identify a wide variety of PDFs to be included in the corpus -- files that are publicly available and not behind firewalls or in private networks.

Conducted between July and August 2021, the crawl identified roughly eight million PDFs.

Common Crawl limits downloaded data to 1 megabyte per file, meaning larger files were incomplete. But researchers need the full PDF, not a truncated version, to conduct meaningful research on them.

The file-size limit reduced the number of complete, untruncated files extracted directly from Common Crawl to 6 million. To get the other 2 million PDFs and ensure the corpus was complete, the JPL team re-fetched the truncated files using specialised software that downloaded the whole files from the incomplete PDFs' web addresses.

Various metadata, such as the software used to create each PDF, was extracted and included with the corpus. The JPL team also relied on free, publicly available geolocation software to identify the server location of the source website for each PDF. The complete data set totals about eight terabytes, making it the most extensive publicly available corpus.

The corpus will do more than help researchers identify threats. Privacy researchers, for example, could study these files to determine how file-creation and editing software can be improved to protect personal information better. Software developers could use the files to find bugs in their code and to check if old versions of the software are still compatible with newer versions of PDFs.