61.1M
61M
Dec 19, 2017
12/17
by
Internet Archive Web Group
A series of open web crawls targeting journal articles, technical memos, essays, datasets, and other research publications. This collection contains WARC and CDX files that end up in Wayback ( https://web.archive.org ). See also bibliographic metadata corpuses at https://archive.org/details/ia_biblio_metadata
Crawls performed by Internet Archive on behalf of the National Library of Australia. This data is currently not publicly accessible.
WARCs from internal crawl testing.
Topics: web, cctld
Data collected by Internet Archive on behalf of the National Library of Spain. This data is currently not publicly accessible.
This crawl was performed in Summer & Fall of 2012 to archive the US Federal Elections.
Topics: US, federal, elections, web, 2012
This collection contains collaborative Election crawls performed by IA.
Topics: elections, web
Crawls of the french domain space performed by Internet Archive on behalf of Bibliotheque Nationale de France. This data is currently not publicly accessible.
National Archives and Records Administration crawl performed by Internet Archive. This data is currently not publicly accessible.
Data collected by Internet Archive on behalf of the National Library of Israel. This data is currently not publicly accessible.
Topic: nlil
National Library of Luxembourg
Topic: Luxembourg
National Library of Austrailia crawl. This data is currently not publicly accessible.
2.8M
2.8M
May 28, 2020
05/20
by
Internet Archive Web Group
this data is currently not publicly accessible.
7.9M
7.9M
Aug 4, 2017
08/17
by
Internet Archive Web Group
Microsoft Academic Graph public corpus (Feb 2016) PDF URLs, filtered to remove large sites (pubmed, citeseerx, arxiv) and already-crawled URLs.
Topics: papers, journals
These crawls of the .es domain were performed in 2011 on behalf of the National Library of Spain (BNE).
Topics: bne, spain, web, 2011
6.7M
6.7M
Jul 6, 2020
07/20
by
Internet Archive Web Group
This collection includes all collaborative Olympic crawls performed by IA for the IIPC.
Topics: olympics, IIPC, web
Topics: bne, spain, web, 2013
Domain crawl of the Australian web domain (.au) performed by Internet Archive on behalf of the National Library of Australia in March-April, 2021.
Topic: crawldata
this data is currently not publicly accessible.
Domain crawl of the New Zealand web domain (.nz) performed by Internet Archive on behalf of the National Library of New Zealand in January-March, 2021.
Topic: crawldata
This crawl of the .nz domain was performed on behalf of the National Library of New Zealand in Spring of 2017.
Topics: nlnz, web, 2017
Crawls performed by the Internet Archive in 2017 on behalf of the National Library of Australia.
Topic: nla web 2017
Domain crawl of the Israel web domain (.il) performed by Internet Archive in October-November 2021 on behalf of the National Library of Israel.
Topic: crawldata
this data is currently not publicly accessible.
Domain crawl of the New Zealand web domain (.nz) performed by Internet Archive on behalf of the National Library of New Zealand in January-February, 2018.
Topics: web, nlnz, 2018
This crawl of the .au domain was performed on behalf of the National Library of Australia in of 2016.
Topics: nla, australia, web
9.1M
9.1M
Jul 17, 2018
07/18
by
Internet Archive Web Group
Web archive data from a crawl of open access PDF URLs provided by Unpaywall.
This crawl of the .au domain was performed on behalf of the National Library of Australia in of 2015.
Topics: nla, web, 2015
These crawls were performed by IA on behalf of the IIPC in Summer 2012 during and prior to the 2012 Summer Olympics held in London, UK.
Topics: London, olympics, web, 2012, IIPC
Crawls performed by Internet Archive on behalf of the National Library of Ireland. This data is currently not publicly accessible.
ccTLD crawl for .br domain
Topics: br, web, 2018, cctld
2015 crawl of museum websites listed in the IMLS Museum Universe Data File. More about the IMLS MUDF can be found at https://www.imls.gov/research-evaluation/data-collection/museum-universe-data-file
Topic: AIT
This crawl of the .es domain was performed in 2012 on behalf of the National Library of Spain (BNE).
Topics: bne, spain, web, 2012
7.9M
7.9M
Apr 9, 2018
04/18
by
Internet Archive Web Group
015-2021-winter domain crawl of the Luxembourg web domain (.lu) performed by Internet Archive December 2021 - January 2022 on behalf of the National Library of Luxembourg / Bibliothèque nationale de Luxembourg.
Topic: crawldata
Crawls performed by the Internet Archive in 2018 on behalf of the National Library of Australia.
Topics: nla, web, 2018
This crawl of online resources of the 114th US Congress was performed on behalf of The United States National Archives & Records Administration (NARA).
This crawl of the .au domain was performed on behalf of the National Library of Australia in of 2014.
Topics: nla, web, 2014
This collection includes content harvested from the Web on behalf of the National Library & Archives New Zealand in January 2016.
Topics: new zealand, web, domain
This crawl of online resources of the 112th US Congress was performed in Fall of 2012 and early winter of 2013 on behalf of NARA.
Topics: nara, 112th, web
This collection includes content harvested from the Web on behalf of the National Library & Archives New Zealand in January 2015.
Topics: new zealand, web, domain
this data is currently not publicly accessible.
Crawls performed by the Internet Archive in 2019 on behalf of the National Library of Australia.
Topics: nla, web, 2019
2.3M
2.3M
Jan 24, 2020
01/20
by
Internet Archive Web Group
This crawl of the .au domain was performed on behalf of the National Library of Australia in Spring of 2013.
Topics: nla, web, 2013
this data is currently not publicly accessible.
Data collected by Internet Archive on behalf of the National Library of Sweden. This data is currently not publicly accessible.
Crawl 00001 of the IMLS Museum Universe Date File.
this data is currently not publicly accessible.
this data is currently not publicly accessible.
2M
2.0M
Apr 26, 2019
04/19
by
Internet Archive Web Group
Crawls performed by the Internet Archive in 2020 on behalf of the National Library of Australia.
Topics: nla, web, 2020
2017 domain crawl for National Library of Ireland.
Topics: ireland, web
KB Curated List Crawl 2019. This data is not currently publicly accessible.
Topic: web
This collection includes content harvested from the Web on behalf of the National Library & Archives New Zealand in February 2013.
Topics: web, domain
780,566
781K
Oct 31, 2018
10/18
by
Internet Archive Web Group
Crawl of "upstream" URLs from CORE (core.ac.uk) metadata dump. Only a partial seedlist of files crawled.
this data is currently not publicly accessible.
2.1M
2.1M
Mar 5, 2020
03/20
by
Internet Archive Web Group
this data is currently not publicly accessible.
1.9M
1.9M
Feb 15, 2019
02/19
by
Internet Archive Web Group
Domain crawl of the New Zealand web domain (.nz) performed by Internet Archive on behalf of the National Library of New Zealand in January-February, 2019.
Topics: web, nlnz, 2019
this data is currently not publicly accessible.
24.8M
25M
Oct 3, 2013
10/13
by
dominic@archive.org
This crawl of the .il domain was performed in 2013 on behalf of the National Library of Israel (NLIL).
Topics: nlil, israel, web, 2013
Crawls performed by the Internet Archive in 2020 on behalf of the National Library of Israel .
Topic: web
This crawl of online resources of the 115th US Congress was performed on behalf of The United States National Archives & Records
Topic: crawldata
1.8M
1.8M
Sep 21, 2017
09/17
by
Internet Archive Web Group
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; line-height: 17.0px; font: 12.8px Menlo; color: #161516; background-color: #ffffff} span.s1 {font-kerning: none} WARCS from Whole Earth Web Archive (WEWA) Domain Crawls
Topic: web
this data is currently not publicly accessible.
this data is currently not publicly accessible.
1.9M
1.9M
Feb 5, 2020
02/20
by
Internet Archive Web Group
Data collected by Internet Archive on behalf of Biblioteca Nazionale Centrale di Firenze. This data is currently not publicly accessible.
These are crawls performed on US Federal Government Web sites prior to their removal or merge with other resources.
Topics: federal, web, closures
This crawl was performed in Fall of 2011 to archive Federal government web sites that were either slated for removal or for merger with other online resources.
Topics: federal, web, 2011
