Chronicling America: Historic American Newspapers, 1836-1922

Yesterday we had a visit from Deborah Thomas from the National Digital Newspaper Program at the Library of Congress, who gave a presentation on the Library's Chronicling America website. I thought I'd write up my notes from her presentation here for anyone who was unable to attend and also for future reference.

About Chronicling America and the National Digital Newspaper Program

Chronicling America (http://chroniclingamerica.loc.gov) is a website that provides access to digitised pages of selected American newspapers from 1836-1922, and is hosted and maintained by the Library of Congress as part of the National Digital Newspaper Program. This program is a partnership between the Library and the National Endowment for the Humanities and has been running since 2005. It funds state projects to digitise historic newspapers from this time period (each grant funds the digitisation of 100,000 pages), which are then made available through the Chronicling America site. The program builds on the earlier United States Newspaper Program, which inventoried, catalogued and microfilmed some 75 million pages from 140,000 historic American newspapers over a period of 25 years. The current programme aims to make a representative sample of those pages available online - right now there are almost 4.3 million pages from 25 states (and the District of Columbia), and it is hoped that eventually (within 15-20 years) all states will be covered.  The latest three states to join in are Indiana, North Dakota, and West Virginia, and their content will start to be made available next Spring.  This state-specific approach is driven by the nature of newspaper publishing and newspaper collections in the United States - local institutions have the most comprehensive collections for their area, and there is no single national US newspaper collection as in many other countries. However, aggregating the content in the website at a national level provides a much better and more accessible online service to researchers, and this is one of the main goals of the project.

Each state is responsible for selecting the newspapers and editions that they will digitise, and so the type of content that you will find within the site varies from state to state. Some have picked long runs of a few large papers; others have picked a lot of smaller titles - Kentucky, for example, wanted to ensure that each of their eighty counties were represented in their sample.  As part of the process, the contributing institutions were asked to write short essays for each newspaper title that they digitised, explaining why they chose that title and its significance and context - these essays are all available as part of the bibliographic record for the title within the US Newspaper Directory (of which more later in this post), so you can see the rationale behind what could otherwise seem like a fairly arbitrary sample.

The material being digitised is all within the date range of 1836-1922.  The end date is determined by US copyright law, whose threshold is 1923 - all content from before that date is in the public domain and may therefore be freely digitised and made available by the Library in this way, whereas anything more recent would require the permission of a complicated network of rights-holders.  This does mean that you are able to download, copy and re-use any of the content from the site entirely freely yourself!  The start date is rather more arbitrary, but was chosen because of the increasing difficulty of digitising the older material and because the format of newspapers changes and is less consistent the further back in time you go.

Searching the site

The Chronicling America interface provides several different routes in to access the content. On the main page you will find a search bar which defaults to the basic full-text search - this allows you to search for terms and restrict your search to a specific state and/or date range. The search box here acts as a proximity search, so if you put more than one term in it will search for instances where those terms appear within five words of each other.  The digitised pages have been converted for full-text search using Optical Character Recognition (OCR), which can have a variable success rate - it's less than 100%, and in some instances can even be less than 50% accurate. However chances are if you are searching for significant terms, those words will turn up more than once in an article which does compensate for this a little!

The advanced search gives you more options to construct your search terms, and allows you to restrict by more than one state and/or newspaper title as well as enter date ranges by day and month as well as year.

 

The next tab (All Digitized Newspapers) is the means by which you can browse the collection, either in its entirety, or by state, ethnicity, or language.  The newspapers that are included are almost entirely English-language, though there are two in French, two Hawaiian, three Spanish and one Choctaw title and more French and Spanish content will be added later this year. 

Once you have found some results, the page image interface is really nice and smooth, with scrolling zoom and click and drag, and really high quality scans. You can navigate through the full edition of that paper, and view or download page images as text and PDF (with the bibliographic information attached, ready for referencing).  You can also 'clip' images to get an image of part of a page - the 'clip image' button will essentially cut out whatever is visible in the viewer at that moment.  All the page images, and clipped images that you make, have persistent URLs, which means you can bookmark or save the link and you will always be able to get back to it.  As an example, this link will take you to the following clipped image:

US Newspaper Directory, 1690-


As well as the full-text material in Chronicling America, the site also provides access to the full US Newspaper Directory.  This contains bibliographic information about all 140,000 titles that were catalogued as part of the United States Newspaper Program, along with information about which libraries hold them. These bibliographic records also include links to full-text digitised versions both within Chronicling America and on other sites where available.  Quite often if something is in Chronicling America it will also be available online from the institution which scanned it in the first place, so you are likely to find some titles duplicated.  For the titles that are included in Chronicling America, this is where you will find the contextual essays written by the people that selected the title for inclusion.

Topics in Chronicling America

The Library of Congress also provides around 100 topic guides to the site, which are designed to give you starting points in researching a whole variety of events, subjects and themes within Chronicling America. The topics available are quite a random selection, ranging from Presidential administrations and elections, to events such as the Haymarket Affair or the Annexation of Hawaii, and people like Booker T. Washington and Nikola Tesla, as well as all sorts of other things like the ping-pong craze of 1900 to 1902.  Each topical guide gives you a bit of the background as well as links to sample articles and (perhaps most usefully), suggested search terms to use. You can see a full list of the topics at http://www.loc.gov/rr/news/topics/topics.html, but check back again in the future as more are being created and added.

Keeping up-to-date with the site

Chronicling America is growing rapidly and continually, so it's worth going back again and again to see if new material has been added that is relevant to your research.  The site also provides you with ways to keep up with the new content as it becomes available and to be alerted when there are additions.  You can subscribe to general updates (of content as well as points of interest and research) both by RSS and email, or subscribe to an RSS feed of just the new content as it is added, from the subscribe link available throughout the site.

 

And finally...

All the data from the site is freely available for re-use and can be obtained via the site's API. To see an example of one of the innovative uses of the data from the US Newspaper Directory, see Stanford University's data visualisation of the growth of newspapers across the United States from 1690-2011.  

Introduction to the US Congressional Serial Set

At last, the long-promised introduction to the US Congressional Serial Set, one of our major resources.

What is the Serial Set?

The US Congressional Serial Set is the official collection of reports and documents of Congress. It began with the 15th Congress in 1817; older documents (from 1789-) can be found in the American State Papers. It is as important to get to grips what the Serial Set is not as what it is, as it is very specific as to which documents/reports are included and which are published separately.

The Serial Set contains the following named categories of documents:

• House & Senate Reports
• House & Senate Documents
• House & Senate Journals (until the 82nd Congress in 1952)
• Senate Executive Reports and Treaty Documents (from the 95th Congress in 1977 onwards). 

It does not include the text of debates (for which, see the Congressional Record), bills & resolutions, Committee Hearings or Prints, or other miscellaneous House or Senate documents.
House & Senate Reports are the official communications of the House & Senate Committees to the full House & Senate. They are mostly legislative reports, but also include special reports on various subjects. They are numbered as H.Rpt. or S.Rpt., so if you see a reference beginning with either of these terms you will definitely be able to find it in the Serial Set.

House & Senate Documents are less closely linked to legislation. Some examples of the type of publication that would be classed as a House or Senate Document are as follows:

• Congress's rules of operation
• Memorials, ceremonial reports
• Results of investigations (including occasionally transcripts of hearings)
• Executive Department materials (annual reports, special reports)
• Materials from outside the Federal Government which were of use to Congress

House & Senate Documents are numbered H.Doc. and S.Doc. One exception to be aware of is that from the 30th to the 53rd Congresses (1847-1895), Documents were further split into Miscellaneous Documents and Executive Documents, with the latter exclusively reserved for documents which came from the Executive Branch. Be careful not to mix these up with the entirely different Senate Executive Reports which are included in the Serial Set after 1977!

The House & Senate Journals are the daily record of business of each chamber. Unlike the Congressional Record they don't record the debates verbatim, but you may find excerpts of speeches included on occasion. Like the Congressional Record, they also do not include the text of bills or resolutions.

Overall, the Serial Set is a vast collection of hundreds of thousands of documents, including thousands of maps and illustrations, covering every conceivable topic and both US and worldwide events - anything Congress had an interest in. Whatever you're researching, you're sure to find something of relevance in the Serial Set, but do be aware of the limitations of the set in terms of the types of documents included.

American State Papers

The American State Papers contain legislative and executive documents of Congress from 1789 to 1838. Documents within the American State Papers are arranged into ten topical classes, as follows:

I. Foreign relations
II. Indian affairs
III. Finances
IV. Commerce and navigation
V. Military affairs
VI. Naval affairs
VII. Post Office Department
VIII. Public lands
IX. Claims
X. Miscellaneous

How to access the Serial Set

We have access to the electronic version of the Serial Set from Readex for 1817-1994, which is available via SOLO/OxLIP+.  The easiest way to find it is to search for 'Congressional Serial Set' - the database you want is entitled US Congressional Serial Set, 1817-1994 [in progress] (with American State Papers, 1789-1838).  There is also the American State Papers (with US Congressional Serial Set) and US Congressional Serial Set Maps, which allow you to search those specifically, but if you're after access to the main thing, the option highlighted below is the one you want.

 

If you want to find documents in the Serial Set post-1994, they are freely available (and full-text searchable) via the GPO's Federal Digital System site - links as follows:

• Congressional Documents
• Congressional Reports

The VHL also has an extensive run of Serial Set volumes from the 19th century (up to approximately 1880) in print down in the stack, as well as the printed volumes of the American State Papers. 

Searching the Readex Serial Set

The Readex interface offers several different ways to search or browse the Serial Set. I will concentrate here on the search and browse options available for the main Serial Set, but if you are searching the American State Papers via the Readex interface you will find it works in basically the same way.

The easiest way by far is if you know the exact document you are looking for, as you can then enter the reference in the publication search and it will take you straight to it.  However, in most cases you won't necessarily know this, and will be wanting to carry out more speculative keyword searches.

When you click through from SOLO or OxLIP+, you will be taken straight to the advanced search screen. Here you can enter your search terms either in the top two boxes (which search the citations), and/or the third box down, which is the full-text search box. You can specify the Serial Set volume if you know it, and add a date range in the last box.  This latter is particularly useful as the Serial Set covers such a long time period.

(Note the links to the publication search and also the bill number search at the bottom)

As the Serial Set is such a large database, you may well find that you get back a lot of results, and it can be difficult to work out what terms to put in in the first place to bring back a manageable number. One really good feature of this database though is the ways in which it allows you to filter after you have carried out an intitial search. Once you have a set of results in the lower part of the page, you will still have your search boxes available at the top, but now you have the option to carry out further searches within your results set only:

This allows you to be quite speculative and broad in your initial search terms, and then throw more terms in in subsequent searches (and the full text is particularly helpful at this point I find) in order to reduce your results set to something more manageable and focused.

One thing to be aware of is the 'also consider these topics' feature, which you will see on the right-hand side of your results list. This lists various associated subject/person/geographic etc terms which you can click on, but if you do click on one of these it will not narrow your search, but rather broaden it to show you all documents that are linked to that term. If you find you've carried out too narrow a search, or what you've come back with is not actually useful to you, this can be a good way to broaden things out again, but don't click on these terms expecting it to filter your existing results!

As well as search options, there is a whole variety of ways to browse the Serial Set. At the bottom of the main search screen you will see a set of tabs, which allow you to browse by a huge number of categories.  Within each tab there are subcategories, so that you can drill down to find the term you want and then the documents that are associated with that subject/person/organisation/committee etc. I find that sometimes these can be very useful, but for other things they are just too broad.  If you are wanting to find a specific person, or documents relating to a specific Act of Congress though, then this can be easier than conducting a keyword search. And as with searches, once you have selected a category you will find that you then have the option to conduct a search just within that category, which can again be a very useful way to start off broadly and narrow down.

Finally, it is also possible to search the Serial Set for documents relating to specific bills and resolutions if you know the bill number. To do this, select the bill number search from the options below the search boxes.

Working with documents

Once you have found the document you are after, there are several things that you can do. The documents are all scanned page images which you can zoom into and move around on the page (particularly useful for maps). You can download pages either individually or in bulk, as JPGs (single pages only) or PDFs.  Each document has a full citation, which you can view, download or print, and you can save documents to a temporary collection and then email yourself the links and references to come back to later.  There is also a full-text search box at the top of the document view page which allows you to search within that document for specific terms, which is particularly useful for lengthy reports. 

Other ways to access the Serial Set

As well as searching the Readex database, we do have a set of printed indexes to the Serial Set, which cover 1817-1969 (as well as the American State Papers from 1789-).  They are available in the reference section on the ground floor, and if you use these they will give you the exact publication reference which you can then use in the publication search in the electronic version. Another useful reference is an online guide to the Serial Set which allows you to browse by agency to find publication and volume numbers.

With thanks to August Imholtz for providing me with some of the descriptions used above.

Follow me to visit libraries and archives in the USA

(Cross-posted to the main VHL blog)

In April, I was selected as the 2011 recipient of the CILIP/ESU Travelling Librarian Award, which supports a UK librarian to travel to the United States to visit libraries and archives. The award is designed to promote and develop UK-US library links, and is an excellent opportunity for me both to learn more about historic library collections in the US and to share information on the American collections at Oxford with librarians in the United States.

I'll be flying out to the US one month today, on 17th September, and will be blogging my trip at http://travellinglibrarian2011.blogspot.com. I'll also be tweeting as @jlrawson. You can view my itinerary on the blog at http://travellinglibrarian2011.blogspot.com/p/itinerary.html - I'll be visiting a variety of libraries and archives with significant historical collections, all in the east coast area. Follow me as I travel and gain an insight into the places that I visit!  If you have things you'd like to know, or questions you'd like me to ask while I'm there, feel free to let me know.

Declassified Documents: DDRS and DNSA

In the week that the full Pentagon Papers were released online, and following the announcement of a new collection added to the Digital National Security Archive, a post on finding and accessing declassified documents seemed appropriate!

Oxford has access to two online databases of declassified documents, the aforementioned Digital National Security Archive and also the Declassified Documents Reference System, both available via OxLIP+.  These two databases are complementary in that they both aim to do the same thing but have a slightly different approach. 

The Declassified Documents Reference System (DDRS) provides access to over 500,000 pages of declassified government documents. The bulk of the documents (as advertised) cover 1945 to the 1970s, however I have found documents there from as early as 1910 and as late as 1992.  They originate from many government departments and organisations: the CIA, FBI, State, Justice and Defense Departments, National Security Council and the White House among others. It includes all kinds of different materials - memos, cables, correspondence, studies and reports, and covers nearly every major domestic and international event during the period.

The Digital National Security Archive (DNSA) also provides access to over 500,000 pages of declassified documents, starting in 1945 and going right up to almost the present day for certain topics.  The main way in which this resource differs from DDRS is that it is organised thematically, with documents arranged into collections on specific areas or events such as Afghanistan, the Cuban Missile Crisis, the Vietnam War, or US military uses of space.  New collections are added relatively frequently, so even if your area of interest is not currently covered it is worth checking back periodically.   As with DDRS, DNSA includes a diverse range of document types and sources.

How to use the database: DDRS

 When you click through to DDRS from OxLIP+, you are brought to the basic search screen, which allows you to do keyword or full-text searches of the entire database. With so many documents to search, however, it can be difficult to enter search terms that are going to bring you back a useful (and not overwhelming) set of results.  I'd generally advise therefore going straight to the advanced search option, which gives you a lot of ways to narrow your search. 

At the top of the page you have the usual options to enter several search terms. Note that the default is to search keyword/subject, ie, the citation and not the full-text.  If you want to do a full-text search, you need to select full-text from the drop down menu.

Below the search boxes there are a whole lot of different ways to limit your search and ensure that you get really relevant results. The most useful one is the issue date, which means the date of the actual document itself and allows you to restrict your search to specific time periods.  Don't confuse this with the 'date declassified', which will in most cases be much later.  Another useful limiter is the source institution, which allows you only to search for documents issuing from a particular agency if, for example, you're interested in just the activity of the CIA or the White House etc. You can also restrict your search by document type, sanitization, completeness or number of pages.

The documents themselves can be viewed either as page images or as transcribed text. Page images is the default; to view the text, click on 'view text' next to the page navigation at the top.  What you can do with the document depends on which view you are in.  If you are viewing the page image, then you can download/print the entire document as a PDF.  If you are viewing the text, then you can print or email the text of the page you are viewing (but only that one page at a time).  DDRS also offers a fairly basic 'mark records' function, where you can create a list of specific records from your various searches as you go along to come back to. However, this is session-bound, so you can't come back to it after exiting the database, and there is also no save/export function other than printing out the screen from your browser!  Within your session, it is also possible to view your search history and get back to earlier sets of results.

How to use the database: DNSA

As described above, documents within the DNSA are arranged into thematic collections, which can be accessed individually or in conjunction with one another.  Each collection also has a guide available, which provides lots of information about what it includes and any limitations, and are well worth taking a look at before you dive in to your research.

Depending on what you are looking for, there are several different ways to search the DNSA database.

Searching documents: Note that unlike DDRS, it is not possible to search the full-text of documents themselves. 

• Quick search: From the box in the top left-hand corner. This performs a basic keyword search across all collections and is therefore likely to bring back a large number of hits.

• Document search: This is the most useful search option for finding material. It offers advanced search options (keyword, name, date, document type) and the ability to select which collections you want to search - scroll down to find the tick boxes at the bottom of the page. You can therefore search only one, or a selected few, or the entire database as you choose and is the real way in which the thematic approach is beneficial as you're only searching relevant documents from the off.

Background searches: These do not search the documents themselves, but provide background information and context to the documents.

• Bibliography search: This searches the bibliography of materials used by the researchers at the National Security Archive when compiling the collections. It can therefore be useful as a background bibliography of the subject area, and will point you on to relevant books and articles, but is not designed to be comprehensive. 
• Chronology search: This gives you a useful summary of key events pertaining to your search.
• Glossary search: Allows you to search for basic descriptive information about significant individuals, terms and acronyms.

Documents can be viewed as page images or downloaded as PDFs, as with DDRS. However, DNSA also allows you to export citations both as plain text and directly into RefWorks and EndNote as well as print and email.  Another excellent feature is the cross-referencing, which allows you to click through to find further documents for people, organisations and subjects listed in any record.  As well as tracking your search history allowing you to mark records to come back to within a session, DNSA offers a 'my archive' feature where you can sign up for an account and save records and documents to come back to another time.

New electronic resource: The American Founding Era Collection (Rotunda)

We have just purchased permanent access to the American Founding Era Collection, which we trialled back in December and January.  If you'll forgive me for re-using old material, I thought it would be helpful therefore to re-post one of the earliest posts on this blog, which goes into detail about what the collection contains and how to navigate it.

The American Founding Era Collection contains digital versions of the published papers of several major figures of the time.  The collections it contains are as follows:

• The Adams Papers
• The Papers of Thomas Jefferson
• The Dolley Madison Digital Edition
• The Papers of James Madison
• The Papers of George Washington
• The Documentary History of the Ratification of the Constitution 
• The Papers of Alexander Hamilton (coming soon!)

With the exception of the Dolley Madison papers, these are digital editions of the print versions, which have been being published in large, ongoing series for many years.  We do have these print volumes at the VHL (which you can locate by searching SOLO), but the digital versions offer a variety of ways to access the papers and are of course fully searchable - no need to go hunting through indexes or worry you might miss a reference along the way, which for such enormous publications is a huge help.

Note on the Adams Papers: The Founding Era Collection only contains these papers from the founding-generation of the Adams family, and so the bulk of the papers available date from the 18th century.

Note on the Dolley Madison Digital Edition:  This is the first ever complete edition of all Dolley Madison's correspondence. This collection was 'born digital', and is currently complete up to 1837. It is included in and can be accessed via the Founding Era Collection, but also has a separate platform, the one on which it was originally built, which also includes annotations that aren't accessible from the Founding Era platform. There are links across to the standalone platform from each document in the Founding Era Collection so that these annotations can be found easily.

Note on the Papers of Alexander Hamilton: This collection was not available when we initially trialled the resource, but we have decided to add it to our purchase. We will gain access to these papers when they become available later this summer.

How to use the database
As well as the full-text search, the browsing options are very powerful. You can browse collections individually or the whole lot at once, and have a choice of doing so by chronology (date of document) or by contents (order of the documents in the published volumes).  There are also browsable indexes available to the Adams, Jefferson, and Washington Papers.

Once you have started browsing any of the contents, chronology, indexes or your search results, there is a navigation compass to take you forward and back through the different levels and documents.  It looks a bit like part of the background (at least, I didn't realise what it was at first!), so in case it's not just me that overlooked it, I thought it was worth pointing out.  The platform's own guide describes how the compass works as follows:

 The navigation compass can be used to move between different places in the current view. The up and down arrows are used to move up and down within the hierarchy. For example, starting at the top of the contents view and repeatedly clicking the down arrow takes you from the publication to a series to a volume and so on down to individual documents. For the chronology view, doing the same thing takes you from decade to year to month to day to documents within that day. The left and right arrows are used to navigate between adjacent items at the same hierarchical level. Thus, if you are in the level corresponding to volumes of a publication, then clicking the left or right arrow takes you to the preceding or following volume, respectively. Again, if you are at the months level of the chronological hierarchy, then clicking the left or right arrows takes you to the preceding or following month. 

It took me a bit of getting used to, but once you've got your head round the structure of the collection, it is an easy way to move from one part to another.  Next to it you always see where you are in the hierarchy.

The documents themselves are transcribed, not page images, and contain links to notes, explanatory references and other documents where relevant. You can print documents, but there are no options for exporting/saving records other than to make a note of the durable URL given in the citation box at the bottom of each document.  This box is a useful addition though, giving you guidance on how to cite the document in bibliographies, which can often be tricky to know how to do for online resources.

The collections are still being added to, as the print publication projects are still ongoing.  There is also a complementary project, Founders Early Access, which is freely available online as well as accessible within the main site.  The Early Access collection contains documents that are in the process of being prepared for both print and online publication.  Once the documents are added to the main collection, they disappear from the Early Access site.  The link for the Early Access collection is: http://rotunda.upress.virginia.edu/founders/FOEA.html and you can keep up with updates to it at http://rotunda.upress.virginia.edu/founders/FOEA-history.html.  It might also be worth pointing out that the University of Virginia is working with the National Archives to make some of the Founders' papers freely available online from 2012. 

The American Founding Era Collection is accessible via OxLIP+.  If you are away from Oxford, you can get into the database as long as you sign in via SOLO/OxLIP+ first with your University single sign-on.

New! Online guide to US History sources

I'm pleased to announce the publication of the online Guide to US History Sources, on the Bodleian Libraries' LibGuides site.  This guide replaces the old yellow paper versions which used to be available in the library, and provides links to and information about a whole variety of resources available for research in US History.


The web address for the guide is http://ox.libguides.com/ushistory.  The benefits of an online guide are that the links to e-resources are active (if you're not connected to the University network you will need to sign in on SOLO first), and the guide can be easily and frequently updated.  Hopefully it will provide you with an excellent starting point for your research.  Please feel free to make suggestions about more resources to include, or if the guide could be made more user-friendly.  We also have a separate online guide available for US Government Publications, and could potentially create further specific guides in the future.



The online guide is designed to be complemented by this blog, which will continue to go into more depth about individual resources and topics.  I'm also always happy to help you if you have questions about resources available either in the library or online.

April 15, 1912: Sinking of the Titanic

A quick post this one, to point you in the direction of an interesting document we have available in the US Congressional Serial Set.  If you came to the talk on the Serial Set by August Imholtz from Readex in January you'll have seen this before, and we've had it out on display a couple of times, so it may already be familiar. It is however an example of the way that the Serial Set contains documents and primary sources for a huge range of historical events.

Following the sinking of the Titanic on April 15, 1912, the US Senate's Committee on Commerce held a series of hearings into the disaster. The Committee was directed by S.Res 283 'to investigate the causes leading to the wreck', and a large number of people testified, including eye-witness accounts of the disaster itself.  One of these accounts, from a Mrs Emily Ryerson, a passenger on the boat, was used as a source for the 1997 film.  The full hearings document runs to 1170 pages, and you can also access the (much briefer!) final report, submitted to Congress on May 28. 

Link to the full hearings [Oxford users only] Mrs Ryerson's testimony is on pages 1107 and 1108.
Link to the full report [Oxford users only]

I'll be writing a fuller post on the Serial Set, what it contains and how to use it, in due course!  If you're curious in the meantime though, there is a slideshow presentation embedded on our online US Government Publications Guide (click on the 'Legislative/Congressional' tab and then scroll down the page).