[sudo-discuss] Wikileaks Plus D & the Kissinger Cables

[Eddan]

now this is really dangerous... Add metadata search functionality to
Wikileaks-released excessively classified diplomatic cables. Make this
comprehensible to people who aren't foreign policy geeks and the Arab
Spring will have been just warm-up practice.

it seems like it would be very useful to better understand what Wikileaks
means by their reverse engineering of the US gov metadata. it would be
_monumental_ to further enhance this treasure trove with some natural
language search and more sophisticated pattern recognition. Sudo-Leaks,
anyone?

[excerpt from
http://wikileaks.org/plusd/about/]

The Kissinger Cables <http://wikileaks.org/plusd/about/#tkc>

The Kissinger Cables comprise more than 1.7 million US diplomatic records
for the period 1973 to 1976. Dating from January 1, 1973 to December 31,
1976 they cover a variety of diplomatic traffic including cables,
intelligence reports and congressional correspondence. They include more
than 320,000 originally classified records, including 286,000 full US
diplomatic cables. There are more than 12,000 documents with the sensitive
handling restriction "NODIS", 'no distribution', and more than 9,000
labelled "Eyes Only". Full cables originally classed as "SECRET" total more
than 61,000 and "CONFIDENTIAL" more than 250,000.

The records were reviewed by the United States Department of State's
systematic 25-year declassification process. At review, the records were
assessed and either declassified or kept classified with some or all of the
metadata records declassified. Both sets of records were then subject to an
additional review by the National Archives and Records Administration
(NARA). Once believed to be releasable, they were placed as individual PDFs
at the National Archives as part of their Central Foreign Policy Files
collection. Despite the review process supposedly assessing documents after
25 years there are no diplomatic records later than 1976. The formal
declassification and review process of these extremely valuable historical
documents is therefore currently running 12 years late.

The form in which these documents were at NARA was 1.7 million individual
PDFs. To prepare these documents for integration into the PlusD collection,
WikiLeaks obtained and reverse-engineered all 1.7 million PDFs and
performed a detailed analysis of individual fields, developed sophisticated
technical systems to deal with the complex and voluminous data and
corrected a great many errors introduced by NARA, the State Department or
its diplomats, for example harmonizing the many different ways in which
departments, capitals and people's names were spelled. All our corrective
work is referenced and available from the links in the individual field
descriptions on the PlusD text search interface:
https://search.wikileaks.org/plusd. For more information on what WikiLeaks
did to prepare the Kissinger Cables please see
here<http://wikileaks.org/plusd/about/#ptk>
.

Not all records from the period 1973-1976 have been obtained. NARA claims
diplomatic records for the period 1973 to 1976 chosen for content deletion
were of a ephemeral character. These records were identified by the "TAGS"
that were attached to them. TAGS ("Traffic Analysis by Geography and
Subject") refers to the content tagging system implemented by the
Department of State for its central foreign policy files in 1973. There are
geographic, organization and subject TAGS. This system was developed to
standardise search terms for departmental uses and was not static - TAGS
were added and deleted as necessary over time. At review, all cables that
only contained "temporary" TAGS, such as embassy logistical or staffing
requests, were permanently destroyed.

Tens of thousands of documents were irreversibly corrupted in this data set
due to technical errors when the documents were moved as computer systems
were upgraded, or so the US Department of State claims. This caused the
content of the document to be lost, though the metadata is still available.
These are often noted by a error message in the content of the document.
The documents lost in this manner are most documents from the following
periods:

   - December 1, 1975 to December 15, 1975
   - March 8, 1976 to April 2, 1976
   - May 25, 1976 to July 1, 1976

You can see the absence of these weeks by constructing a Timegraph of
"TAGS" as this term occurs in the content of nearly every document:
https://search.wikileaks.org/plusd/graph

Top Secret documents are also not available. During a migration of records
the Department of State printed out all Top Secret documents for
"preservation purposes" and the electronic versions were destroyed
permanently. These documents now only exist as hardcopies and so are
unavailable online in any form, even if declassified.

The documents not deleted either remained classified (or were deemed
unreleasable for other reasons), or were declassified and publicly
released. For the former, a "withdrawal card" was provided giving some
limited metadata about the document, the fields of which that were decided
as releasable vary from document to document. This metadata provides some
information about the document, for example the date and destination, that
can be used for research purposes and also allows a detailed FOIA request
to be made for the document. These FOIA requests can be directed to NARA's
Special Access and FOIA staff. For more information about this, please see
their online guide here <http://www.archives.gov/foia/foia-guide.html>. You
will need the document number and the To and From information.

There are nine different "Types" of document included in the Kissinger
Cables. The majority are of type "TE" - telegram (cable), which are
official diplomatic messages sent between embassies and the US Secretary of
State conveying official information about policy proposals and
implementation, program activities, or personnel and diplomatic post
operations. From 1973 onwards diplomatic cables were mostly electronic,
therefore most cables made releasable include the body (content) of the
cable. However, the other types of documents are paper records, including
airgrams and diplomatic notes. These are stored on microfilm (from 1974
onwards, as the Department of State did not microfilm documents until then)
and so were not released with the full content of the documents, even if
marked for public release. Although the body of the message is not
available online the full index (metadata) is provided for those "P-reel"
documents that were marked for release. Even though the whole document has
not been digitised the metadata is still useful for research purposes and
the documents can be requested under the Freedom of Information Act. For
those documents on P-reel that were not declassified and released a P-reel
"withdrawal card" is provided giving limited metadata. To access P-reel
documents that have a withdrawal card you should follow the same FOIA
procedure as for Telegram withdrawal cards. For the content of P-reel
documents which have been released, the process depends slightly on which
year the document you are requesting was created, but all requests should
be directed to:archives2reference at nara.gov.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sudoroom.org/pipermail/sudo-discuss/attachments/20130408/6595f0a0/attachment.html>