⌂

# Documentation

## Download

Click on https://ecacorpus.eu/misc/ecacorpus.zip

Unzip and click on `index.html`, the only file in the root directory, no installation is required. The zip files contains only data, no programs. It is Portable Web Package (PWP), essentially a directory structure viewable with web browser. The ECA Corpus can also be explored online at https://ecacorpus.eu. Both, downloaded and online, are identical, as long as they have the same timestamp.

## Overview

A single zip file can contain the full structured data of an organisation such as ECA since its creation in 1977; ECA is a good example for this exercise. The zip file contains: tabular data, full text of the files (eventually), and generated reports with graphics. Anyone can simply unzip the file and:

- Explore it locally on their computer with any browser.
- Load it on a web server to explore it online.

Deeper analysis using both current and future technologies can be envisaged. For example:

- Studying the full text of reports, not only their descriptions.
- Creating SQLite databases to facilitate further analysis; indeed, the whole ECA Corpus could be packaged as one SQLite database.
- Running traditional data analysis or modern methods such as AI.
- Building tools such as vector databases to search or analyse the documents.

## Data structure

A dossier represents a publication or other document types. Each dossier is identified by a unique dossier number. There are two main components:

Register

  The Register is the list of dossiers. It is a table containing the dossier number, title, and several additional fields. Once the Register has been cleaned and completed, its data will later be integrated into the Store. Currently, the Portable Web Package is generated from the Register, which it is used for both, the `ecacorpus.zip` file and the `ecacorpus.eu` website.

Store

  The Store contains the full text of each dossier in all available languages and formats. It is organized as a directory structure with a root folder named `store`, which contains a set of numbered folders; each folder corresponds to one dossier. At present, the Store consists of an empty skeleton directory structure, ready to receive full-text files in all languages and formats. All texts should be converted to plain text to facilitate further processing and ensure long-term preservation.

  If a dossier corresponds to a section of a larger document, only the relevant sections should be included. For example, in the case of the Official Journal C139/1979, only the material from page 15 concerning ECA should be included. Each opinion should be copied to the appropriate dossier, as this issue of the Official Journal contains six opinions.

  Example of dossier in the Store

## Data quality

General
  The data in the ECA Corpus was scraped from the ECA search engine and the ECA lists (see below). Hence, it probably contain errors and omissions.
Missing documents
  It is impossible to know if there are missing documents as the ECA does not have a comprehensive register. Totals are not the same when comparing: ECA list vs. ECA search engine vs. ECA Corpus.
Data cleaning
  Further cleaning is required.
Full text files
  To do: copying the full texts to the store, taking only the relevant parts, converting to plain text.
Dates
  Which date should be indicated? Example with Annual report of 1977:
  - ECA website: 01-01-1978
  - ECA: 30 November 1978
  - OJ: 30 December 1978
Official Journal links
  Example with annual report of 1977:
  - ECA website: direct to English without other languages selection, poor scanning.
  - Publications Office: landing page with all the languages, better scanning.

Corrigenda must not be counted as separated reports, just corrections of original reports. There are seven corrigenda in the ECA Corpuse database, so the distortion is small.

## ECA lists

Might not be up to date.

- Annual Reports
- Specific Annual Reports
- Special Reports
- Opinions
- Reviews

## Abbreviations

- O: Official Journal. Link to EUR-Lex, multiligual landing page.
- N: Not-published. Type of document that should be published in the Official Journal, but it was not.
- X: Non-applicable. Type of document not published in the Official Journal.
- DOI: Digital Object Indentifier
- ELI: European Legislation Identifier, "... to make legislation available online ..." Is it appropriate to give an ELI to ECA reports as this implies that they are legislation?
- EDATA: European data

## Links

- ECA Legal framework
- Articles 246 to 248 of the EC Treaty
- Historical Archives of the European Union - ECA
- BibliotECA Discovery
- Official Journal (OJ)
- data.europa.eu
- Directive on open data and the re-use of public sector information
- Interoperable Europe Act
- Organisations in the list of Specific Annual Reports
- Generated multilingual graphics