# ASPy end-to-end test dataset: KH-9 Mapping Camera, Iceland

Test data for the end-to-end tests of [ASPy](https://github.com/adehecq/aspy), a Python wrapper around the
Ames Stereo Pipeline that processes declassified KH-9 Hexagon imagery into DEMs and orthoimages.

The dataset holds one KH-9 Mapping Camera strip of three overlapping scenes over Iceland. It contains
both the raw scans and the preprocessed images, so the preprocessing and the stereo tests can run on
their own, in parallel.

It is served by the IGE OPeNDAP (THREDDS) server and mirrors the Zenodo record
[10.5281/zenodo.23529348](https://doi.org/10.5281/zenodo.23529348) (version 1.1), with the files sorted into folders.

- **Catalog:** https://ige-meom-opendap.univ-grenoble-alpes.fr/thredds/catalog/meomopendap/extract/ASPY_DATA/kh9mc-iceland/catalog.html
- **Direct download:** `https://ige-meom-opendap.univ-grenoble-alpes.fr/thredds/fileServer/meomopendap/extract/ASPY_DATA/kh9mc-iceland/<path>`,
  with `<path>` as listed below (e.g. `aux/reference_dem.tif`)

## Scenes

| Entity ID | Acquisition date |
|---|---|
| DZB1216-500280L002001 | 1980-08-22 |
| DZB1216-500280L003001 | 1980-08-22 |
| DZB1216-500280L004001 | 1980-08-22 |

## Files

```
kh9mc-iceland/
├── raws/<id>.tgz
├── preprocessed/<id>_final.tif
├── metadata/
│   ├── entity_ids.txt
│   └── scenes_metadata.gpkg
└── aux/
    ├── reference_dem.tif
    ├── unstable_mask.tif
    ├── quickview_reference_dem.png
    └── quickview_unstable_mask.png
```

| File | Description | Used by |
|---|---|---|
| `metadata/entity_ids.txt` | Entity IDs of the three scenes, in the format read by `usgsxplore` | preproc, stereo |
| `metadata/scenes_metadata.gpkg` | Scene footprints and metadata, as written by `aspy search` | preproc, stereo |
| `raws/<id>.tgz` | Raw scans as distributed by USGS EarthExplorer | preproc |
| `preprocessed/<id>_final.tif` | Preprocessed images, as written by `aspy preproc` | stereo, and reference for preproc |
| `aux/reference_dem.tif` | Reference DEM over Iceland, 30 m, EPSG:3413, ellipsoid heights | stereo |
| `aux/unstable_mask.tif` | Mask of unstable terrain, excluded from the coregistration. Same grid as `reference_dem.tif`, uint8, 1 = unstable, 0 = stable | stereo |
| `aux/quickview_*.png` | Previews of the reference DEM and the unstable mask, to check them without downloading | — |

Each file can be downloaded on its own, so a single test only fetches the files it needs.

## MD5 checksums

```
e7660e36014497c0c05a741c33889514  metadata/entity_ids.txt
9ea1d6f2343e02e450ca50d79aed368a  metadata/scenes_metadata.gpkg
d05fc5d771a91a6cd0a4327e3a13e207  raws/DZB1216-500280L002001.tgz
9ba49ed6965e319300e14b64eac5bb9f  raws/DZB1216-500280L003001.tgz
59a6a48cb4f1ce77b5774e1a7ad9c7e3  raws/DZB1216-500280L004001.tgz
027926ace807390a347128981fbfc846  preprocessed/DZB1216-500280L002001_final.tif
e397bc7ebff8a4bf45c7a435dedf0305  preprocessed/DZB1216-500280L003001_final.tif
fade5805b205e8350b80d3059f969ff5  preprocessed/DZB1216-500280L004001_final.tif
ef5416ce482612448d4bc316544894f9  aux/reference_dem.tif
83fcaa7daa1d6a1a1440f447d8b150ef  aux/unstable_mask.tif
285e9886a4ba54a604e19beedae41465  aux/quickview_reference_dem.png
2b9989c5fd92c5b0ca7c9f15629f6316  aux/quickview_unstable_mask.png
```

## How the files were produced

- **ASPy version:** TODO (version or commit)
- `scenes_metadata.gpkg`: written by `aspy search`
- Raw `.tgz`: `aspy download`, unchanged
- `_final.tif`: `aspy preproc` with the default options
- `reference_dem.tif`: copy of `dem_GLODEM_Iceland_30m_epsg3413.tif`, a regional 30 m DEM prepared by
  TODO (name). Heights are above the WGS84 ellipsoid, even though the file metadata says `GEOID=egm2008`.
- `unstable_mask.tif`: built by ASPy's `prepare_aux` with `unstable_mask_sources = ["rgi", "worldcover"]`, over the
  whole extent of `reference_dem.tif` and on its grid. Unstable pixels are the RGI 7.0 glacier outlines and the
  ESA WorldCover 2021 classes 0, 10, 50, 70, 80, 90 and 95 (no data, tree cover, built-up, snow and ice, water,
  wetland, mangroves). A 30 m pixel is unstable as soon as one 10 m WorldCover pixel inside it is.
- `quickview_*.png`: the quickviews written by `prepare_aux` alongside the mask

## Changelog

- **1.1**: `unstable_mask.tif` replaced. In 1.0 it was a copy of `dem_GLODEM_Iceland_30m_epsg3413_wmask.tif`, a
  water-masked DEM rather than a mask, which masked the whole reference DEM during the coregistration.
  Added the `quickview_*.png` previews.
- **1.0**: initial release.

## Expected layout in an ASPy project

The tests place the files at the default paths of an ASPy project. Note that `raws/` becomes `raw/`, and that
each preprocessed image goes into its own `<id>/` folder:

```
data/
├── raw/<id>.tgz
├── preprocessed/<id>/<id>_final.tif
├── metadata/
│   ├── entity_ids.txt
│   └── scenes_metadata.gpkg
└── aux/
    ├── reference_dem.tif
    └── unstable_mask.tif
```

## Sources and licences

- **KH-9 Hexagon imagery:** declassified imagery from the U.S. Geological Survey (USGS EarthExplorer,
  Declass-2 collection). Public domain.
- **Reference DEM:** derived from the Copernicus DEM GLO-30 (TODO: confirm), converted from EGM2008 to
  ellipsoid heights. Free to use and redistribute with the attribution: "© DLR e.V. 2010-2014 and © Airbus
  Defence and Space GmbH 2014-2018 provided under COPERNICUS by the European Union and ESA; all rights
  reserved". Cite: European Space Agency (2024), Copernicus Global Digital Elevation Model,
  https://doi.org/10.5270/ESA-c5d3d65.
- **Unstable mask:** derived from the Randolph Glacier Inventory 7.0 (RGI 7.0 Consortium, 2023, Randolph Glacier
  Inventory - A Dataset of Global Glacier Outlines, Version 7.0, https://doi.org/10.5067/f6jmovy5navz), CC-BY-4.0,
  and from ESA WorldCover 10 m 2021 v200 (Zanaga, D. et al., 2022, https://doi.org/10.5281/zenodo.7254221),
  CC-BY-4.0, "© ESA WorldCover project 2021 / Contains modified Copernicus Sentinel data (2021) processed by
  ESA WorldCover consortium".

This dataset is distributed under CC-BY-4.0. Third-party data keeps its own licence and attribution requirements.