fork/cellxgene
1
0
Fork 0
mirror of https://github.com/chanzuckerberg/cellxgene.git synced 2026-09-15 12:47:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

Clean up dead/hosted code [zh2310] (#2430)

Browse Source 
* Clean up dead/hosted code

* Remove schema conversion tool and related
* Remove cxg references
* Remove locust

* missed a spot

* Remove aws secret manager

* Merge branch 'main' into brodgers/2310/code-cleanup-v1

* cleanup merge
This commit is contained in:
Ben MR
2021-09-17 13:41:12 -07:00
committed by GitHub
parent 69e159916e
commit ef2ab07ca0
35 changed files with 5 additions and 2882 deletions
Download Patch File Download Diff File Expand all files Collapse all files

164
dev_docs/cxg.md
View File

@@ -1,164 +0,0 @@
## UPDATE (9/30/2020): Starting today, the name Corpora will only be used as the internal project name, with cellxgene Data Portal being the official product name
# CXG Data Format Specification
Document Status: _draft_
Version: 0.2.0 (_DRAFT, not yet approved_)
Date Last Modified: 2020-07-23
## Introduction
CXG is a cellxgene-private data format, used for at-rest storage of annotated matrix data. It is similar to [AnnData](https://anndata.readthedocs.io/en/stable/), but with performance and access characteristics amenable to a multi-dataset, multi-user serving environment.
CXG is built upon the [TileDB](https://tiledb.com/) embedded database. Each CXG is a TileDB [group](https://docs.tiledb.com/main/api-usage/object-management), which in turn includes one or more TileDB multi-dimensional arrays.
This document presumes familiarity with [TileDB terminology and concepts](https://docs.tiledb.com/main/), the [Corpora schema](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema.md) and its [H5AD encoding](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema_h5ad_implementation.md), and the AnnData/H5AD data model.
This document also leverages the current cellxgene schema, which is documented in the [REST API spec](./REST_API.md).
### Terminology
Unless explicitly noted, the AnnData conventions and terminology are adopted when referring to general annotated matrix characteristics (eg, `n_obs` is the number of observations/rows/cells in the annotated matrix). Where implied by context, eg, "TileDB array attribute", domain-specific terms are used.
Where capitalized, [IETF RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) conventions are followed (ie, conventions MUST be followed).
_Author's note:_ if you see any ambiguous terms, please call them out for clarification.
### Reserved
The `cxg` prefix is used for CXG-specific names.
### Encoding Data With TileDB Arrays
The TileDB array schema authoritatively defines the characteristics of each array (eg, the type of `X` is defined by [`X.schema`](https://tiledb-inc-tiledb-py.readthedocs-hosted.com/en/stable/python-api.html#tiledb.libtiledb.Array.schema)). In some cases, additional metadata is required for the CXG, and is attched to the array using the TileDB [array metadata](https://docs.tiledb.com/main/basic-concepts/array-metadata) capability.
All TileDB arrays MUST have a uint32 domain, zero based. All X counts and embedding coordinates SHOULD be coerced to float32, which is ample precision for visualization purposes, and MUST be a numeric type. Dataframe (metadata) types are generally preserved, or where that is not possible, converted to something with equal representative value in the cellxgene application (eg, categorical types are converted to string, bools to uint8, etc).
CXG consumers (readers) MUST be prepared to handle any legal TileDB compression, global layout and tile size. CXG writers SHOULD attempt to encode data using best-effort heuristics for time and space considerations (eg, dense/sparse encoding tradeoffs).
## Entities
### CXG
The CXG is a TileDB group containing all data and metadata for a single annotated matrix. The following objects MUST be present in a CXG, except where noted as optional:
* __obs__: a TileDB array, of shape (n_obs,), containing obs annotations, each annotation stored in a separate TileDB array attribute.
* __var__: a TileDB array, of shape (n_var,), containing var annotations, each annotation stored in a separate TileDB array attribute.
* __X__: a TileDB array, of shape (n_obs, n_var), with a single TileDB attribute of numeric type.
* __X_col_shift__: (optional) TilebDB Array used in column shift encoding, shape (n_var,), dtype = X.dtype. Single unnamed numeric attribute.
* __emb__: a TileDB group, which in turn contains all (zero or more) embeddings.
* __emb__/\<embedding_name\>__: a TileDB array, with a single anonymous attribute, of numeric type, and shape (n_obs, N>=2).
* __cxg_group_metadata__: an empty TileDB array, used to store CXG-wide metadata
### obs and var
All per-observation (obs) and per-feature (var) data is encoded in a TileDB array named `obs` and `var` respectively, with shape (n_obs,) and (n_var,). Each TileDB array has an array attribute for each obs/var column. All TileDB array attributes will have the same type and value as the original data, eg, float32, with the following exceptions:
* bool is encoded as uint8 (1/0)
* categorical is encoded as string
* Numeric types are cast to 32-bit equivalents
In addition to the obs/var data, both TileDB arrays contain an optional 'cxg_schema' metadata field that is a JSON string containing per-column (attribute) schema hinting. This is used where the TileDB native typing information is insufficient to reconstruct useful information such as categorical typing from Pandas DataFrames, and to communicate which column is the preferred human-readable index for obs & var.
The `cxg_schema` JSON string is attached to the TileDB array metadata, and is a dictionary containing the following top-level names:
* "index": string, containing the name of the index column
* \<column-name\>: optional, a JSON dict, contain a schema definition using the same format as the cellxgene REST API /schema route
For example:
```
{
"index": "obs_index",
"louvain": { "type": "categorical", "categories": [ "0", "1", "2", "3", "4" ]}
"is_useful": { "type": "boolean" }
}
```
### X
TileDB array, with a single anonymous attribute, shape (n_obs, n_var), containing the count matrix (equivalent to the AnnData `X` array). MUST have numeric type, and SHOULD be float32. The TileDB schema defines type and sparsity, and both dense and sparse encoding are supported.
### X_col_shift
Optional TileDB array, used to encode-per column offsets for column-shift sparse encoding. The TileDB array will have a single anonymous attribute, of the same type as the X array, and shape (n_var,).
If the X array is sparse, and X_col_shift exists, then all values in the i'th column were subtracted by X_col_shift[i].
### emb and embedding arrays
A CXG must have a group named `emb`, which will contain all embeddings. Embeddings are encoded as TileDB arrays, of numeric type and shape (n_obs, >=2). The arrays SHOULD be coerced to float32, and MUST be a numeric type. The TileDB array name will be assumed to be the embedding name (conventionally, embedding names in CXG are _not_ prefixed with an `X_` as they are in AnnData).
CXG supports zero or more embeddings. Note that cellxgene currently _requires_ at least one embedding.
### cxg_group_metadata
Required, but empty TileDB array, used to store CXG-wide metadata. The following fields are defined:
* __cxg_version__: (required) a semver string identifying the specification version used to encode the CXG.
* __cxg_properties__: (optional) a dictionary containing dataset wide properties, defined below.
* __cxg_category_colors__: (optional) a categorical color table, defined below.
#### cxg_properties
The properties metadata dictionary contains dataset-wide properties, encoded as a JSON dictionary. Currently, the following fields are defined:
* title: string, dataset human name (eg, "Lung Tissue")
* about: string, fully-qualified http/https URL, linking to more information on the dataset.
All implementions MUST ignore unrecognized fields.
#### cxg_category_colors
This optional field contains a copy of the category color table, which MAY be used to display category-specific color labels. This is a JSON dictionary, containing a per-category color-table. Each color table is named `{category_name}_colors`, and is itself a dictionary mapping label name to RGB color. For example:
```
{
"louvain_colors": {
"0": "#FFFFFF",
"1": "#000000"
}
}
```
## Corpora Schema Encoding
The [Corpora schema](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema.md) and [Corpora AnnData encoding](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema_h5ad_implementation.md) define a set of metadata and encoding conventions for annotated matrices. When a Corpora dataset is encoded as a CXG, the following shall apply.
### Corpora metadata property
A CXG containing a Corpora dataset will contain a property in the __cxg_group_metadata__ field named `corpora`. The value will be a JSON encoded string, which in turn contains all properties defined in the [Corpora AnnData uns](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema_h5ad_implementation.md#uns) container. For example:
```
{
"corpora": {
"version": {
"corpora_schema_version": "1.0.0",
"corpora_encoding_version": "0.1.0",
}
}
}
```
The `corpora` metadata, if present, MUST contain the version information. Optionality of other values in this object will follow the specifications set forth in the relevant Corpora schema specification (ie, optional fields are optional, required are present, etc), with the following changes:
* the contents of `corpora_encoding_version` MUST be identical to the `cxg_version`, as this field is defined as the current object encoding version, *NOT* the source data encoding version.
* the entire encoding will be JSON, rather than a hybrid Python/JSON encoding, but will otherwise follow the data structure defined by the AnnData Corpora encoding.
* the `<obs_column>_colors` will be omitted in favor of `cxg_category_colors`
### Other Corpora fields
All other Corpora schema fields will be encoded into a CXG using the conventions defined in the [Corpora Schema AnnData Implementation](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema_h5ad_implementation.md). For example, fields in `AnnData.obs` will be encoded in the CXG `obs`array as defined [above](#obs-and-var).
### Compatibility with CXG 0.1.0
For backwards compatibility and continuity with CXG version 0.1.0, the following MUST be implemented.
#### Presentation Hints
* The [Corpora `title`](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema.md#presentation-metadata) value MUST be saved in the `cxg_properties.title` field.
* The [Corpora `color_map`](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema.md#presentation-hints), when present in the dataset, MUST be saved in the `cxg_category_colors` field and NOT in the `corpora` field.
* The [Corpora SUMMARY `project_link`](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema.md#presentation-hints), if present, MUST be saved in the `cxg_properties.about` field.
Where these values differ in the final CXG, the `cxg_properties` values WILL take precedence.
## CXG Version History
There were several ad hoc version of CXG created prior to this spec. This describes the _proposed_ next version of CXG, which incoporates support for Corpora schema semantics. Prior verisons:
* _unnamed_ - an unnamed development version. Did not include explicit versioning support in the data model, but can be detected by the absence of __cxg_group_metadata__ and any version property. Created in early 2020, and not actively used in production
* 0.1 - the first and current version, defined to support the capabilities of the mid-2020 cellxgene. Created in early 2020, and in active use. Includes everything in this spec, excluding Corpora schema support. __NOTE:__ this version is encoded with a short-hand (malformed) semver version number.
* 0.2.0 - this specification.

175
dev_docs/schema_guide.md
View File

@@ -1,175 +0,0 @@
# Cellxgene Schema Guide
Datasets included in the [data portal](https://cellxgene.cziscience.com/) and hosted cellxgene need to follow the schema
described [here](https://github.com/chanzuckerberg/corpora-data-portal/blob/main/backend/schema/corpora_schema.md). That
schema defines some required fields, requirements about feature labels, and some optional fields that mostly help with
presentation.
The number of fields is rather low, and we expect that information needed to populate those fields should either already
be present in datasets prepared by a submitter or be easy to obtain. However, this still leaves the task of actually
manipulating the dataset so that it follows the schema: adjusting field names, ensuring proper ontologies are used,
converting gene symbols to a common set, etc. This can be tedious and error-prone, and at the beginning of the hosted
cellxgene project, this was always done with engineering support. As we increase the rate at which we add data, we want
to eliminate the need for engineering support so that ultimately submitters themselves can create files that follow the
schema.
## `cellxgene schema apply`
To enable this, we have a new cellxgene subcommand, `cellxgene schema`, that handles applying and verifying the schema.
Its first subcommand, `cellxgene schema apply`, takes three inputs:
1. A source h5ad file. The input needs to be an AnnData file, so if a submitter has, say, a serialized Seurat or
SingleCellExperiment object, it needs to be converted to AnnData first. This can be done with
[sceasy](https://github.com/cellgeni/sceasy) or via
[Seurat](https://satijalab.org/seurat/v3.1/conversion_vignette.html).
2. A configuration yaml file that describes the fields to add and conversions to apply (see below).
3. A name for the new h5ad file that should follow the schema.
### Configuration yaml
The configuration yaml file describes how to apply the schema. This is an example of a "skeleton" yaml that has all the
fields required for the 1.0.0 schema but is not yet filled in with any logic:
```
uns:
version:
corpora_schema_version: 1.0.0
corpora_encoding_version: 0.1.0
contributors:
title:
layer_descriptions:
preprint_doi:
publication_doi:
organism_ontology_term_id:
obs:
tissue_ontology_term_id:
assay_ontology_term_id:
disease_ontology_term_id:
cell_type_ontology_term_id:
sex:
ethnicity_ontology_term_id:
development_stage_ontology_term_id:
fixup_gene_symbols:
```
#### Unstructured metadata
The first section is `uns`, which includes metadata fields that describe the whole dataset (see
[here](https://anndata.readthedocs.io/en/latest/) for further description of `uns` and `obs`.).
The first line is `version`, which is required for most of our tooling to work. The schema version is set at
1.0.0 in the example above, but of course for future versions that should be changed.
Next is `contributors` which describes who is adding the dataset to the portal. If you consult the schema, you see that
contributors is a list where each element can have `name`, `email`, and `institution`. So when filled out, the
`contributors` field should look like this:
```
contributors:
- name: Mary B. Scientist
email: mbs@singlecell.edu
institution: Single-Cell University
- name: Robert J. Scientist
email: rjs@usingle.edu
institution: University of Single Cell
```
`title` is the name of the dataset, and is just a string that gets displayed in the portal and cellxgene to identify the
dataset.
`layer_descriptions` is free text descriptions of the different
[layers](https://anndata.readthedocs.io/en/latest/anndata.AnnData.layers.html) of the AnnData file. It should look like
this when complete, depending on what layers are present:
```
layer_descriptions:
X: CPM and logged
raw.X: raw
```
Note that one of the layers needs to be "raw", that is, the AnnData file must contain raw counts.
The two DOI fields are optional but can be included if the dataset is associated with a publication or preprint. Note
that the DOI should be a full url:
```
publication_doi: https://doi.org/10.1073%2Fpnas.83.15.5372
```
Finally, the `organism_ontology_term_id` field is the species of the donor organism from the NCBITaxon ontology. The
value for _Homo sapiens_ is `NCBITaxon:9606`:
```
organism_ontology_term_id: NCBITaxon:9606
```
Note that the schema also requires a human-readable `organism` field, but this doesn't need to be included in the yaml.
When the `cellxgene schema apply` script encounters an ontology field, it looks up the label for the term(s) and inserts it
into the appropriate field.
#### Observation metadata
The next section is `obs`, which is metadata than can vary for each observation (and "observation" usually means cell).
These fields are all ontology fields except for `sex`, which has its own enumerated set of permitted values.
There are two ways to fill in the `obs` fields. The first is useful when there is only one value for all the
observations in the dataset. This is not uncommon, for example all cells often come from the same assay. In that case
just insert the ontology term:
```
assay_ontology_term_id: EFO:0009922
```
The second is for when there is an existing field in the dataset that needs to be mapped to the schema field. For
example, the submitter may have included cell type annotations in a field called `CellType`, and those annotations may
just be free text. This doesn't follow the schema because it needs to be in `cell_type_ontology_term_id` and
`cell_type`, and it needs ontology terms and labels, not just any text. In that case the field can be a dictionary:
```
cell_type_ontology_term_id:
CellType:
t-cell: CL:0000084
b-cell: CL:0000236
```
This will look at the `obs.CellType` field in the dataset, and where it has the value "t-cell", it will insert
`CL:0000084` into `cell_type_ontology_term_id` and its label `T cell` into `cell_type`.
Now there are often situations where there is no valid ontology term for some field. For example, the dataset may have
been produced via an assay not present in `EFO`. Or, a particular cell type may have no entry in `CL`. In that case, a
free text description can be used in the `ontology_term_id` field:
```
assay_ontology_term_id: Sci-Plex
cell_type_ontology_term_id:
CellType:
t-cell: CL:0000084
b-cell: CL:0000236
new cell type: new cell type
```
In these cases, the `cellxgene schema apply` script will leave the ontology field blank and move the free text
description into the label field. So the `assay_ontology_term_id` in the new dataset would be `""` but `assay` would be
`Sci-Plex`.
#### Gene symbol harmonization
The last section describes how gene symbol conversion should be applied to each of the layers. This is similar to the
`layer_descriptions` field above, but there are only three permitted values: `raw`, `log1p`, and `sqrt`:
```
fixup_gene_symbols:
X: log1p
raw.X: raw
```
This tells the script how each each layer was transformed from raw values that can be directly summed. `raw` means that
the layer contains raw counts or some linear tranformation of raw counts. `log1p` means that the layer has `log(X + 1)`
for each the raw `X` values. `sqrt` means `sqrt(X)` (this is not common). For layers produced by Seurat's normalization
or SCTransform functions, the correct choice is usually `log1p`.
### `cellxgene schema validate`
The next `cellxgene schema` subcommand is `cellxgene schema validate`, and it validates that a given h5ad follows a
version of the schema. It accepts two parameters:
1. The h5ad file to check
2. The version of the schema to check against.
If the validation succeeds, the command will have a zero exit code. If it does not, it will have a non-zero exit code
and will print validation failure messages.