cellxgene/dev_docs/schema_guide.md

Cellxgene Schema Guide

Datasets included in the data portal and hosted cellxgene need to follow the schema described here. That schema defines some required fields, requirements about feature labels, and some optional fields that mostly help with presentation.

The number of fields is rather low, and we expect that information needed to populate those fields should either already be present in datasets prepared by a submitter or be easy to obtain. However, this still leaves the task of actually manipulating the dataset so that it follows the schema: adjusting field names, ensuring proper ontologies are used, converting gene symbols to a common set, etc. This can be tedious and error-prone, and at the beginning of the hosted cellxgene project, this was always done with engineering support. As we increase the rate at which we add data, we want to eliminate the need for engineering support so that ultimately submitters themselves can create files that follow the schema.

cellxgene schema apply

To enable this, we have a new cellxgene subcommand, cellxgene schema, that handles applying and verifying the schema. Its first subcommand, cellxgene schema apply, takes three inputs:

1. A source h5ad file. The input needs to be an AnnData file, so if a submitter has, say, a serialized Seurat or SingleCellExperiment object, it needs to be converted to AnnData first. This can be done with sceasy or via Seurat.
2. A configuration yaml file that describes the fields to add and conversions to apply (see below).
3. A name for the new h5ad file that should follow the schema.

Configuration yaml

The configuration yaml file describes how to apply the schema. This is an example of a "skeleton" yaml that has all the fields required for the 1.0.0 schema but is not yet filled in with any logic:

uns:
    version:
        corpora_schema_version: 1.0.0
        corpora_encoding_version: 0.1.0
    contributors:
    title:
    layer_descriptions:
    preprint_doi:
    publication_doi:
    organism_ontology_term_id:
obs:
    tissue_ontology_term_id:
    assay_ontology_term_id:
    disease_ontology_term_id:
    cell_type_ontology_term_id:
    sex:
    ethnicity_ontology_term_id:
    development_stage_ontology_term_id:
fixup_gene_symbols:

Unstructured metadata

The first section is uns, which includes metadata fields that describe the whole dataset (see here for further description of uns and obs.).

The first line is version, which is required for most of our tooling to work. The schema version is set at 1.0.0 in the example above, but of course for future versions that should be changed.

Next is contributors which describes who is adding the dataset to the portal. If you consult the schema, you see that contributors is a list where each element can have name, email, and institution. So when filled out, the contributors field should look like this:

contributors:
  - name: Mary B. Scientist
    email: mbs@singlecell.edu
    institution: Single-Cell University
  - name: Robert J. Scientist
    email: rjs@usingle.edu
    institution: University of Single Cell

title is the name of the dataset, and is just a string that gets displayed in the portal and cellxgene to identify the dataset.

layer_descriptions is free text descriptions of the different layers of the AnnData file. It should look like this when complete, depending on what layers are present:

layer_descriptions:
  X: CPM and logged
  raw.X: raw

Note that one of the layers needs to be "raw", that is, the AnnData file must contain raw counts.

The two DOI fields are optional but can be included if the dataset is associated with a publication or preprint. Note that the DOI should be a full url:

publication_doi: https://doi.org/10.1073%2Fpnas.83.15.5372

Finally, the organism_ontology_term_id field is the species of the donor organism from the NCBITaxon ontology. The value for Homo sapiens is NCBITaxon:9606:

organism_ontology_term_id: NCBITaxon:9606

Note that the schema also requires a human-readable organism field, but this doesn't need to be included in the yaml. When the cellxgene schema apply script encounters an ontology field, it looks up the label for the term(s) and inserts it into the appropriate field.

Observation metadata

The next section is obs, which is metadata than can vary for each observation (and "observation" usually means cell). These fields are all ontology fields except for sex, which has its own enumerated set of permitted values.

There are two ways to fill in the obs fields. The first is useful when there is only one value for all the observations in the dataset. This is not uncommon, for example all cells often come from the same assay. In that case just insert the ontology term:

assay_ontology_term_id: EFO:0009922

The second is for when there is an existing field in the dataset that needs to be mapped to the schema field. For example, the submitter may have included cell type annotations in a field called CellType, and those annotations may just be free text. This doesn't follow the schema because it needs to be in cell_type_ontology_term_id and cell_type, and it needs ontology terms and labels, not just any text. In that case the field can be a dictionary:

cell_type_ontology_term_id:
  CellType:
    t-cell: CL:0000084
    b-cell: CL:0000236

This will look at the obs.CellType field in the dataset, and where it has the value "t-cell", it will insert CL:0000084 into cell_type_ontology_term_id and its label T cell into cell_type.

Now there are often situations where there is no valid ontology term for some field. For example, the dataset may have been produced via an assay not present in EFO. Or, a particular cell type may have no entry in CL. In that case, a free text description can be used in the ontology_term_id field:

assay_ontology_term_id: Sci-Plex
cell_type_ontology_term_id:
  CellType:
    t-cell: CL:0000084
    b-cell: CL:0000236
    new cell type: new cell type

In these cases, the cellxgene schema apply script will leave the ontology field blank and move the free text description into the label field. So the assay_ontology_term_id in the new dataset would be "" but assay would be Sci-Plex.

Gene symbol harmonization

The last section describes how gene symbol conversion should be applied to each of the layers. This is similar to the layer_descriptions field above, but there are only three permitted values: raw, log1p, and sqrt:

fixup_gene_symbols:
  X: log1p
  raw.X: raw

This tells the script how each each layer was transformed from raw values that can be directly summed. raw means that the layer contains raw counts or some linear tranformation of raw counts. log1p means that the layer has log(X + 1) for each the raw X values. sqrt means sqrt(X) (this is not common). For layers produced by Seurat's normalization or SCTransform functions, the correct choice is usually log1p.

cellxgene schema validate

The next cellxgene schema subcommand is cellxgene schema validate, and it validates that a given h5ad follows a version of the schema. It accepts two parameters:

1. The h5ad file to check
2. The version of the schema to check against.

If the validation succeeds, the command will have a zero exit code. If it does not, it will have a non-zero exit code and will print validation failure messages.