sparse column shift encoding. (#1502)

Many of our matrices are log normalized, which tends to eliminate
the number of non zero values (if there were any).  This prevents
the matrix from being stored as a sparse matrix.  The solution here
is to use a simple transformation to make it sparse again.  The most
common value from each column is subtracted from that column.  These
values that were subtracted are saved in an array called X_col_shift.

The cellxgene code needs to understand how to undo the transformation when
operating over the X matrix.

- added script to create a synthetic dataset for testing
- added a script to convert an existing CXG dataset to a sparse CXG dataset
This commit is contained in:
bmccandless
2020-06-02 08:23:52 -07:00
committed by GitHub
parent d0577b94af
commit 76523d4f32
8 changed files with 297 additions and 63 deletions
+2 -1
View File
@@ -60,7 +60,7 @@ def skip_if(condition, reason: str):
return decorator
def app_config(data_locator, backed=False):
def app_config(data_locator, backed=False, extra={}):
args = {
"embeddings__names": ["umap", "tsne", "pca"],
"presentation__max_categories": 100,
@@ -74,6 +74,7 @@ def app_config(data_locator, backed=False):
}
config = AppConfig()
config.update(**args)
config.update(**extra)
config.complete_config()
return config