sparse column shift encoding. (#1502)

Many of our matrices are log normalized, which tends to eliminate
the number of non zero values (if there were any).  This prevents
the matrix from being stored as a sparse matrix.  The solution here
is to use a simple transformation to make it sparse again.  The most
common value from each column is subtracted from that column.  These
values that were subtracted are saved in an array called X_col_shift.

The cellxgene code needs to understand how to undo the transformation when
operating over the X matrix.

- added script to create a synthetic dataset for testing
- added a script to convert an existing CXG dataset to a sparse CXG dataset
This commit is contained in:
bmccandless
2020-06-02 08:23:52 -07:00
committed by GitHub
parent d0577b94af
commit 76523d4f32
8 changed files with 297 additions and 63 deletions
+1 -2
View File
@@ -241,8 +241,7 @@ class DataAdaptor(metaclass=ABCMeta):
duplicate_columns = list(set(labels_df.columns) & set(obs_columns))
if len(duplicate_columns) > 0:
raise KeyError(
"Labels file may not contain column names which overlap "
f"with h5ad obs columns {duplicate_columns}"
"Labels file may not contain column names which overlap " f"with h5ad obs columns {duplicate_columns}"
)
# labels must have same count as obs annotations