* 1510-smoke-test
* config default
* update tests
* update test config
* fix linter errors
* more comments
* address comments
* use npm install in push_tests.yml
* use environment.default.json
* adding docs
* Take care of @mweiden's nits
* Save screenshots in the __tests__/screenshots/ directory
* typo
* docs
* Add chart tests (#1580)
* merge tests
* check if bin creation returned null before rendering charts (#1576)
* check if bin creation returned null before rendering charts
* refactor chart rendering into functions (#1577)
* little fixes from PR
* reintroduce fix to check for null values
* change getAllByClass to return element
* slice instead
* new stackedbar test
* feedback-1573-test (#1579)
* feedback-1573-test
* enable whole test set
* revert tests
Co-authored-by: Timmy Huang <tihuan@users.noreply.github.com>
* tweak test to actually render chart
* include snapshot
* remove async
* fix getAllHistograms
* properly grab id
Co-authored-by: Timmy Huang <tihuan@users.noreply.github.com>
Co-authored-by: Matt Weiden <538456+mweiden@users.noreply.github.com>
Co-authored-by: Severiano Badajoz <sbadajoz@chanzuckerberg.com>
Many of our matrices are log normalized, which tends to eliminate
the number of non zero values (if there were any). This prevents
the matrix from being stored as a sparse matrix. The solution here
is to use a simple transformation to make it sparse again. The most
common value from each column is subtracted from that column. These
values that were subtracted are saved in an array called X_col_shift.
The cellxgene code needs to understand how to undo the transformation when
operating over the X matrix.
- added script to create a synthetic dataset for testing
- added a script to convert an existing CXG dataset to a sparse CXG dataset
Support for sparse tiledb arrays for the X matrix
1. cxgtool can now output sparse matrices
2. cxg_adaptor and diffexp_cxg updated to handle sparse matrices
3. added a test in test_diffexp to test sparse diffexp and get_X_array
There seem to be breaking changes in Chrome that are causing this to
fail. We've weighed the risk of disabling the feature with the issues
we've encountered using it and decided to disable it for now.
6.7 does not have the `hidden` flag used in the code. Users building the
app with an older version of click within the current range specified by
requirements.txt may fail.
* add sentry webpack plugin
* allow override of webpack config
* work around cheerio inability to parse jinga templates
* webpack can not minify jinja templates
* allow script injection to specify other attributes
* allow script injection to specify other attributes
* Adjustments to make plugin systems work
* Add sourcemaps for javascript in prod webpack
* Update .gitignore
* Fix spelling errors
Co-authored-by: Matt Weiden <538456+mweiden@users.noreply.github.com>
* app_config, fix bug with list/tuple command line arguments.
There was a error caused by pyyaml using lists, and click using tuples.
Now tuples are automatically converted to lists when the config is
updated.
* Add server plugin system
Plugins are optional modules loaded at runtime. Specification:
* Plugins are loaded from the server.plugins module (directory
server/plugins)
* The import_plugins method is run as part of the initialization of the
server module in __init__.py
* Add plugins to the EB build process
* Remove bit of dead code
* Respond to feedback from @bmccandless
* hosted, update order to look for config file.
The app now uses a local config.yaml file bundled with the artifact
(if present), if it exists, then looks in the CXG_CONFIG_FILE
environment variable. This is the reverse of previous behavior.
The purpose of this change is to move away from using the
config file on s3, since that could lead to problem where an older
version of the app uses a newer version of the config.
Also in this PR:
1. Changed documentation around dataroot, to describe the posibility of using lustre.
2. Added a few improvements around the secret manager region name. If we use lustre for dataroot and a local config file, then we will no longer be able to
auto determine the region for the secret manager. I plan to start using the
environment variable option for hosted cellxgene.
* small edit to README
Co-authored-by: Severiano Badajoz <sbadajoz@chanzuckerberg.com>
* add unsafe-inline directive to style-src
* debugging - turn on csp reporting
* revert reporting only csp
* do not inline JS and CSS in build
* enable HTTPs only when in production mode
* remove debug printf
* fix clean target
* revert force_https removal
* Return empty colors for .cxg v0.0 files
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1440
The CxgAdaptor.get_colors method currently assumes that the .cxg file has
cxg_group_metadata. As a result, the /api/v0.2/colors endpoint always fails for
.cxg v0.0 files.
* Add test fixture
* Add user-defined category-label colors
Fixes https://github.com/chanzuckerberg/cellxgene/issues/1152
As described in https://github.com/chanzuckerberg/cellxgene/issues/1307
* Respond to feedback from @bkmartinjr in nodejs
* Respond to feedback from @bkmartinjr in python
* Add tests to the server module
* Autoformat python, run linter
* Make colors_get error handling specific
* Respond to feedback from @bkmartinjr
* Respond to feedback from @bkmartinjr
* Fix whitespace
* Fix python lint errrors
* Update documentation
* Add --disable-user-colors option to launch and cxgtool.py
* Fix python formatting
* Rename '--disable-user-colors' to '--disable-custom-colors'
* hello world
* stuff
* successful build
* updates"
* maybe a basic example
* simplify
* reamde into dockerfile
* some more stuff
* Release procfile
* package.json at top levle
* don't release in procfile for now
* more package.json stuff
* copy assets
* merge master
* not in the relase phase
* revert not necessary
* pin gunicorn version
* reset common.mk
* Update package.json
Co-authored-by: Madison Dunitz <dunitzm@gmail.com>
* Improve diffexp for tiledb
- The rows from the A and B sets are gathered and processed at the same time. In this
way the matrix is only accessed once instead of twice for each tile.
- There is now a single thread queue that gets shared between all callers of the diffexp.
This will slow down work if diffexp gets too busy.
- There is a target_workunit amount of work given to each thread. Previously the
workunit was (rows selected * width of tile), which could be small. Now multiple
column tiles can be combined into one workunit. If the target is too small then
thread and other overheads may reduce performance. If target_workunit is too large
then the size of the gathered sub matrix may take up too much memory.
- add configuration parameters (max_workers, cpu_multiplier, and target_workunit)
* Specialize diffexp for tiledb
This patch adds a new diffexp algorithm which is tuned for tiledb.
This algorithm was written by Bruce and is adapted here to plug into the
current framework. The anndata_adaptor still calls the original
algotithm (which was move from diffexp.py to diffexp_generic.py).
The cxg_adaptor now calls the new diffexp_tiledb version. Some
code is shared between the two.
This is part 1 of the diffexp for tiledb. Further tuning and
global throttles are still needed.
A script to run and time diffexp with various options is also
added: test/run_diffexp.py.
* s3 region should have a single config param
The s3 region can also now be automatically determined to further
reduce errors.
This patch also fixes a bug with order of handling the config params.
The tiledb config needs to be fixed before attempting to load
(need to handle_adaptor before handle_single_dataset)
* warning on maxCount for diffexp
* cleanup logging
* clarification
* make the limits configurable
* make diff exp limit work
* danger!
* remove debugging code
* fix merge with master
* fix unit tests
Co-authored-by: Colin Megill <colinmegill@gmail.com>
fixes an issue with "cellxgene launch" which had a bad interaction between
command line parameters and config file parameters.
Now, the config files are applied first, followed by the parameters that
were provided in the command line.
There is also now a check that each of the config attributes is type checked.
