Commit Graph

113 Commits

Author SHA1 Message Date
Bruce Martin
5ea9ac8a9e Add provenance header to labels CSV file (#1041)
* add last mod time lookup to data locator

* store data locator in Driver

* save metadata header in labels csv

* lint

* fix tests

* change datetime format to second precision
2019-11-14 12:33:48 -08:00
Bruce Martin
238b7420f7 update deprecated Pandas API (#1021)
* update use of deprecated API

* update pandas depencency
2019-11-08 09:49:52 -08:00
Bruce Martin
823d819eee allow user to save an empty label dataframe (#1012) 2019-11-04 14:14:49 -08:00
Bruce Martin
711f3b7048 Add support for anndata backed mode (#943)
* initial cut at backed mode

* make flask multithreading conditional on debug flag

* update X access to support backed mode

* lint

* improve help message for backed mode

* fix tests

* add MatrixProxy to normalize supported matrix types

* add FAQ entry for --backed

* remove use of matrix.T

* clean up

* add ability to disable diffexp from CLI; add hueristic to detect likely slow diffexp calculation, and warn user

* fix tests

* do not print diffexp speed warning if diffexp is disabled

* tweak wording of diffexp speed messages

* add FAQ entry on --disable-diffexp

* revise heuristic for warning about slow diffexp

* use quick tooltip delay on diffexp button
2019-10-08 11:16:07 -07:00
Severiano Badajoz
44cb276cdf Add configurable about this dataset links (#907)
* add about arg

* add simple url validator

* attach about link to config api

* add links to configDefaults

* add conditional link in top left and menu item

* whitespace

* change to lower case

* move --about arg before click.command()

if this fixes it I have no idea why

* change link>URL

* be more descriptive about URL

* Make error more explicit

* refactor attach_data to accept about

* format

* change icon

* add trailing parenthesis

* whitespace
2019-09-25 19:23:49 -07:00
Bruce Martin
109c9e70ec Add obs index to label file (#928)
* remove debug logging

* load and save obs/row index in label file

* lint

* update tests
2019-09-19 09:16:01 -07:00
Bruce Martin
3660a6cc27 Experimental - manual annotations (#837)
* icons, partway

* redux for values

* onChange

* cancel

* annotations lifecycle for category names

* copy categorical

* edit category

* add Dataframe.withColsFrom

* render user annotations; default add/delete annotation category

* add label name to actions

* category name edit

* error checking improvements

* change schema field isUserAnnotation to writable

* always have an unassigned label; implement delete label

* implement add new label and edit label name

* label current cell selection

* fix select exact bug in crossfilter

* clean up categorical reducer

* fix tests

* remove debugging printf

* implement subset/reset for user annotations

* undo redo support for user annotations

* remove duplicate button from categories

* add modal

* remove obsolete duplicate annotation reducers

* remove old debugging printf

* connect modal to annotation create and dup

* initial full-stack wiring

* finish up end-to-end wiring

* fix existing unit tests

* fix pytests to match new schema API

* remove debugging printfs

* add label file rotation

* remove obsolete comment

* add fbs encode/decode tests

* add tests for writable annotations

* simplify code

* fix hashing bug with FBS encoding

* lint

* fix smoke tests

* improve error checking in Dataframe.withColsFrom

* add unit test for Dataframe.withColsFrom

* add unit test for Dataframe.columns and Dataframe.renameCol

* fix bug in FBS encode, add better error checks, refactor

* add FBS encode/decode test

* add clarifying comment

* clean up action type names; fix state inconsistency in crossfilter update

* change autosave timer to 2.5sec

* sort categorical metadata render order so it remains consistent

* add temporary autogenerated label for add-new-label operation

* fix hover-over label menu interference with cell highlighting

* remove debugging code

* add missing reducer cases & fix typo

* make dataframe memoize more general purpose

* add dev mode for annos

* fix error on select duplicate

* handle zero occupancy categories

* correctly maintain unclipped AND clipped world

* correctly handle zero length FBS matrix and label files

* ensure all writable categorical schema contains an unassigned category

* handle case where building occupancy stack for category with no members

* dialog for creating label, disable button if duplicate or empty

* visually separate writeable

* edit category

* fix edit category name

* remove debugging code

* fix edit annotation label

* visually define unassigned, change options

* Pull in requirements.txt from `master`

* label currently selected cells

* duplicate label

* lint

* fix pytest merge issues

* rename --label-file to --experimental-label-file

* remove debugging console log

* spelling error fix; fix bug found in PR review.

* lint
2019-09-18 07:33:41 -04:00
Bruce Martin
20317fd08f Add URL data locators to launch sub-command (#920)
* initial commit of URL support for launch

* lint

* modify tests to use new data locator

* add locator unit tests

* fix typo in faq

* more lint

* update faq per PR review
2019-09-15 09:01:53 -07:00
Charlotte Weaver
3d98797d8c Installer (#840) 2019-07-12 12:29:22 -07:00
Charlotte Weaver
5effe4bbbb removed --diffexp cli param (#826) 2019-06-26 14:46:58 -07:00
Charlotte Weaver
d5deb1579f [EASY] fix max-category-items (#813)
* fix max-category-items

* match default for max category items

fe had 1000, be had 100
2019-06-13 15:45:43 -07:00
Bruce Martin
a6142bdf93 improve graph scale and centering (#796)
* add gutter to embedding canvas

* improve layout scale and translate

* fix lint

* pin tables to version 3.5.1

* fix lasso coordinate smoke tests
2019-05-30 13:31:41 -07:00
Bruce Martin
3dc45d6330 do not hard-wire column names in annotations (#785)
* enforce column name uniqueness for obs and var

* parameterize the column name containing obs and var user-readable names

* use the new annotation index value from schema

* update f/e unit tests

* PR review suggestions

* lint
2019-05-24 21:00:54 -07:00
Bruce Martin
2357d0c1b8 layout change UI (#776)
* add layout to schema

* add layout choice action and reducer

* multi layout UI

* update unit tests

* add missing file

* update test schema

* fix duplicate test id

* fix tabs

* PR lint

* fix pytest
2019-05-22 13:21:33 -07:00
Bruce Martin
efa1709158 add multi-layout support to back-end (#766)
* add multi-layout support to back-end

* remove obsolete code

* temporary code to apply heuristic choice of default layout

* fix tests

* update python tests

* more py lint

* PR review changes

* more PR lint

* PR lint
2019-05-16 14:49:22 -07:00
Bruce Martin
b9a1e30652 large file size guardrails (#763)
* large file guardrails

* fix lint

* PR review

* remove unused import

* use standard slice for CSR

* revert change
2019-05-13 18:13:16 -07:00
Charlotte Weaver
8a94b1e086 fix #754 (#755) 2019-05-07 12:43:20 -07:00
Bruce Martin
b521a17ffd improve column access speed for sparse matrices (#742)
* improve column access speed for sparse matrices

* add FAQ entry about data format performance

* add note about using --sparse flag for prepare command

* clean up for PR review

* Update docs/faq.md

Co-Authored-By: bkmartinjr <bruce@chanzuckerberg.com>

* improvements to big data faq
2019-05-01 15:44:01 -07:00
Charlotte Weaver
9f10d8095a GUI app (experimental) (#730)
* add default config

* first pass

* flake8

* cleanup

* first pass at using qthreads

* cleanup

* WIP

* better error handling

* improved UI

* bugfix

* fix merge bugs

* import order

* cleanup

* make gui requirements optional

* pr review requested changes

* Update server/gui/main.py

Co-Authored-By: csweaver <charlottesweaver@gmail.com>

* pr review request

* qt child class methods -> camelCase

* whitespace
2019-04-30 12:52:15 -07:00
Charlotte Weaver
ea187f48e0 add scripts from cli (#680)
* add scripts from cli

* add warning when including scripts

* flake8 fixes

* confirm scripts injection
2019-04-22 12:30:57 -07:00
Charlotte Weaver
9c6273eb94 core library (#711)
* move app creation to function

* create engine without load

* flake 8 fixes

* cleanup original scanpy test

* add default config

* handle missing data

* test data changes

* unify update

* load data isn't static anymore

* make app a class
2019-04-22 12:24:07 -07:00
Charlotte Weaver
73852dd6f4 use flask's json (#703) 2019-04-09 12:59:09 -07:00
Sidney Bell
c9a56fa73a Update scanpy version (#688)
Update to version 1.3.7
2019-04-04 12:16:30 -07:00
Sidney Bell
16f93397ae Support diffmap and phate layouts. Explicitly handle embeddings with >2 components. (#662)
* Add diffmap and phate to supported embeddings

* Explicitly pull the first two components of any given layout
2019-03-21 19:06:05 -07:00
Charlotte Weaver
e875ed739b More informative out-of-memory error (#644)
* more informative error message: memory error

* flake fix
2019-03-12 14:19:00 -07:00
Colin Megill
a0f54b4871 Add menu (#631)
* add menu

* Added library versions to config

and tests

* add template version number
2019-03-06 11:56:46 -05:00
Bruce Martin
57c4e9ff33 Flatbuffer cleanup (#598)
* dead code and route removal

* more dead code cleanup

* fix scanpy_engine tests

* lint

* add missing catch in filter parsing

* update scanpy NaN tests

* more fbs tests and dead test removal

* remove forced default for content type negotiation

* bit of cleanup

* more fbs test cleanup

* lint

* remove swagger

* swagger cleanup

* lint

* correctly handle lack of templates

* more dead code removal

* remove unused files

* fix dev build

* lint
2019-02-19 08:50:29 -08:00
Isaac Virshup
0f8d7a55de Set API path based on access address (#568)
* Make api paths relative

* Remove request import

* Set publicPath to be relative
2019-02-11 09:01:37 -08:00
Charlotte Weaver
10693b08cc Add __init__ file so fbs can be imported (#562) 2019-01-17 17:11:31 -08:00
Bruce Martin
5d60505407 remove --nan-to-num CLI parameter (#548)
* remove --nan-to-num CLI parameter

* factor tests better

* lint - remove unused variables
2019-01-10 15:03:03 -08:00
Bruce Martin
b90447c387 binary wire format with flatbuffers (#509)
* first flatbuffer schema

* do not lint auto-generated files

* add flatbuffers package

* add flatbuffer module

* wire up /data/X/T route

* use flatbuffers for matrix data fetc

* clarity and comments

* add flatbuffer layout route

* clean up obsolete code

* fix tests

* move flake8 config to setup.cfg

* add comments

* lint

* rework layout routes for fbs

* add more type support to fbs

* lint

* add flatbuffer support for annotations

* function name improvements

* fix botched merge with master

* remove unused import

* route cleanup for flatbuffers

* rename function for clarity

* add missing globals to Jest tests

* fix client JS tests

* fix routes for Python tests

* comments for clarity

* non-finite floating point hardening

* more non-finite number handling

* lint

* fix tests for summarizeAnnotations

* harden diffexp calculation against FP errors

* cleanup unused code

* lint

* add encoding tests for flatbuffers

* application type specified as strings

* fix spelling error

* improve variable names

* add note about documentation gap

* rename FBS DataFrame to Matrix
2019-01-09 14:26:05 -08:00
James Taylor
f2eb2cad82 Use url_root from Flask rather than hardcoding hostname. (#520)
Allows running in a container on a remote host, should also allow running
behind a proxy with a url prefix.
2018-12-19 16:20:26 -08:00
Charlotte Weaver
f0f7200f0b Move jsonification to engine level (#511) 2018-12-18 21:14:24 -08:00
Charlotte Weaver
83154577e4 Black -- formatter for python (#508)
* Add black

* use black to format code

* Black version
2018-12-12 09:44:47 -08:00
Charlotte Weaver
5c6cc597c8 Default warning in debug mode (#497) 2018-12-05 13:38:34 -08:00
Bruce Martin
3bfeadc2b9 improve handling of non-finite floating point values (#490)
* improve HTTP error reporting

* generate standards-compatible JSON

* add --nan-to-num work-around for non-finite floating point values

* lint

* update tests

* correctly set Infinities to min/max

* REAMDE update for --nan-to-num

* define constant for repetitive warning message

* clarify where NaN errors will occure
2018-12-04 14:56:16 -08:00
Bruce Martin
af0d1f6fb2 issue #480 workaround (#484)
* only load annotation var names

* remove incorrect usage of var annotation data

* temporary workaround for issue #480

* lint

* issue warnings only once per item
2018-11-29 16:33:21 -08:00
Charlotte Weaver
76c0cbf16b loose -> lose (#479) 2018-11-28 13:07:10 -08:00
Charlotte Weaver
455d987398 Add custom format for warnings (#477)
[cellxgene] Warning: <message>
2018-11-28 12:47:11 -08:00
Bruce Martin
933fac5fff Allow float64 to down-cast to float32 (#472)
* warn if annotations will down-cast to float32

* correctly handle falsey data values
2018-11-27 11:17:30 -08:00
Bruce Martin
138d30909a improve type handling for non-string annotation data (#465)
* improve type handling for non-string annotation data

* improve clarity of code
2018-11-26 10:32:29 -08:00
Bruce Martin
141f802824 differential expression improvements (#452)
* add cutoff for low expression genes in topN selection

* remove debugging printfs

* change cli param name for diffexp cutoff

* change CLI param name

* second try at diffexp - using lfc sort with pval cutoff

* use lfc cutoff

* update comments to match code; cap p-value adjustment to max of 1

* lint

* explain diffexp in readme

* add link

* add diffexp-lfc-cutoff to test config

* update test to match revised diffexp spec

* fix latent bug in GET arg handling that was breaking tests

* lint

* comment cleanup

* fix variance overestimation so it is symmetric

* lint
2018-11-16 14:44:56 -08:00
Bruce Martin
00a68276a2 diffexp performance & UX improvements (#431)
* new diffexp REST API spec

* new diffexp REST API; faster diffexp and dataframe slicing

* first draft of fast diffexp

* convert variance calculation to two-pass method

* lint

* update front-end use of API

* fix typo in spec

* disable content compression

* catch index filter format errors

* clean up of dead code

* resolve PR review comments
2018-11-14 12:51:24 -08:00
Charlotte Weaver
7d5c054b90 Launch validation (#414)
* WIP

* Cleanup

* Validation

* typo

* Max category limit 100 -> 500
2018-11-07 13:01:13 -08:00
Bruce Martin
2d5bc9d0c9 Performance work, plus fix #405 (#406)
* remove memoization

* update to flash 1.0.2; turn on threading

* stand-alone helper routines for array slicing

* fix issue #405

* reset diffexp state when world changes

* performance work in dimension creation; fix world slicing bug

* update tests to match new state mgmt api

* update flask

* do not make dimensions for useless annotations

* update test to match optimizations
2018-11-05 13:57:58 -08:00
Jeremy Freeman
0c8a07ac13 CLI refactor (#396)
* refactor cli to improve ux and enable easy incorporation of prepare as a subcommand

* switches to use click, which removes some boilerplate and gets us some improved ux for free

* changes the entry point for the cli

* changes the name of the browser option to --open and makes the default false
2018-11-02 10:52:50 -04:00
Bruce Martin
b181751493 add --obs-names and --var-names CLI params (#371)
* add --obs-names and --var-names CLI params

* fix lint

* performance improvements in scanpy engine

* fix lint

* fix typo

* correctly handle sparse formats in diffexp

* fix diffexp and 1d slicing

* diffexp uses t-stat, not pval; clean up arg handling

* make _slice a static method

* revise scanpy tests to match new API
2018-10-30 14:07:38 -07:00
Charlotte Weaver
26d222cd75 Move computation options to app (#386)
no need to import scanpy engine for CLI help so speeds up time response
2018-10-26 17:07:21 -07:00
Charlotte Weaver
6c23a72e5f CLI renaming and phrasing (#385)
* Minor naming and phrasing changes from UX review

* category-selection-limit -> max-category-items
* Indicate load may taking a long time
* program -> command (for launch, prepare)
* debug -> verbose
* flask-debug -> debug

* Developer mode for debug

verbose on
open browser off

* move examples from epilogue to prefix
2018-10-26 15:59:34 -07:00
Charlotte Weaver
16b0741a6c Hide stack traces unless in debug mode (#384) 2018-10-26 13:36:53 -07:00