From 2df93d6d94a3afe58602a36da30f81b9aa6542b5 Mon Sep 17 00:00:00 2001 From: Sidney Bell Date: Thu, 13 Jun 2019 16:55:09 -0700 Subject: [PATCH] Add `prepare` example and update demo datasets (#810) * Update example datasets w/ pbmc3k and tabula muris * Add `prepare` overview and example * Add S3 data links * Incorporate PR feedback & copyedits * Switch to letter pointers * unix line endings * path --- docs/data.md | 121 +++++++++++++++++++++++++++-------- docs/prepare-cmd-example.jpg | Bin 0 -> 9049 bytes 2 files changed, 93 insertions(+), 28 deletions(-) create mode 100644 docs/prepare-cmd-example.jpg diff --git a/docs/data.md b/docs/data.md index 0ebfdb97..721232ca 100644 --- a/docs/data.md +++ b/docs/data.md @@ -1,29 +1,94 @@ ---- -layout: default -title: data -description: Data ---- +--- +layout: default +title: data +description: Data +--- + +# Using `cellxgene prepare` + +#### What is `cellxgene prepare`? + +`prepare` offers an easy command line interface (CLI) to preliminarily wrangle your data into the required format for previewing it with `cellxgene`. + +#### What is `cellxgene prepare` _not_? + +`cellxgene prepare` is not meant as a way to formally process or analyze your data. It's simply a utility for quickly wrangling your data into cellxgene-compatible format and computing a "vanilla" embedding so you can try out `cellxgene` and get a general sense of a dataset. + +#### What input formats does it accept? + +Currently, we accept `h5ad` and `loom` files, as well as `10x` directories, and are hoping to accept more formats in the future. + +While we'd like to support quick conversion from seurat and bioconductor, these packages don't currently output a python-parseable intermediate file type. In the meantime, you might check out the [converters](https://satijalab.org/seurat/v3.0/conversion_vignette.html) that are under early development. + +#### What can `cellxgene prepare` do? + +`prepare` uses scanpy to: + +- Handle simple data normalization (from a [recipe](https://www.pydoc.io/pypi/scanpy-0.2.3/autoapi/preprocessing/recipes/index.html)) +- Do basic preprocessing to run PCA and compute the neighbor graph +- Infer clusters +- Reduce dimensionality to generate embeddings. + You can control which steps to run and their methods (when applicable), via the CLI. The CLI also includes options for computing QC metrics, enforcing matrix sparcity, specifying index names, and plotting output. + +**To see a full list of available arguments and options, run `cellxgene prepare --help`.** + +#### How do I use `cellxgene prepare`? + +As a quick example, let's construct a command to use `prepare` to take a raw expression matrix and generate a processed `h5ad` ready to visualize with cellxgene. + +We'll start off using the raw data from the pbmc3k dataset. This dataset is described [here](https://icb-scanpy.readthedocs-hosted.com/en/stable/api/scanpy.datasets.pbmc3k.html), and is available as part of the scanpy API. For this example, we'll assume this raw data is stored in a file called `pbmc3k-raw.h5ad`. + +Our `prepare` compose our command looks like this: + + +Let's look at what `prepare` is doing to our data, and how each step relates to the command above. You can see a walkthrough of what's going on under the hood for this example in [this notebook](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-prepare-example.ipynb). + +**1 - Compute quality control metrics and store this in our `AnnData` object for later inspection (A)** +**2 - Normalize the expression matrix using a basic preprocessing recipe (B)** +**3 - Do some preprocessing to run PCA and compute the neighbor graph (auto)** +**4 - Infer clusters with the Louvain algorithm and store these labels to visualize later (auto)** +**5 - Compute and store umap and tsne embeddings (C)** +**6 - Write results to file (D)** + +# Example datasets to use with cellxgene + - -# data vignette: how to use cellxgene prepare - -#### coming soon! - -# example datasets to use with cellxgene - -### Examination of single cells from primary human pancreas tissue -cells: 2,544 -tissue(s): pancreas -data: [Human Cell Atlas Data Portal](https://prod.data.humancellatlas.org/explore/projects?filter=%5B%7B%22facetName%22%3A%22organ%22%2C%22terms%22%3A%5B%22pancreas%22%5D%7D%2C%7B%22facetName%22%3A%22project%22%2C%22terms%22%3A%5B%22Single+cell+transcriptome+analysis+of+human+pancreas%22%5D%7D%5D) -paper: [Enge, Martin, et al.](https://www.cell.com/cell/fulltext/S0092-8674(17)31053-X?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS009286741731053X%3Fshowall%3Dtrue) - -### Tabula Muris -cells: 53,800 -tissue(s): muscle, pancreas, bone, large intestine, heart, brain, fat, mammary gland, tongue , diaphragm, bladder, spleen, thymus, lung , skin, liver, trachea, kidney -data: [Tabula Muris Data](https://github.com/czbiohub/tabula-muris-vignettes/tree/master/data) -paper: [Tabula Muris Consortium](https://www.nature.com/articles/s41586-018-0590-4) - -### Transcriptional profiling of 1.3 million brain cells -cells: 1,330,000 -tissue(s): brain -data: [10x Genomics](https://community.10xgenomics.com/t5/10x-Blog/Our-1-3-million-single-cell-dataset-is-ready-to-download/ba-p/276) +**To download and use these datasets, run:** +`curl -O [URL]` +`unzip [filename.zip]` +`cellxgene launch [filename.h5ad] --open` + +### Peripheral blood mononuclear cells + +Healthy human PBMCs (10X). + +- Source: [10X genomics](https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.1.0/pbmc3k) +- Cells: 2,638 +- File size: 19MB +- [Raw data](http://cf.10xgenomics.com/samples/cell-exp/1.1.0/pbmc3k/pbmc3k_filtered_gene_bc_matrices.tar.gz) +- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/pbmc3k-processing.ipynb) +- Download: `curl -O https://cellxgene-example-data.czi.technology/pbmc3k.h5ad.zip` + +### Tabula muris + +20 organs and tissues from healthy mice (Smart-Seq2). +Rich metadata and annotations. + +- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/237446v2) +- Cells: 45,423 +- File size: 174MB +- [Raw data](https://figshare.com/projects/Tabula_Muris_Transcriptomic_characterization_of_20_organs_and_tissues_from_Mus_musculus_at_single_cell_resolution/27733) +- [Processing](https://github.com/chanzuckerberg/cellxgene-vignettes/blob/master/dataset-processing/tabula-muris-processing.ipynb) +- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris.h5ad.zip` + +### Tabula muris senis + +22 organs and tissues from healthy mice at ages 3mo, 18mo, 21mo, and 24mo (Smart-Seq2). +Rich metadata and annotations. + +- Source: [bioRxiv, CZBiohub](https://www.biorxiv.org/content/10.1101/661728v1) +- Cells: 81,478 +- File size: 3.9GB +- Raw data [geo link coming soon!] +- [Processing](https://www.biorxiv.org/content/10.1101/661728v1) +- Download: `curl -O https://cellxgene-example-data.czi.technology/tabula-muris-senis.h5ad.zip` diff --git a/docs/prepare-cmd-example.jpg b/docs/prepare-cmd-example.jpg new file mode 100644 index 0000000000000000000000000000000000000000..57af0f30198f32155e67d0a434b3627e3b64cbd9 GIT binary patch literal 9049 zcmb_>2UHYUw{A5cO-2xq)MUv)G7=<77Dm--qKYB_0s(+B`UGwk z0Rx5CHZK7{T^-;A000NT0FeTiXbXfsfnXZ|>kk_MSka#VfSC#Uvo{mtzj`rKGBN+l z2E>03yx9T7G_4(79bK#)ofrgp?*n3TD(YB2XG0%<*!TW0Z=)dLQ&B)A)}}9R(zhRS z-ZyOk2@W8RnSu#o0x(EGm?WT^4uBr*(Jjy)#~-($FCYv|tXtS%99%qn^njY%XxA|@ zF|aUi-TL_$kQcfiz#_Rt%E&K^eMjp#mU*qg>>Qjz!Xlz#55(mkDJUu_tElQc)z#BCFf@W%SiZEfwy||_b$jLh+QT#8ZD3Gv zNN8AW+;8y-iAnF1vvYFu@(YlKMU_?6HMMp14UO%;cYN;b>h9?s8AXkaPfSitFD@;w ztgfwZY;GNVKRh}t%U&#Ii>`z=X zfINVMjSa@Wg#!kIadC0*2*?Nt@bL+#NbeAnL8$5JAk?(93@p5C3{2e2w6yG^9NhQ$ zg@lCY*&axV2}tq^3JKhxCjsH&;u7E!P!bYS3NX?#3jB|cn>n;B#c!qo0!$EEOqe8q zG;mt*ia;ixiq=g?8fAW5jiur zcOg70XfI*>#c4((t4pJ^fu3;7X!Ua1>)Q&AUGB^G)C%nwM01%0{>1cYwl*}32O_PM zjJUDby4@yG+^(1hnS{kKMf}e%Ya7GNOKZaIX~0yUy1Z@S%;`74I{CoHBJ)JE3SM-$ z1C&t~55A%(xH&ogGGz(>A^{~irKj~A#w>@iyi-3H3aOcS`Nbxk8%%_Hb?7YrRylVC3bWFM``5v%-ww{UPQ_e&zCusA5vuDYkycJ?*oItF3j{9ldi?Qha2# z-P!6(K_r|$c4`&p(?#x!_a$4N$=2!4vM2ZjP5$=?K3(g@fTZK7flc>%DmF4lA8)DR zB-Bdd==16->WIDxcR~J$j|wcedZ?HIJ*k4Qy`3FcnZunOCWZ7lX3D*~%1e9lKB$e2%@x#1ru2f#~QnCb^ugn6@D6BE`6+NO;%J8hFhp9UKU&rn7y zIwn7Is|(JpN3$|biC07D+qskwLrDY98QOQiYauI(Wmfwg(w_E(CkkvZ=zY# zaDW}3v!y*d?Ugbk>;RA6*T7{K-< zRt0v#!?D%i+hO+Zbgp(R+FUt+?r;7$QiO03qp|A>?1XYp?qry7SzbK3&+A3sPET`- zxZ~1tFzGzO@Sw?b)+JnqW*+L}8O47_wSv%t9q(MVwC9y2Pbll(yBtb7=_Hgnk1Q&r zm?O_{#!#BR8<3vHj%n3rzwDHQju~{cx_HKq=RM5Io>mYv@G5;N=oieu8E%deZA_a_ zd061cv$j`d!zJm+E4b!;$o4gR2&~(3<)bRvTE1zbu$EJ#ep2gymSqg1rWwj2+l@X; zd%gLcV0`@~KN}jlFha0#z0|V z1QY^c1HCGP<;?VHHo{|LCk;}GCjkb(!(ar()qthdT-@OEYIJ;!I{NJS9B<5D4Lr-W@-hZy!m|ythV+Lo{qpnJ9v6j#$>8v(lAzj-Kz_& zX*5V+b$eJ@fwY#h^`kP+U8{GUM$Tu8Imz?RP=+^wub|*smvcRtu^RIF`bP1f z8-UB;f^>RElfZJ!OMD?N!xq?;>>8X$LwPn*pQ5f|(t=t^XQIH_o5l{DlM5L;xtg_t zV=`x`?mRFj0!tx5^H74zIVt2J{lrIU>Q$a&{BFs|3@Bb!gZ26t9zBM9rc6>0L>?es zeR|}jVS?gaXfmguaxi=F+~Y2eUy?Z>SZ_`Z>|Azsx}0mRNu$>|BEQ9reK>wyFJWD~ z-d|yvJA}2Z%N}M_m6s;3IVQ|ufr9Yt@EMT>lE_%%y-#i4ie1tQxun+HNLG(+QUnl1!c@-5nhR95|#J) z7>Tx?7CkVG6=iRl#E@I3y0qhkDCu~Je956i6-qQ8OGXA1cmk8?bjcXy&54h$h}~bN zXFm%RPCx!x5xXXmvuu#If+pD}Hmvi4u_IIgu%l)2v^)*Mz^>4#j9vqWzFYzec7MTW zX-pZ2bkgPF=yVi+Pp4h*$g$Lu^{3Q6(-t%C1wE2{{E8OlPdVCXy}ljqXaslEP1New zMVI+hRZk#7Do)EjJej?)m-MNVfs`lx*z)q!LMtf4Xo+=9n1<4kM3}g?%7%2lIMAev zDN>v$_`wa}J6@uH_{=V-U80Q{Cd|qzNm^zSJ_LTm@{&rg`>@p@K>{JUslrCtN{OltR1zMF3;sRB8kJVNi1tx z1?Fj<&EbYEiBi!XXw_D^5eExHA^mMAo;M%b7oR45AsLFy!S)t){% z()K(*w()bL2~8oBNInM(CjVK<)Ad?x!G60R9>*HuTU={%c;D_<#27ZrQbID}ZJz;0 zkBF;XDeL*`8tw0h6oik!^u&&U)h_l-*LWdsPT?8Rx*{XWw{X31Zg0d_XuUadoSCO% zlMGt?0Kj|Wj$xf(kgFfzYiYT1+w-;^?}OcP`?A>6Cl~i|=dVx~3qERPg;nO(1q%)c)49>mTm^ti3&`vxgji!=C&UE`8K6AWFeI}MVup?;CW zy#Os!ho79!ZK=R8XG686EK8=LQS(Zxf<&HP--v$@>W%a6x=G_~>+ozB&CK^uQyv^L z)wd1Sl>+W&P^&$3(zrg>tBlC5F_>O0gWmu=xU}J^u0_(c++((9Z&Q1V(inDoo8fzG8W96$7sjl?FfTOLx^uZThz1h zMh(LGe&mMJpal3giB8LJiv~!?E^Fxsz}_QjTgF|A*eqbYs(g>HG*`EJG~&+Ta%I;0kr!dQE3;xni|_J_I(gwCps-I>JswoL|ScKRGoD zYz^4UMOg-xQQox#2s`KHn+m++5jyoPbPNcbnrD@tDUx_GZ<$mDJ&*O`mw{Nv49X{K z2v=h7YRGPz*n2ZK#SS1sa}C_BwakiQ>g-EsQ?1Gww4dGUA1_=?2}a%XomE^}Utyu-SF~+aNsRyZV zGU3}Y5dYbr_#DHh-5DOTD>|25$45J62zP`#Mb;9bBK}}y&#m~HHs6J_h}DW|m!p}k zQtE{A;*^=rpsUOXBMXD(8pc<~B{_A{B1$8^Quv5Eve*8TO%1wx<^5A<6DbQa4(KVCO=M@C~_Yyey?Z(vHo}fe z$&KL`zDtQ+JTo`9MO_e7?!XsN^ZD|uy6tLyG9Ms*Ohi2HD{-~A{WxspN!WpMS?N$A zSbF6^cLBx2ySocxNMlK#^b;T|jETu;uiqn7M_5t#KAbe0K35CryQMgW+2^F>T|9`M z!@7JitykWSED~Ea$6w~07+Kid1*TfhgBw8__D_nT1r=3|Y9W?0k{pLf6VX0IK&m(S zt>6&H)@trF)`X>zlWD^oL)F+%I?mBxjmK6kfII7|o^D8QI@F@bDlW45X*yINH&4UG zWr?0>cGi~O29bih#=k(!9YZJ6B|EcUwp=a*x*0l6)TvK(vJJC7fq$hz)yO)jNOUT( z6`qghk%8if-a#E-Ib>BoN{V&oqa1LS*%{A=iO-cQEc$s5HFGDe)7}6P)><-Dl{T*` z=V%h^bokW+FJija_gUjzXDsgjR;O@3OF4ud{zL#=2jNl-1J;Bx2xtf&&G!vAr|Uc_ zrbXxz$U=Wab6$#5WH^i@l?*=)(W)3K+~S|q`%DIsF_3{6$TS%A$t-8XyEZs!)aeE~ z6=SETm|jud^NBtpFIE67x2EA^LhqU|KHuu#q4?(7%`9DdFEYxeAm<&zO>O}!J>TV6 zz*=LSiN|4yXZTLxExviWK@=?v6VNkV58_@QDf^ZgYJQhd{sYybcJA#B{v zqAFbB{?*d5M2p~)0L6;hNb99)1(igLOdsmQPnZ3+!!l%My&OF1RqYWV(|i$H60NDt zglW&l={9B}=ea|pr<&LzRXGtadU!0vWF?(Hbh_U%pOlUlv-b3vA}p*7Z?`zaBT23G zJ{_;w$RwzDMTjv)itT9W(sqCBlhyWNQovNMypZ^GB6eAzp!nEH&X>#^qlP_`z$J5Y z=OvHP@MDR(<#d{f2Zb+2>=*snp*4@fs3g0`l;#|~pYQdQinC+RVgZ0eQ@TFi#tcfZH$Y`k*AM{7$`Egbq;SxwX_uLp}CHeA6>2W z2UmuC20Cma@1lrp2M!>pdC2~jMnQHLq&X(bw3*w|Kh+z%F|43)^MmcQWq)8dE7dZ8 z1w^U-6x)~HAK@*Q^ZXnCQ5jtFUJ(-~-#cN3Omo7*w#&k=`(azHMC9OXc+@77&xba=ATHgR~^>uH6tUB!* z;G>=|O|I=};=bJ|X(0D*UeJs+WjpZ=plh~2_wBqqn47O)+A^7;@2UHWZt2}e0gpx! z+F5P%xd&M?JQbm_8Z9r3Uvqx%1i>tqU~>2tY+pjR{Tcce;qWZc70|f{R<(497jA4*S2?brkMwC04XAuneQ8% zHNinUJiG;sv1G|hYKiswoZi}9`BFu7e=b$EPdisJkf^R9Dgu`sbuX|7Zry%tHC5Wv0ef6;;>}hm4P7eHB z4Esu=OVh@1+*Jc;6x-G%5N1~HkeHdW+}Sk~_|i3+Iw2c$Bpv^ByxO3qMQG{^vz$Cv z7xyzy5uIoSi=PmSeTJrfB9wvL)*JNcEvikloltj%MfYvjWYE8cf*&^PN&f;Ky%zr> zMTT0_I^or$9!vY^0g@P&PSt=bFFaeLLTv61tO zLV&TF;S&3f?D-KP(ugu(>tn`~W-ZH{DVx2I<6|#iO`=3iUV@v3|G=ib@n-EwY!5fZ zHchYds-l4H!0($Tr+@2oVa}9dTxSBZ5Pn^wq z^^ro+?cX^o%W#2c;yN-AYKQHIkE%#r>#B*FU!I-kTBu#2#u8$~qF#GVZ1_I*a^!q` z!M`cT)tJ}`ez`5VAr7xaxLDEuw$t((2KKH8U7DTf-}sE5n2h%@9u9q8+3(=YY8fTr zXPSEjpD*>Xm`)Stk;>Go4VB}J86wI^0IwQK|Gj8!Mc%y_XX>72n+kBijRb9Hlp#i! zM|PNMV#g&6#&Ja*!uh~bE{4BkjDMHCgRxQk{}8h; z*4t0?@9OcaFd0pAx5?yS3Xrozeb!jlb>*6QKVe)O$o%v}y&XdL(V~nQxK(KH&wOmB zBRdo)SDk#XAz@8r-D5TRS@Xc;phjjU-p9Kr%kej&P~>CXy!a=V=sW;E)BD)jN#v<7 z_K~2zHuibJ5C8LlX%!;{|ND75@E%O5swaO{hp+ISYq7eLe)R(5z3*!-W0r_{`JXZ!K_U}o!-GG z;l^h8viW8mt%WDt;0E}6k+^bE>EdAnlyvQtEcPCod@r`u{|PtI05 z3TNMqkSnC>#>^bdV@amV;Mr)3?WgwoRCn9+ic9cs!h|q*6qaY$kydk zAL~ZH1%OWU-%hFvJ22txjJwaQVN1p}bv1!64R!AzD7wrZ*3|u#QNXBre=PUzsebIE z%6{RVG{eN*H<`p=(7#=&!?bu=(x4cI0MU=Ye2V^cf2wcv+bp!s2;4fj9y-~~IaZ*#G^>Aq>nD$iK=+V$O=B5iH z!_GR;yI}n87dR|l&USyxDbn@A4pICr2{y)B&(PcE{A0mtgQGBiM)5<@4DvPwKY`bX zESUa%iO=6NA+#f;hL3AxY%CNpdE3AumF}x5&ym!_5x8(mD!e;4-l<_Gy`C;*bwNG&a=>jSp>Qqf;LAQt%#e#K0U)860HUy_6+>cAVHSZ32>*i9@#V?)eY&3cY6P~yAMrODr$H`gaBUvTI~db}fZ{(Tj*z`2kWKo-qaOMqi-HOw?Qx>LbCBzQm?QP z3+CSBrAi0Hc>vFUryO)Yjwkg(yX`g6xEmGEQDF@a=edj1!~VV)2{#(gnB+a1(o-dr zk2tVaZE9ySc5KpowRfE=i<+izw{GN}{~Bpmo&SKr$r{st%%NFnX