Add documentation for Dask/Multiprocessing support and Xarray accessor - #878
Add documentation for Dask/Multiprocessing support and Xarray accessor#878rhugonnet wants to merge 4 commits into
Conversation
|
@belletva @adehecq @atedstone @adebardo @marinebcht @erikmannerfelt @ould-a This new documentation draft is ready for your review! 😄 The link is here: https://geoutils-rhugonnet.readthedocs.io/en/add_accessor_daskmp_doc/feature_overview.html I suggest you start by reading the new "Feature and scalability overview" in the "Getting started" section, then move on to the "Scalability" section which essentially contains all the novelty. I also added this new "Cheasheet: From GDAL" page to help users make the link. We could also add other Python packages there (i.e. table that @remi-braun started)? Very happy to see this in a near-finalized stage after so many years of us working on it! 😊 |
|
Hi @rhugonnet, I have some comments, but honestly it's not much:
And again: such a wonderful job |
|
I forgot a point :
|
|
Thanks!
|
And again, congratulation for your amazing work !! |
|
I just finished my 2nd round of reviews. Very nice documentation ! 😍 Thanks for all the effort you made to explain the concepts and write a clear documentation!! 🙌 Replying to your main questions below:
If not too much effort, yes I would encourage going in that direction! It is so much easier when different tools use the same naming convention and I like that we "accidentally" end up with most of the same terms 😆 I actually suggested having an
Since I am still a big user of the Geoutils classes, I would keep those examples. Also a lot of our base users are used to this system now, so I think it is too early to completely change the structure. Something to re-evaluate later?
Maybe we could show the same example with the 2 approaches? Using tabs, like in xDEM CLI documentation could be a nice way to avoid long pages? |
|
Thanks for the feedback! 😉 |
|
Reminder: Last week for feedback, then I'll consolidate and merge! |
|
This is fantastic - thank you for all the hard work on bridging xDEM with Xarray! 🚀 The documentation is very helpful and really nice to see supported operations in the Table summary 🤩 After browsing the documentation I have a few comments:
|
|
I did not find how do you set a mask in a Xarray (ds.rst.set_mask do not work) and how to you cast an Raster to an Xarray ? |
|
Thanks a lot @friedrichknuth! 😉 On @marinebcht's question:
No masked arrays through Xarray, so the arrays are forced to floating type to support NaNs instead. So an equivalent would be simply: For conversion: |
|
Hello, I don't want to put pressure or whatever, but @rhugonnet do you have a release date in mind? 😇 |
Yes, sorry, I was busy with deadlines on research projects the past weeks 😅 |
| For a raster output, one typically wants to write to file lazily to avoid loading it in-memory: | ||
|
|
||
| ```{code-cell} python | ||
| # ds_reproj.rst.to_file("reproj_rast.tif", compute=True) |
There was a problem hiding this comment.
Hello, I was trying to use this function but it does not work. Just saw the commented line. It is in progress ? Thanks 😊
There was a problem hiding this comment.
erratum !! Just saw your corrected it in this pr :) thanks :)
There was a problem hiding this comment.
I tried to use this, but I add errors :
ds = gu.open_raster(gu.examples.get_path("exploradores_aster_dem"))
ds.rst.to_file("myraster_input.tif")
gives ValueError: Key '_FillValue' already exists in attrs, and will not be overwritten. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.
ds = gu.open_raster(gu.examples.get_path("exploradores_aster_dem"), chunks={"x": 200, "y": 200})
out_res = (ds.rst.res[0] * 2, ds.rst.res[1] / 2)
ds_reproj = ds.rst.reproject(res=out_res, resampling="bilinear")
ds_reproj = ds_reproj.compute()
ds_reproj.rst.to_file("myraster_reproj.tif")
gives rioxarray.exceptions.InvalidDimensionOrder: Invalid dimension order. Expected order: ('y', 'x'). You can use DataArray.transpose('y', 'x') to reorder your dimensions. Data variable: concatenate-450f8600ebabf585198dedfd53dc348d
I need to move now so I can't investigate. So I put this here in case...
Didn't fully get to it, but I got everything else out of the way. So I can merge this when I come back from holidays in ~2 weeks! 😉 |
7 participants
This PR finally adds the documentation for the Xarray accessor
rst, as well as the Dask and Multiprocessing support through chunked implementations that we have been steadily developing for the last 2-3 years! 🥳Thanks in particular to @vschaffn @ameliefroessl for their big contributions at various stages!
Link to the new doc (landing on the Scalability page with schematics that was a bit of work!): https://geoutils-rhugonnet.readthedocs.io/en/add_accessor_daskmp_doc/scalability_logic.html
Details
The documentation changes are the following:
Raster.method() or ds.rst.method()for each method linked through a singleRasterBasecall withRasterBaseitself not being visible (for users using the search button). Also added fullRaster+RasterAccessorfull autoclass summary for those also curious to search through the class details. But those two pages are "hidden" in the table-of-content structure to avoid duplicating the main API page (users can only land there through search or clicking the class object name); and the page starts with a link pointing back to the main API, if that's not where they wanted to land.I haven't yet updated the "Quick start", but I think we really need a better example there, and we should use the Xarray/Pandas accessor directly.
Same for the "Fundamentals" section, there's some editing to do there so that's it's not too "GeoUtils object-focused", but balanced whether it's about accessors or GeoUtils objects.
Finally, we have to see for the "Examples" (in feature pages, and in the Sphinx gallery), do we switch all to Xarray/Pandas accessor?
In particular, as I was already doing diagrams in Python code for something else, I kept my inertia and decided to add some to the PR to describe our implementations visually! 😄
We might think of doing something similar to describe functions themselves in the future (interpolation, etc).
Finally, I realized that it was annoying to explain the mirror for
Raster/rstbut not for vector/point cloud at the same time, even though the accessors for those 2 are much easier to add.So I might add them when I get the chance in the next weeks while this PR is being reviewed 😉
Resolves #677
Resolves #673
TO-DO TO FINALIZE
proximitysupport status withmap_overlap(yes, and report caveat of max distance?)to_filefrom accessor, and uncomment in "Usage" page,editandfill-nodatamirroring GDAL,plot()to RasterBase (API not showing, also for Scalability?).