Add RangeIndex

[benbovy]

• Closes Regular (linspace) Coordinates/Index #8473
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

Work in progress (just Ready for review (copied and adapted the example from #9543 (comment)).

[benbovy]

I've made further progress on this. Some design questions (thoughts welcome!):

Create a new RangeIndex

• I'm not sure yet about the public API? Currently RangeIndex.__init__ is "private" (more flexible and easier for internals) and there are two public factories RangeIndex.arange and RangeIndex.linspace inspired from Numpy API. Creating a new dataset with a range index would look like:

import xarray as xr
from xarray.indexes import RangeIndex

index = RangeIndex.arange("x", "x", 0.0, 1.0, 0.1)
ds = xr.Dataset(coords=xr.Coordinates.from_xindex(index))

<xarray.Dataset> Size: 80B
Dimensions:  (x: 10)
Coordinates:
  * x        (x) float64 80B 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Data variables:
    *empty*
Indexes:
    x        RangeIndex

• RangeIndex doesn't support set_xindex. Do we want to support it? If yes, how would look like the input coordinate? An existing range with explicit values from which RangeIndex would try to infer a constant step value? Is that useful to have? Since the point of RangeIndex is to avoid materializing coordinate values in memory... Or a 1D coordinate with three values representing start, stop and step? Any other alternative?

Index import

Should we expose all public built-in Xarray indexes at the top level? Or only at the xarray.indexes level?

Currently the Index base class and CFTimeIndex (not an Xarray index but could eventually be refactored so) are exposed at the top level, while PandasIndex, PandasMultiIndex and RangeIndex (this PR) are only exposed at the xarray.indexes level. We might want to uniformize that.

[benbovy]

Note: this Xarray RangeIndex is designed for floating value ranges. For integer ranges it is probably best to use a PandasIndex wrapping a pandas.RangeIndex. I added a note in the docstrings here. More work on the documentation is needed but probably in a later PR addressing Xarray indexes in general.

[dcherian]

Could consider adding a closed kwarg like pd.Interval, but in a future PR of course.

[dcherian]

Can these all be cached_property?

[benbovy]

Would there be much benefit of caching those simple aliases to attributes of the underlying transform?

[max-sixty]

Suggested change

	Examples
	Note that all `start`, `stop` & `step` must be passed, which is more explicit than `np.arange` or `range`
	Examples

(optional, no strong view)

[benbovy]

Note that all start, stop & step must be passed

This isn't exactly true, but yes the API here is more explicit than np.arange and range, e.g., RangeIndex.arange(10.0) means start=10 while np.arange(10.0) means stop=10.

RangeIndex.arange(10.0) doesn't make much sense, though, considering the default value of stop=1.0. I'll see if we can get closer to np.arange using tpying.overload.

[max-sixty]

RangeIndex.arange(10.0) doesn't make much sense, though, considering the default value of stop=1.0. I'll see if we can get closer to np.arange using tpying.overload.

yeah. no objection to the more explicit approach — it's useful-but-a-bit-magic that arange / range changes the meaning of the first arg based on how many are supplied

[benbovy]

Mimicing numpy.arange behavior is surprisingly difficult! (at least for me, I've been struggling with this).

I got it close with some simple logic, but then I hit the same issue than numpy/numpy#17878 (i.e., RangeIndex.arange(start=10) returns a range in the [0, 10) interval, which makes no sense). I could fix it after some heavy refactor making the code / API ugly to a point I'm not sure I want to push it here :).

Numpy relies on the Python C API PyArg_ParseTupleAndKeywords() but AFAIK there seem to be no easy way to know from Python whether a value has been passed as positional or keyword argument.

[max-sixty]

!!

I think the explicit approach is very valid, but maybe we just call it out / ensure people need to pass kwargs where it could be confusing

[benbovy]

In 4a128d0 I've chosen to mimicking numpy and pandas API anyway, with some simple logic and clearly documenting the caveat above.

I find myself (likely others too) doing range(10) or np.arange(10.0) so many times while I doubt many will write something like np.arange(start=10).

pandas.RangeIndex(start=10) actually still returns RangeIndex(start=0, stop=10, step=1) and I haven't seen anyone complaining in the pandas issues (or I missed it).

[wpbonelli]

Hope it's alright to chime in here.

RangeIndex doesn't support set_xindex. Do we want to support it? If yes, how would look like the input coordinate? An existing range with explicit values from which RangeIndex would try to infer a constant step value? Is that useful to have? Since the point of RangeIndex is to avoid materializing coordinate values in memory... Or a 1D coordinate with three values representing start, stop and step? Any other alternative?

One use case for set_xindex (i.e. situation where a materialized coordinate pre-exists an index) could be to "alias" a dimension coordinate?

E.g. say you have a regular grid with x/y/z dims/coords, and you want to index not only positionally but explicitly with e.g. i/j/k or row/col/lay. You could write an integer range index which accepts those names for the dimension lookup. Is that reasonable? Maybe I'm thinking about this wrong. I asked a very beginner question along these lines a few weeks ago, still trying to wrap my head around indexing.

(I understand for integer ranges one wants a PandasIndex wrapping RangeIndex — is there an example of that anywhere?)

[benbovy]

@wpbonelli I don't think this is well documented, but you could do:

da = xr.DataArray(np.zeros((5, 10)), dims=("rows", "cols"))

da.coords["r"] = ("rows", pd.RangeIndex(da.sizes["rows"]))
da = da.set_xindex("r")

da
# <xarray.DataArray (rows: 5, cols: 10)> Size: 400B
# array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
# Coordinates:
#   * r        (rows) int64 40B 0 1 2 3 4
# Dimensions without coordinates: rows, cols

da.xindexes["r"]
# PandasIndex(RangeIndex(start=0, stop=5, step=1, name='r'))

This relies on the fact that Xarray internally keeps track of the pandas.Index object when wrapped as variable data, though.

A more explicit way of achieving the same result:

r_index = PandasIndex(pd.RangeIndex(da.sizes["rows"], name="r"), dim="rows")
da = da.assign_coords(xr.Coordinates.from_xindex(r_index))

[benbovy]

This is ready for another round of review! I don't think that CI failures are related to anything in this PR.

[wpbonelli]

@benbovy thanks! sorry to hijack the thread.