Tracker: planned changes for 0.5

[dhardy]

To follow up on the release strategy in #205, here is a rough plan for Rand 0.5:

• Merge PRNG code improvements Cherry-pick changes to PRNG algorithms #209
• Add Hc128Rng Add HC-128 #210
• Add new error types New error handling + type #225
• Add SeedableRng::from_rng to replace Rand impls on PRNGs; update seed type Port new SeedableRng trait #233
• Rename IndependentSample → Distribution and remove Sample Replace distribution::Sample with Distribution + polymorphism over Rng #256
• Replace Rand with Uniform distribution Replace distribution::Sample with Distribution + polymorphism over Rng #256
• Split Rng to Rng: RngCore RngCore and extension trait Rng #265
• Move Rng etc. to new rand-core crate [requires SampleRng; Move Rng to rand-core crate? #264 Add rand-core sub-crate and update version numbers #288]
• Add HalfOpen01 distribution

Related issues which need solving:

• Naming of try_fill_bytes: Rename try_fill or fill_bytes? dhardy/rand#27
• Name of "default" distribution: Uniform; Distributions: Uniform, Uniform01, Open01, etc. dhardy/rand#81
• Distribution names: Distributions: Uniform, Uniform01, Open01, etc. dhardy/rand#81
• Fate of Rand: Fate of the Rand trait dhardy/rand#83
• Re-exporting: Re-export rand-core members in rand? dhardy/rand#15
• Decide which Distribution::sample prototype to use Distribution sample prototype and sized-ness of Rng #287
• Remove default implementations for RngCore methods?

There are some extra changes which should be relatively easy to add after the above:

• Add new floating-point sampling code Port new Range implementation and only have one uniform float distribution #274 (partial)
• Add new Range code Port new Range implementation and only have one uniform float distribution #274
• Add CryptoRng: RngCore extension trait Small changes from experimental branch #273
• Switch thread_rng() to HC-128? Replace ISAAC with HC-128 dhardy/rand#53
• Optional logging support: Log support dhardy/rand#39, Log feature: optional logging dhardy/rand#43
• New iterator code Rng::iter #275 rejected; Deprecate gen_iter and add iteration examples #286 merged

Desired additions:

• Merge Add binomial and Poisson distributions #96 (requires testing/review)
• Add Bernoulli distribution and gen_bool then deprecate gen_weighted_bool (see Revise Rng methods #293)

Finally, a list of other planned changes, which probably won't make 0.5:

• Replace gen_ascii_chars with Alphanumeric distribution: Add Alphanumeric distribution #279
• Revise character distributions (more classes, possible renaming): likely not desired
• Add SeedableRng::from_hashable: Seeding PRNGs (traits and bit size) dhardy/rand#18, SeedableRng::Seed and from_hashable dhardy/rand#62
• Change the RNG used by StdRng: Replace ISAAC with HC-128 dhardy/rand#53
• Change the RNG used by weak_rng: Requirements for a small fast RNG dhardy/rand#60, Shootout: small, fast PRNGs dhardy/rand#52
• Publish one or more PRNG algorithm crates and remove XorShiftRng, IsaacRng, ChaChaRng, etc. from rand (don't export): To provide (or not) specific PRNGs? dhardy/rand#58
• Revise WeightedChoice & sequences code: Sequence sampling: seq, WeightedChoice dhardy/rand#82
• Fork protection: WIPEONFORK [Linux] dhardy/rand#59
• Revise other Rng methods: gen_iter, gen_weighted_bool, gen_ascii_chars, choose, shuffle: Revise Rng methods #293, Deprecate gen_iter and add iteration examples #286, Add Alphanumeric distribution #279

[pitdicker]

If we remove reseed from SeedableRng this is going to break RNG implementations. Together with the introduction of rand-core, I think it makes sense to drop the Sized requirement.

[dhardy]

But the Sized thing only concerns Rand, which really doesn't have much to do with the Rng implementations.

[dhardy]

Most of the above is implemented now.

There are still some changes to Range and possibly also float sampling which I believe @pitdicker is working on; it would be nice to get those in the 0.5 release, and then there is the rand-core sub-crate (#264), which could be done now or could wait until the next release (probably better now so that the impls module can be exported only from rand-core, as in my master branch).

About the release process: I think the best method might be to publish a 0.5.0-pre.0 pre-release, then wait a minimum of two weeks total for comment and testing (and at least 1 week since the last pre-release) during which no major new features will be merged.

For rand-core I guess the version number should start at 0.1.0-pre.0 and be released on the same time-schedule; after this release I don't think there will be any need to synchronise versions/releases between the two crates (and hopefully rand-core can hit 1.0 sooner).

Also during the release process we need to work out what kind of compatibility release to make for 0.4.

[dhardy]

I just added two extra desired features: CryptoRng and switching thread_rng to HC-128.

Now would be a good time to add any extra requests for this release (but be warned that larger features may get delayed to 0.6 if not ready to merge soon).

[pitdicker]

There are still some changes to Range and possibly also float sampling which I believe @pitdicker is working on;

Yes, copying the Range code with the RangeImp method from your branch was fairly easy (thanks to the huge merge you did 👍). I have used fixed-precision float methods as discussed in dhardy#85 (comment) for Uniform and Range. The maximum precision methods will need a little more thinking, so I'll leave them for another PR. I think it is best to merge on top of #268, because MockAddRng makes fixing the test from WeightedChoice easier (uses Range internally).

For switching thread_rng to HC-128 I have things ready, waited on #252 before making the PR.

Hopefully will finish things tomorrow.

[pitdicker]

I think we are making pretty good progress towards 0.5 😄.

From #286 (comment)

Speaking of alternatives, there's still all your FP code in my experimental branch... have to figure out whether we'll use that at some point.

I would still like to see it become available. I think it is settled that it should change from pure conversion functions to distributions, that can use arbitrary precision (not a fixed 32 or 64 bits). So I will basically have to rewrite things. We have to come up with names, and figure out how to expose a second kind of uniform floating point ranges.

That is why I left it out of #274. Shall we leave it until after rand 0.5?

Do you still want to do the rand-core split?

One thing I really want to see is

Change the RNG used by weak_rng: dhardy#60, dhardy#52

It should not be all that much work to polish the PCG and XoroshiroMT implementations in my small_rngs crate, and also not to move all the PRNGs out to a separate crate.

@dhardy, did you somehow get a notification from this?

[dhardy]

That is why I left it out of #274. Shall we leave it until after rand 0.5?

Perhaps; I don't think we should block 0.5 waiting on it at least.

Do you still want to do the rand-core split?

I asked, and the answer seems to be yes. This is easy to implement but I thought better to wait until most other stuff is done though; I may get to this soon now. (There's additional motivation: we can make the impls module available only through rand-core.)

I'm not sure about changing weak_rng though it definitely makes sense. Did we decide which RNG to use yet? Ultimately it would be nice for this to be in a separate crate though it could be in rand for the next release; on the other hand it should be easy to add another crate once rand-core is ready anyway.

No, I missed the notification on your history article; I'll have a read.

[pitdicker]

Did we decide which RNG to use yet?

The choice was between PCG and my Xoroshiro variant. But to be honest I found a problem with the last one, trying to prove it was correct (a while ago actually). A widening multiply does not guarantee uniformity or equidistribution. It should be a 'real', 128 bit multiply. This can be done by using a widening multiply plus a normal multiply, causing a little slowdown (but still faster than PCG). Just to err on the safe side I would make PCG the default, and also because it is better known.

[dhardy]

That sounds good enough to me; I'd rather err on the side of strong than fast-and-weak personally. But you should probably make a post in dhardy#60 to clarify to the people who commented there what the plan is. Also I think perhaps we should remove weak_rng and instead add a SmallRng wrapper and make users do SmallRng::new(). (We could add FastRng too and make the latter use Xoroshiro*, potentially.)

[vks]

I think removing weak_rng is a good idea. I'm not sure about the SmallRng name. Why not just use the name of the RNG? Reproducibility is important, I think changing the implementation of SmallRng should be considered a breaking change. I would rather use the name of the algorithm and recommend RNGs for different purposes in the documentation instead of abstracting them away.

In my opinion, the following kinds of generators should be considered:

1. A cryptographic, default one. (E.g. ISAAC or HC128)
2. A fast one without linear dependencies. (E.g. PCG or truncated xoroshiro.)
3. A faster one with some linear dependencies. This is what practically almost everyone is currently using if cryptographic properties are not needed. (E.g. xorshift, xoroshiro)
4. A small generator for easier seeding of larger generators from i.e. a u64. (E.g. splitmix64)
5. A generator with a large (> 2^128) period for massively parallel computations (xorshift1024*).

The tradeoffs between 1., 2. and 3. should be clearly documented.

(Personally, I think 2. and 5. are extremely niche. 4. could be hidden behind some abstraction or replaced with 1. I don't see a convincing reason to use xoroshiro128* over xoroshiro128+.)

[dhardy]

@vks please see dhardy#58. The intention is to provide both names (e.g. StdRng and Hc128Rng), but put the former in rand and the latter in other crates, and to clearly mark the ones in rand as non-reproducible.

"Practically almost everyone" doesn't fly as an argument; there are still many people using MT19937, and while it might be sufficient for their purposes I won't recommend it just because it's popular.