[Process RFC] Empowering New Scoped Module to the Project

[Hzfengsy]

In this process RFC, We'd like to propose a process to encourage scoped modules and set expectations about what we anticipate in such inclusion.

rendered
discuss thread

[areusch]

thanks for posting this @Hzfengsy . my apologies for not following up on the discuss thread in a more timely fashion. I am mostly good with this RFC, but there are two more things I think we should clarify before merging:

1. what is considered a "required module" in the project? I think based on our definition of optional module being behind a feature flag, we could state that a required module is one which cannot be excluded from the build via config.cmake option. We don't have an easy way to do this with Python, as we currently import that code in-place. But, I think the namespace criteria set forth here is enough to draw a line there.

2. The voting mechanism. I realize this will be a bit more controversial--but, while I acknowledge that other projects use methods other than consensus when deciding to do things such as adopt whole codebases into the tree, I'm not actually sure it makes sense here.

   Here is why: ultimately, an RFC that brings an S0 proposal does nothing more than provide justification for a series of PRs implementing the feature. In this way, an S0 RFC is a Code-Modification Proposal, so I think votes on such RFCs should follow the ASF guidelines for votes on code modification. As the RFC stands now, a committer could simply go and -1 each following PR if they wanted to--it seems like it should be better to sort out any disagreements on the initial RFC.

   There have been some examples cited where other ASF projects implement different rules for some code changes, such as replacing a shipped part of the codebase with another implementation or adopting a subproject wholesale into the tree. We certainly could do that here for S0 proposals.

   I think those examples are different than S0 proposals--they're actually more like an S0 -> S1 RFC in my mind. They might create a giant merge commit, or a 20,000-line commit that isn't reviewed according to the project's standards. Such large changes are often so disruptive to a project as to influence the project's overall direction and, in turn, its community. In those cases, you could see how these issues could fracture the community, and different voting rules may be necessary in the name of forward progress.

   I don't think this is the same situation as agreeing to incubate some code that will be added incrementally, so I don't see a good reason we should deviate from the standard RFC process or code-modification voting scheme for S0 additions.

I'm good with the remainder of this RFC outside of these two concerns, and particularly the second one. I agree TVM needs to be open to new code in order to continue to grow. I'm open to hearing folks' thoughts here as well.

[comaniac]

I support this RFC. My points have been illustrated well in the discuss thread.

My two cents about the points @areusch raised:

I think based on our definition of optional module being behind a feature flag, we could state that a required module is one which cannot be excluded from the build via config.cmake option. We don't have an easy way to do this with Python, as we currently import that code in-place

IMHO we could also do that in Python by crafting a module import mechanism. Specifically, instead of just import everything, we should encode some logic to judge whether we should import this module or not. The simplest implementation could be like:

1. In C++ side, we register an API to indicate whether this module is enabled. This methodology has been used to provide build information such as LLVM and CUDA versions.
2. In Python side, we could have

   if get_global_func("support.NewNamespace_enabled", allow_missing=False)():
       import NewNamespace
   else:
       NewNamespace = DisabledNamespace()

Or we could also have this mechanism in the module implementation:

# In python/tvm/new_namespace/__init__.py
if get_global_func("support.NewNamespace_enabled", allow_missing=False)():
    from . import XX
else:
    XX = None

Here is why: ultimately, an RFC that brings an S0 proposal does nothing more than provide justification for a series of PRs implementing the feature. In this way, an S0 RFC is a Code-Modification Proposal, so I think votes on such RFCs should follow the ASF guidelines for votes on code modification. As the RFC stands now, a committer could simply go and -1 each following PR if they wanted to--it seems like it should be better to sort out any disagreements on the initial RFC.

-1 is totally fine and I don't think this Process RFC forbids this. On the other hand, if an RFC is suspensive for a while but we still cannot make every -1 voter happy, it doesn't make sense to me to reject the RFC (we don't actually "reject" it, but prevent it from merging forever is basically the same as reject). In this case, if the RFC is supported by majority people, then it makes sense to me to have a workaround to merge it.

[comaniac]

Is there any guide about how to get this PMC member, assuming the S0 RFC is not proposed by a PMC member? Also what should we do if the champion doesn't response on time during the upstreaming process? Should the owner find another PMC/committer for help?

[sunggg]

Thank you for sharing your thoughts, @areusch. I think it might be helpful for better discussion if you can clarify a little further especially about your main concern (2nd one). Particularly, I want to hear more about your thoughts on the following things:

1. Are you suggesting the consensus is the right way to go for this optional module? If so, I'm wondering how you define the consensus. IMHO, reaching unanimous might be practically impossible for the large community like this and it does not seem to fit for S0: by definition, S0 is to satisfy clear needs from some users in the community, not the entire users. I'm afraid if too high bar for optional module may discourage contributors.
2. This quote below sounds more about the standard process of how we digest a giant commit, which could be orthogonal to voting rules. IIUC, this seems like the process of integration after the vote of deciding whether community members want to accept the proposed optional module or not. Personally, I believe voting should be based on the idea or proposal rather than the code size.

They might create a giant merge commit, or a 20,000-line commit that isn't reviewed according to the project's standards. Such large changes are often so disruptive to a project as to influence the project's overall direction and, in turn, its community.

Thank you!

[slyubomirsky]

I wonder about a situation where some parties indicate up front that they are resolutely opposed to ever permitting an S0 module to become S1. Even if this process permits the module to be merged as S0, it would essentially be known in advance that it is unlikely ever to be fully integrated into the project. Is having such a module remain indefinitely in the optional state an acceptable outcome? The text of the RFC makes it sound like the S0 state is intended to be temporary until the module is either made permanent or deprecated; is that a reasonable inference?

In particular, the point about discussing future plans for a module may be worth further consideration:

There should be discussions about how the proposal fits into the project to bring clarity. We also acknowledge that not all S1, S2 level decisions can be made at the beginning. Additionally, an S0-module should show a clear positive fit to some(but not all) aspects of the project and clear cohesion to some of the existing modules. As the development evolves, more discussions will happen in future RFCs with additional evidence that help us to make informed decisions.

If some community members indicate their opposition to an eventual RFC, that would render the future roadmap moot, no?

I guess this is a question about having the hard discussion up front or later.

[mbaret]

A general concern I have with this RFC is that it discusses S0 at length while also introducing the concept of S1 and S2. I think a more complete proposal would address what it means to be an S1 module (i.e. a 'core' component) and also how a deprecation process (S2) should proceed. I think that would give greater confidence in accepting S0 modules as we'd have a better understanding of what will happen with them in future.

Additionally, I think it'd be helpful to include in the 'background' why the TVM project accepts such S0 modules directly into the main repo (vs. them living on a fork/other project). There are other OSS compiler projects which maintain a high-bar to merge new modules directly (LLVM/MLIR comes to mind) so it'd be good to explain why we're taking an alternative (but still valid) approach.

[mbaret]

Most components in TVM are namespaced so on its own I'm not sure how relevant this point is. Perhaps the intent of this is covered by the later requirement that there be 'no disruptive changes'.

[mbaret]

I'm not sure this counts as 'background' any more but is instead part of the proposal.

[mbaret]

I'm not sure MLIR specifically is a relevant example here. MLIR is a compiler infra so by design people are meant to derive many different dialects. That's a technical ambition rather than a governance decision.

[mbaret]

Again I don't think the Torch-MLIR/TOSA example is relevant here, they are different projects under separate governance.

[Hzfengsy]

Great point. We mistake the TorchMLIR here. There are similar sets of in-tree computational graph dialects that we intended to refer to – both TOSA and Linalg for example are in the MLIR tree and serve as computational graph dialects.

Thanks for pointing it out. The text is updated!

[mbaret]

Same comment with MLIR as previously :)

[Hzfengsy]

Updated to Linalg/TOSA

[mbaret]

This has been pointed out by others already, but this process does create a risk of modules getting 'stuck' in S0 should they subsequently get blocked on S1 transition (or even stuck in S1 if we can't get consensus for S2).

[Hzfengsy]

Note that these things happen at different time points and certainly have different considerations. This is also common practice in many ML projects. Take PyTorch as an example, the criteria for accepting TorchFX for example, is certainly much higher than deprecating TorchScript in favor of TorchFX – as of now there is no deprecation of TorchScript but nevertheless FX thrives as part of the core PyTorch compiler.

We should also note that we have different sets of evidence at different points due to the nature of developments, and everyone in the community is bound to evaluate this updated evidence accordingly. For example, when TorchFX got started it was not intended for the main torch compilation flow, but as the project evolves and Dynamo is built up, it is now clear that it helps as part of the core PyTorch compiler stack.

In short, it is common practice in OSS projects to bring an open perspective to different scopes of impact of a module and evaluate them accordingly.

[mbaret]

This is a lower threshold than I've seen in similar projects (which typically adopt majority or 2/3rds majority). Do we have a particular justification for this number?

[Hzfengsy]

Thank you for bringing this up. 2/3 majority decision normally applies to S1/S2 level decisions that have a bigger scope of impact. This is mainly considering different levels of open-mindedness people normally apply to decisions due to the difference in terms of scope of impact.

[mbaret]

So just to clarify this, if all but 3 PMC members voted against an S0 module, but 3 still endorsed it, would that vote succeed? Because I would expect that a module with minority support would have an extremely hard time getting PRs merged and could cause a lot of friction in the community.

[Hzfengsy]

That's a great point. To prevent such cases, I change the text from "More than three PMC members" to "Majority of the PMC members". Thanks for your kind advice

[mbaret]

This actually seems overly restrictive in some regards. Requiring 3 endorsing committers who agree to own/maintain is a very high bar and would be exclusionary to newer contributors.

[Hzfengsy]

We need a clear set of people who can review/maintain the code and maintain the overall review quality bar following the same process of all modules, so at least one is necessary. The volunteers also do not have to write code, but need to be responsible for code reviews and follow up actions, and contribute their time on a part-time basis. Likely the supporting PMC members would be able to serve that role initially. This being said, we anticipate new contributors being bought in as committers. The goal is to grow a set of contributions rather than restricting them.

[mbaret]

I suppose my question here is what would we lose by setting the threshold at a single owning committer? We already have a number of areas in the S1 code that aren't owned by 3 or more committers, so it's not clear what we'd gain from having a much higher requirement for S0 code. It's also quite a commitment to own code where you haven't been part of the development at all, so I think finding 3 volunteers could be troublesome for certain contributions.

[Hzfengsy]

Thanks for the suggestion. The proposal makes sense to me. And I have updated the RFC from 3 committers to only one committer, which is aligned with typical code-changing PRs

[areusch]

thanks for the replies here, everyone.

@comaniac

-1 is totally fine and I don't think this Process RFC forbids this. On the other hand, if an RFC is suspensive for a while but we still cannot make every -1 voter happy, it doesn't make sense to me to reject the RFC (we don't actually "reject" it, but prevent it from merging forever is basically the same as reject).

I guess my point here though is: we have this problem for PRs too. We trust committers and PMC members not to blockade PRs so long as everyone is collaborating. For ordinary RFCs, we have the same situation. We achieve lazy consensus through discussion. In cases where there is a disagreement that cannot be resolved, and the argument has run its course, I agree it may become necessary to relax the requirement of lazy consensus in the name of progress. I am not aware of such a case in the project right now.

To be explicit: in the case of the Relax RFC, the authors have not answered some key questions that contribute to "discussions about how the proposal fits into the project." My understanding is that those answers are coming so the argument is not deadlocked.

@sunggg

Are you suggesting the consensus is the right way to go for this optional module? If so, I'm wondering how you define the consensus. IMHO, reaching unanimous might be practically impossible for the large community like this

Sorry--to clarify my original post, I am suggesting we adopt the same voting mechanism as we do for PRs and ordinary RFCs.

it does not seem to fit for S0: by definition, S0 is to satisfy clear needs from some users in the community, not the entire users. I'm afraid if too high bar for optional module may discourage contributors.

It seems like 3 PMC approvers is a higher bar for newcomers than merely getting a +1 from a committer.

This quote below sounds more about the standard process of how we digest a giant commit, which could be orthogonal to voting rules.

This was meant to serve as an example of the effect of adopting a resolution such as replacing the implementation of a large component or adopting a submodule could wind up impacting the project in ways that materially determine its future. It is common for projects to fork when two different factions cannot reach consensus on a change of this magnitude. In such cases, I could see the need to relax the consensus requirement in order to decide which way the original project will go.

By contrast, an S0 proposal clearly limits its blast radius and should not impact a project to the degree of forking. I don't see a clear reason why we cannot use the same voting mechanism we do for PRs and RFCs here.

[Hzfengsy]

Thanks for the input and feedback from the community. Here I'd like to clarify some questions.

For @areusch

As the RFC stands now, a committer could simply go and -1 each following PR if they wanted to

Note that the reviews of each PR are brought to their own context, and we anticipate grounded constructive actionable comments, such as adding test coverage here/there. The module establishment part can be into subjective discussions that are less grounded, and community empowerment becomes more critical in such cases.

An S0 module only has things to do with its scope. We can expect a well-scoped module to benefit from more open-mindedness in evaluating module establishment. S0 modules are not temporal because every module is subject to depreciation with different levels of consideration. We also anticipate that if the S0 module continues to be helpful to the users in its well-scoped fashion, it is totally fine to remain at S0 if changing to S1 is not necessary for the use cases in mind. The change to S1 only has things to do with changes in the scope of impact but not how functional the module is, etc.

For @slyubomirsky

some parties indicate up front that they are resolutely opposed to ever permitting an S0 module to become S1

We also like to come back to the fact that it is impossible to cover all aspects in the beginning. e.g., TorchFX, as of now, does not anticipate the usage of TorchDynamo and other cases, and that was the reasoning. We value grounded discussions with technical reasonings. Stating that blank statement upfront is unlikely a welcoming move to the overall community, such a hypothetical statement also does not align with the community over code principle (and blank -1 that is not grounded does not carry weight). Additionally, everyone is supposed to wear the project hat. A statement that is based on the interest of a party (and not directly the project), although essential to consider, does not carry binding weight. Having the S0 module evolve provides opportunities for community members to have more grounded cases and help the community collectively work towards goals that would help S1 conversations – while continuing to provide value already through S0-level empowerment. Everyone will evaluate the conversations at the point of S1 proposal and provide grounded discussions.

I guess this is a question about having the hard discussion up front or later.

In the OSS project, we all know that it is not possible to get evidence of all perspectives in the very beginning. For example, nobody proposed TorchFX to become the backbone of PyTorch compilation when it was introduced. It is also impossible to have grounded conversations to sort everything out in that first stage. Different decisions are made at different time points with different sets of evidence at that time point.

@mbaret

Again I don't think the Torch-MLIR/TOSA example is relevant here, they are different projects under separate governance.

Great point. We mistake the TorchMLIR here. There are similar sets of in-tree computational graph dialects that we intended to refer to – both TOSA and Linalg for example are in the MLIR tree and serve as computational graph dialects.

This has been pointed out by others already, but this process does create a risk of modules getting 'stuck' in S0 should they subsequently get blocked on S1 transition (or even stuck in S1 if we can't get consensus for S2).

Note that these things happen at different time points and certainly have different considerations. This is also common practice in many ML projects. Take PyTorch as an example, the criteria for accepting TorchFX for example, is certainly much higher than deprecating TorchScript in favor of TorchFX – as of now there is no deprecation of TorchScript but nevertheless FX thrives as part of the core PyTorch compiler.

We should also note that we have different sets of evidence at different points due to the nature of developments, and everyone in the community is bound to evaluate this updated evidence accordingly. For example, when TorchFX got started it was not intended for the main torch compilation flow, but as the project evolves and Dynamo is built up, it is now clear that it helps as part of the core PyTorch compiler stack.

In short, it is common practice in OSS projects to bring an open perspective to different scopes of impact of a module and evaluate them accordingly.

There are other OSS compiler projects which maintain a high-bar to merge new modules directly (LLVM/MLIR comes to mind) so it'd be good to explain why we're taking an alternative (but still valid) approach.

The overall suggestion over S0 is not about lowering the bar to code itself. In that regard, we still come up with the statement of "an S0-module should show a clear positive fit to some(but not all) aspects of the project and clear cohesion to some of the existing modules." It mainly comes back to the scope of impact and level of open-mindedness when evaluating the related modules.

The goal of the RFC process is to clarify that and follow this practice with motivations also outlined in G1 and also empower the community members who clearly voiced their needs for such empowerment.

This is a lower threshold than I've seen in similar projects (which typically adopt majority or 2/3rds majority). Do we have a particular justification for this number?

Thank you for bringing this up. 2/3 majority decision normally applies to S1/S2 level decisions that have a bigger scope of impact. This is mainly considering different levels of open-mindedness people normally apply to decisions due to the difference in terms of scope of impact.

[Hzfengsy]

S1-level module

There are a few suggestions for clarifying the S1-level module. An S1-level module is a
module that does not follow the restrictions outlined in the S0. Specifically, an S1-level module is usually used as dependencies by other major modules in the project.

The considerations that go into the S1-module are also not about how good the code is. As a matter of fact, if it is possible to maintain clearly isolated components while being useful in general to a broader set of users. Keeping modules in S0-level scoping is even desirable. When there is a need to affect a broader set of modules, however, certainly such broader scope would trigger more careful discussions among more people. It again boils down to the level of open-mindedness due to the different scopes of impacts.

Requiring 3 endorsing committers who agree to own/maintain is a very high bar and would be exclusionary to newer contributors.

We need a clear set of people who can review/maintain the code and maintain the overall review quality bar following the same process of all modules, so at least one is necessary. The volunteers also do not have to write code, but need to be responsible for code reviews and follow up actions, and contribute their time on a part-time basis. Likely the supporting PMC members would be able to serve that role initially. This being said, we anticipate new contributors being bought in as committers. The goal is to grow a set of contributions rather than restricting them.

Is there any guide about how to get this PMC member, assuming the S0 RFC is not proposed by a PMC member? Also what should we do if the champion doesn't response on time during the upstreaming process? Should the owner find another PMC/committer for help?

Great question. We anticipate the champion to signal such intent during the pre-RFC stage (at the forum). Normally we expect champions to be able to land a commitment of their time. A new champion is certainly possible if it is not the case.

[mbaret]

@Hzfengsy

There are similar sets of in-tree computational graph dialects that we intended to refer to – both TOSA and Linalg for example are in the MLIR tree and serve as computational graph dialects.

TOSA/Linalg are both graph dialects, but they don't fulfill the same function - TOSA is a higher-level representation than Linalg. As an example, IREE implements a lowering from TOSA to Linalg as part of its TFLite support. The closest example I can think of is TCP vs. TOSA which was prompted by this lengthy thread. However, that caused a huge amount of debate as to whether it was correct to have them both in the repo (similar to what we're seeing with Relax).

The solution in the LLVM community is to have TCP first exist as an LLVM incubated project outside of the mono-repo. To quote the LLVM developer policy on why they have such a process:

The burden to add a new project to the LLVM monorepo is intentionally very high, but that can have a chilling effect on new and innovative projects. To help foster these sorts of projects, LLVM supports an “incubator” process that is much easier to get started with. It provides space for potentially valuable, new top-level and sub-projects to reach a critical mass before they have enough code to prove their utility and grow a community. This also allows collaboration between teams that already have permissions to make contributions to projects under the LLVM umbrella.

This sounds extremely similar to the problem we're tackling with this RFC, just with an alternative solution. I think we need to be clear why we're arguing not for an incubation process (the MLIR/LLVM solution), but instead for this expedited approval process. It would also be good to include that argument in the RFC to make it clear that other processes have been considered.

[yzhliu]

@mbaret I think "incubator process" needs to make sure the incubated project gets support from the community. That's what they mentioned about "LLVM umbrella" or "Apache umbrella". LLVM and Apache are large enough organizations that they have dedicated mentors and more importantly, they attract outside contributors and users even for a small, incubating project sitting outside of main branch. If we are able to do the same I'm OK with the "incubator process", but I think at this stage what @Hzfengsy proposed in this RFC is the best trade-off.

In addition, incubation is mostly used when majority of community is uncertain about in-tree but still offers opportunities of dev in the case of MLIR/TCP conversation, it is clear that majority of core-dev opposed the in-tree dev

In the S0-module proposal, majority of community supported in-tree dev, and they already offer clear values to some community members.

[mbaret]

@yzhliu To clarify, I'm not suggesting we adopt an incubation process for TVM. I'm just explaining that we can't reference MLIR as an example of a community which uses the process described in this RFC, because it in fact uses a completely different process.

In the S0-module proposal, majority of community supported in-tree dev

This process explicitly allows a module to be merged without majority support - only 3 PMC members are required. In principle that means a '3 for / 22 against' vote on the PMC would actually pass in favour of merging a new module which seems highly unusual to me.

[areusch]

@Hzfengsy

Note that the reviews of each PR are brought to their own context, and we anticipate grounded constructive actionable comments, such as adding test coverage here/there. The module establishment part can be into subjective discussions that are less grounded, and community empowerment becomes more critical in such cases.

Taking a step back here: why do S0 modules need to live in main? All of our development actually happens on forks (i.e. you fork the repo on GitHub)--the question is: how much development goes on outside of main before we merge? Forks could technically live for years outside of main (as Relax has). So what are the benefits of merging:

B0. Joins development of the S0 module to TVM's S1 and S2 modules, distributing the need to rebase and handle merge conflicts throughout the TVM community.
B1. Solicits a wider audience to interact with the module by publicizing it as part of TVM's tree.
B2. Fosters further development of the module inside of TVM's core forums. This also helps some community members to better engage by potentially removing hurdles (i.e. engineers at industry companies may be restricted by company policy to engaging with only specified OSS communities).
B3. Grants access to TVM's CI resources and runs tests in the mainline regression suite.
B4. Permits development of tighter integrations with the remainder of the project by removing merge conflicts.

There are also some drawbacks:

D0. Increases the load on TVM's CI.
D1. Increases the scope of TVM's codebase, potentially making it harder to maintain. For example, changes to core components such as the test infrastructure, the runtime, the object system may require sweeping knowledge of the project. As the number of components in the project increases, the number of engineers with that knowledge decreases relative to the size of the codebase. If that number drops to 0, the codebase will rot.
D2. Could contribute to effectively splitting the community by creating multiple arenas for development. While there are benefits to this in terms of exploring new techniques, allowing this as a permanent state means that effectively TVM is many projects with sub-communities rather than one. This may be a reasonable thing to strive for, but for D1 reason it is not clear that those projects should be in a monorepo.

A theme across these is that some aspects are community-related (i.e. publicizing a flow, consolidating discussions, allowing us to explore multiple avenues to solving a problem) and some of these are essentially technical/logistical problems (i.e. maintaining a CI, maintaining the core infrastructure, etc). Weighing the community aspects of these benefits and drawbacks, I believe that the benefits are worth the drawbacks and I am in favor of allowing new contributions and approaches in TVM.

However, I feel we are still not really addressing the technical side of things: there is no such thing as a truly S0 module, because we state it is

No disruptive change to the rest of the codebase

The problem is that because we are in a monorepo, there are technical/logistical aspects to integrating a module where "a failure in one module is a failure for all of TVM," the most obvious example being CI:

• A flaky test is checked in; someone is trying to make a change to an S1 module, but the CI fails sporadically due to the flaky test to impede progress.
• Someone is trying to improve the way we shard tests with pytest; in the S1 modules, we have e.g. properly parameterized the tests, but in some S0 module, the parameterization was not properly followed. Now that engineer needs to go and learn about how the S0 module works, fix its tests, and find a reviewer for those improvements before they can proceed.
• Someone adds a new CI image to test the S1 modules on a new hardware backend. The S0 module is excluded from that backend. Now a user comes along and tries to use the S0 module with the backend, and files a bug. Can the owner of the hardware backend truly claim they "support TVM" if they don't fix the S0 module bug? What mechanism have we put in place to ensure that users approaching TVM understand that they are using an S0 module?
• Someone is modifying a core TVM API; an S0 module uses it in a novel way. Now the modifier has to go and understand how the S0 module works so that they can modify the API in a way that doesn't break it.

While i agree with you that

The module establishment part can be into subjective discussions that are less grounded, and community empowerment becomes more critical in such cases.

I am concerned that the situations I elaborated above could similarly lead to people leaving the community if it becomes too fragmented. I also feel a bit like the perspective of maintaining the core APIs and test infrastructure is not adequately represented in the discourse so far, and may not be represented in the proposed voting mechanism.

There is a lot of detail here about the establishment of an S0 module, down to even introducing a new voting process. When it comes to transitions between those states, there is really not much detail:

• S0 -> deprecation: When a S0 module no longer has an active set of maintainers, the module will be deprecated. The removal of the module is easy as they are contained in the respective folders, with no modules that come and depend on them.
• S0 -> S1: When developers propose to incorporate a S0 module broadly into existing flows/components. Each such composition would require its own RFC and following the existing RFC process.

For instance, suppose that yes there is a maintainer for an S0 module, but tests keep breaking or they are relatively unresponsive, dragging down any of the core maintenance tasks above? What is the bar after which it's reasonable for a core maintainer to propose deprecation? Proposing deprecation is a lot more unwelcoming than requesting clarifications to new code or saying "no for now," and is a lot more challenging because it might involve finding a new home for code that is already being used by some members of the TVM community, but for which we are finding it hard to keep maintained.

I am not trying to be unwelcoming to new code here. But these considerations can often be caught at the RFC level, when questions about test strategy, modularization, and documentation can be discussed at a level that might be accessible to a broader section of the community. Developing the same level of understanding through reading PRs both takes considerably more time and energy than reading RFCs, and may come at a time when a module is half-landed. For this reason, it is not clear to me why we should relax the voting consensus requirements on S0 proposals only to land in a half-baked state that may deadlock on a committer.

I would summarize my concern here in two areas:

1. There isn't a clear boundary in TVM between what is "S1" and what is "S0" right now. It may not be clear to core maintainers where their responsibilities end and the S0 maintainers' responsibilities begin. Modularizing TVM could help here, but we haven't done that yet.
2. Already today, "supported by TVM" is quite ambiguous--which flow, on which hardware, with which model? S1 modules here are simply those that are "not S0." How should a user know what these are? Allowing S0 modules is good, but there is currently no distinction in how they're represented to users.

I'm concerned that this proposal could use more detail given that we are a monorepo with the problems I mentioned above. I don't know these concerns necessarily need to get addressed in this specific proposal (e.g. more clearly delineating the S1 modules), but I also don't feel that this proposal has adequate detail right now given the state of the project.

[tqchen]

Thank you, everyone, for the discussions in this thread. To help us build future constructive conversations and avoid mixing conversations in different threads, I suggest that we only apply the proposed process to future RFCs to be opened. Thank you, everyone, and let us continue to have civil and constructive discussions.

[jroesch]

While discussion continues I am asking that we hold off on voting so that all active contributors have a chance weigh in and comment. I know of at least two PMC members with perspectives to share that cannot comment until mid November. Holding off a few weeks seems like the right tradeoff for maintaining our shared values around community and community ownership at the cost of a little patience.

[Hzfengsy]

@mbaret

TOSA/Linalg are both graph dialects, but they don't fulfill the same function

The definition of "same" is subjective; of course, different people can have different opinions that are less grounded. For example, what if many, or even the majority of people think a proposal contains sufficient differences(e.g., in the sense of TOSA/Linalg or TorchScript/TorchFX) that also serve community needs but some don't, does that still count as "same"?

The current RFC provides a guideline, considering that "An S0-module should show a clear positive fit to some(but not all) aspects of the project and clear cohesion to some of the existing modules."

The "clear positive fit to some" indicates the complementary aspect(that S0-module does something other modules do not do, as a result not the same). Note such evaluation can usually be subjective.

We are also not singling out MLIR as a single case here. The TorchScript/TorchFX example serves as a better example in illustrating the situations we are facing here.

The closest example I can think of is TCP vs. TOSA which was prompted by this lengthy thread. However, that caused a huge amount of debate as to whether it was correct to have them both in the repo (similar to what we're seeing with Relax).

The solution in the LLVM community is to have TCP first exist as an LLVM incubated project outside of the mono-repo. To quote the LLVM developer policy on why they have such a process:

Thanks for bringing this up as a reference. Reading through the TCP thread, it is clear that the TCP proposal would not even get majority support from the MLIR core devs (most of the core devs signaled preference of not in-tree devs in that particular thread). An incubation process helps to incubate a module where the majority of the community is still unsure. We do anticipate different fork development methods will be helpful before the S0 stage.

We are solving a different problem. Specifically, this process RFC outlines the guideline in situations when the majority agree that in-tree dev is helpful and the discussions of module establishment become subjective.

In such cases, considering the scope of impact, users' needs and, more importantly, community empowerment in general. We think the call for the community over code and open-mindedness in the current proposal is appropriate. It also reflects G1 and aligns with the case such as TorchFX starts by offering clear positives to some aspects and continues to offer more aspects to the project.

The open-mindedness of PyTorch in allowing modules(such as TorchScript and TorchFX) to co-exist contributes positively to the growth of the PyTorch ML ecosystem. As an ML project in nature and acknowledging G1 and community being critical, this proposal aims to bring that practice formally.

One of the last factors is that many of us (considering many who supported the proposal) already were evaluating some of the previous RFCs following the open-mindedness principle already and pushed for community empowerment – which led to some of the RFC acceptance as well as broad inclusion of the community. If we all adopted a stronger subjective opinion and put away consideration of community, many past RFCs would have been rejected or delayed in the subjective debates, including RFCs from those who had stronger subjective opinions. We would hope to continue pushing the community over code principles. While also clarifying the clear transitions and scope of impact.

[mbaret]

@Hzfengsy

I don't think it's fair or accurate to dismiss legitimate concerns of community contributors as 'subjective'. @areusch has already enumerated in some detail an 'objective' list of impacts that an S0 module can have on the wider project. I think at a minimum we should be addressing those points before implying that objecting to S0 modules is somehow improper (which of course it is not).

What is true is that in a diverse community there are contributors who value things differently. For instance, there may be community members who put a much higher value on moving fast compared to others whose primary interest is predictability and stability. You can construct perfectly objective arguments in favour of either of these philosophies, and having both voices in the community is an enormous benefit to continuing development and success of TVM. Let's keep it that way and make sure everyone feels welcomed, recognising that constructive debate (like that happening on this very RFC) benefits us all.

[Hzfengsy]

Hi, @areusch

Thank you, for posting the analysis of the benefits and drawbacks of merging a module. I would like to point out that there are a few critical pieces that are missing (mainly on the community side):

• Welcome new contributors who would become an added force, these contributors also contribute normally not only contributes to the particular S0 module but also to other modules.
• Enable TVM to solve problems that it was not able to atm and empower new users/developers who need them.

Thank you also for bringing up some of the factors to put into consideration in the RFC process:

• Discussions of test strategy, modularization, and documentation during the RFC process and not land half-baked code.
• Consider the relations among S0 modules to other modules, mainly on implicit impositions that S0 made on, say existing S1 APIs.
• Consider the overall complexity of the codebase and presentation to the users.
• Ease of maintenance overall for some of the core infra.

These are great goals that we all carry when evaluating new modules(including S0) and I also support these goals. They all align with G0.

The proposal text does not imply when an S0 module gets established, it will lack these considerations throughout the process. As a matter of fact, we would also take these goals into account when evaluating any proposals(including S0).

One possible point of confusion here is connecting S0/S1 to importance, or how good the code is. The proposal intentionally avoids that, an earlier version of the proposal used the term "optional" which causes confusion, and we switched to a clear definition based on "scope of impact" rather than other factors such as importance, or how good the code test coverages are.

An S0 module can become more important to some of our users than some of the S1 modules or have better code/tests coverage. The fact that they are more well-scoped simply means it is easier to remove them if needed, and there is less refactoring needed when refactoring other parts of the code.

Admittedly, this does not mean that S0 can simply be ignored when working on other modules. It is naturally true that there are considerations around testing and API changes when working on some of the more central modules, they should be considered during RFC and follow-up actions. We want to come back and also acknowledge that the scope of impact is not binary – an S0 module in nature will have non-zero, but (sometimes significantly) less impact compared to non-S0, when it comes to changes in other modules and deprecations.

This proposal calls for putting the non-binary scope of impact and community into consideration for module establishment. Of course, such consideration should be put together with other considerations, including tests, documentation, modularization, fit into the overall project, etc.

All the members are expected to take these factors into account when evaluating an S0-RFC and make collective decisions for module establishment. This comes to the decision process. There are many discussions that can be made during the RFC process, which is very helpful, for example. Making sure that the proposal is technically correct, providing suggestions to testing coverage, documents, etc. All of us are taking those into account when making that collective decision.

Then there comes about reasonings that are relatively subjective reasonings in module establishment.

• Say we are introducing a new module to the codebase, majority of core devs think that the module provides new functionality that merits the module establishment, but some think that the new module has overlapped functionality(think about TorchFX and TorchScript case) that does not merit a module establishment.
• In some cases, the majority of core devs think that the establishment of a new module to support a not yet supported by the framework outweighs the cost of extra flow maintenance while some do not.

These are binary decisions that need to be made collectively by the community, they are likely subjective and different people have different pov and it is natural that people can disagree. Note that everyone also takes these factors into account. The proposal aims to provide guidelines and highlight different levels of open-mindedness.

The proposal also helps to clarify the case for deprecation so we can get a cleaner codebase. Previously there wasn’t any text.

Indeed we made the “S0 -> deprecation” part to be broad enough so we can handle various cases and we can talk about deprecation openly without having hard feelings about them. For example, in the case of “tests keep breaking or maintainers are relatively unresponsive”, would match this clause to start a discussion of deprecation without hard feelings. Such clarification is a positive step towards a more maintainable codebase overall. Actually having such broadness in the proposal would help, so we can welcome contributions and not have hard feelings when talking about deprecations broadly.

As a community we operate by empowering each other. Indeed it is hard to have a 100% clean cut of responsibility boundaries due to some factors being discussed. The scope of impact on the other hand is also not binary. For example, many of us volunteered to review, contribute to core modules, documents, advocations, writing tests and welcoming new members across our “boundaries”.

The proposal of course does not mean to indicate that S0 has zero impact on each of us. It certainly has a smaller scope of impact which is worth some different levels of open-mindedness. Remember the contributors/supporters of those modules also are contributing in different ways. They are all valuable and that is what makes us together as a community.

[Hzfengsy]

@mbaret

I don't think it's fair or accurate to dismiss legitimate concerns of community contributors as 'subjective'. @areusch has already enumerated in some detail an 'objective' list of impacts that an S0 module can have on the wider project. I think at a minimum we should be addressing those points before implying that objecting to S0 modules is somehow improper (which of course it is not).

I don't want to dismiss every concern and comment in the discussion (also other feedback from the community). It spent me some time to reply to @areusch as the comments are a bit long and I'm a non-native speaker, which costs me a bit more time to organize my words.

Let's keep it that way and make sure everyone feels welcomed, recognising that constructive debate (like that happening on this very RFC) benefits us all.

Please directly point it out if I miss any public voice. And please let me know if you feel un-welcomed during the discussion and the reasoning. I'm not a native speaker, so maybe I will make mistakes in wording and may make you uncomfortable. If so, please also let me know, and I will apologize.

Again, we are working on the points and trying to address them before voting, which will absolutely benefit TVM.

[Hzfengsy]

Hi all, as suggested in the thread, we held this thread for a while. And now it can be a good time to come back.

Let me summarize the previous discussion here:

• Scoped module
  A scoped module (S0-module) can be:

  • Clearly isolated in its own namespace.
  • Clearly needed by some users in the community.
  • No disruptive change to the rest of the codebase
  • Can be easily deprecated by removing the related namespaces
  • Can be turned off through a feature toggle to contain the overall dependency from the rest of the modules.

• voting mechanism
  Establishing a scoped module is not a typical code change, which needs to get majority support from PMC. This kind of voting mechanism is also used in other Apache Projects (e.g., Apache Hadoop process and Apache Hive Bylaws

• The community should evaluate the scoped module with a variety of factors, wearing the project hat:

  • Fit into the overall project and rest of the modules and project.
  • Test strategy, modularization, and documentation.
  • The scope of impact of the added module, and levels of open-mindedness.
  • Competitive landscape of the overall MLC space and enablement of the project towards goals that are not supported atm.
  • Community empowerment in general: e.g. contributors who would become an added force of offset development complexity, and also in a lot of cases contribute to other existing modules.

• Scoped module does not mean to be a low-quality module. All code changes for scoped modules are in the same code review mechanism

Please let me know if I miss any public voice and considerations. And let's continue on discussion and finalize the RFC.

[tqchen]

Thank you, everyone, for the discussion so far, and thank you, @Hzfengsy for driving the proposal. After reviewing the thread and watching all the relevant conversations. I would like to come out and share my support for the proposal.

I am making this statement wearing Apache TVM hat.

I would like to summarize a few important perspectives from our community members here as guidelines:

How should we operate as a community?

When considering proposals, there are many factors being considered by members. A lot of them have to do with the impact to code:

• Impact to Code: cohesiveness to the overall vision, testing, clarity among modules, quality, stability, the overall complexity of the codebase, and presentation to the users.

This is what we are getting used to in normal software development settings. There are even trade-off factors here. On the one hand, indeed, new modules bring different levels of impact on these perspectives depending on their scope. However, in many cases, new modules do not bring impact in a purely negative way.

MLC(machine learning compilation) is a fast-evolving field. The ability to reinvent ourselves and keep up with innovation while keeping the main component stable is the key to survival in the ML landscape. PyTorch is exemplary in terms of development philosophy on bringing in related modules. In many ML frameworks such as PyTorch, allowing space for multiple relevant solutions(e.g. Dynamo, TorchScript, LazyTensor) to coexist and evolve is critical for success. It is also important to allow flexibility of growth rather than plan everything out in the beginning. When TorchFX first came out, there was no clear prediction of Dynamo being relevant. Even today, there is no detailed plan of replacing TorchScript with Dynamo. Nevertheless, the community members continue to use the relevant modules and have an evolving conversation. ML is fast evolving and we should never expect to lock down every aspect of a plan in the beginning. Instead, open doors for diverse community members to come and participate and contribute their perspectives. Such open philosophy is what the TVM community(or every successful ML OSS project) should aspire to be.

This brings us to another very important factor, which is OK to ignore in software development settings but should always but put first in Apache projects:

• Impact to the community: bringing in new community members and growing the community. A lot of that ties back into the "community over code principle".
  When evaluating modules, we should certainly consider how the proposal empowers new users/developers. Can the proposal welcome new developers? Can they solve different needs of the community? The TVM project exists because of the community.

That means we need to empower each member with different needs, and sometimes that could mean that we would do something different than our normal development rationales to empower the community members – especially when many, or even the majority of community members, shared their voice and support. This does not mean our original rationale is wrong, but we are more open-minded and bring that community empowerment as the first principle.

How can we empower each other

Being in the same community means we share the same burdens(of maintenance, advocations) and joys together. Any process or system is not perfect, and the intention was never to assume people as adversarial.

The first principle of the Apache community is community over code, which means we empower each other. Some in the discussion thread raised the question about what if a person operates in an outright stubborn manner – like declaring outright rejection of later S1/S2 proposals or code PRs without reviewing the additional context being brought up, and not having a constructive conversation with additional context. Such behavior would certainly be frowned upon in any OSS community, and this is not a behavior I would expect from any committer/reviewers.

This also comes from the exclusive mindset and inclusive mindset we have in the community. Indeed it is an additional burden for us to help maintain, and development the code that is less relevant to our interest. One could outright dislike code that causes regressions irrelevant to our own customer use cases. It is harder to directly settle a lot of those conversations through a simple technical debate since many can have disagreements about what are important parts – to some graphs IR and executor are important, to many other TVM users, they might only care about FFI and TensorIR while building their own graph integration.

When in confusion, always remember to come back to the community. Remember that we empower each other. These are codes that are developed by members who also volunteer their time reviewing our code, writing test cases, and contributing to common foundation components such as codegen, FFI. While the need perhaps is less urgent to ourselves, empowering them would help the community to grow stronger.

We should also remember that these additional developers can contribute to modules, improve designs(new and existing one) that also reduces our own burden. Taking a short time horizon it is easy to ignore the case, but taking a longer time pov I always find that becomes the the real case and this is why community is important.

I can give examples of when we do so. For example, the AOT proposal comes with quite a few complexities and burdens that not all of us fully agree to – a divergent ABI that could have been optimized through C Packed, an IR node that later caused regression in auto-scheduling. Extra code path that only supports a limited set of models and dynamism due to (reasonable) restrictions. To many of us, it is also not necessarily an executor that we need in our daily applications.

One could have forced the AOT proposal to always take C Packed that is consistent with other server-level modules, or block the module because they contain overlapping features that are implemented differently nor support all models. We did not choose to do that; instead, we welcomed the modules because we knew that community members would need them and would benefit from them in a timely module establishment, even if some of the detailed plans still needed to be flushed out. We know that they will empower additional community members. A proposal for graph IR, or auto-scheduling layer is not that different in nature. While each module might need a different level of technical deliberation, community empowerment should always be an important factor.

Time is also an important consideration – as we are facing competition, and time to market is important to ML. When reviewing proposals that have a lot of community support, we would always try to keep a constructive conversation going and facilitate the establishment.

Being inclusive and empowering each other is an important factor in bringing a welcoming and productive community. This does mean we need to take burdens, and we also openly contribute to each different perspective of the community that supports each other.

Making ways for each other, empowering each other, instead of exclusively forcing each other would go a long way, since that is what brings the community together. I would also recommend us reading the growing open source blog post that describes Pytorch's success experiences, “you have to be absolutely stubborn on the principles / philosophy that you are driving, but everything else is changeable.”

It is important to bring each other along the journey in an inclusive way. Having modules co-exist is an important way to do so without disrupting the usability of existing modules.I and many other PMC members have personally done so in evaluating past proposals (like the above example) and would expect our colleagues to do the same.

How shall we handle disagreements

Being in a big community means we have a diverse view. When one says, "this code produces this bug", likely no one will disagree. But as we evaluate code and community factors, each of us could bring disagreements because we might have our own opinion on how things should be done considering the factors.

Shall we always disallow new modules that have relevant features to coexist with existing ones? Should we outright reject the phased approach and always ask for a detailed plan before accepting a module? After considering the arguments, shall we empower community members so some of them can enjoy the new module, and help us to grow our developer community?

There is no single perfect answer here, and we all need to evaluate things by considering all the factors. First of all, it is important to acknowledge that each of our views is both valid in many cases. It is very important to understand each others' view, and re-evaluate the proposal under such a context. Perhaps we should consider more impact to test in this case; perhaps we should put more weight on community empowerment. After acknowledging and understanding the views, having a conversation around the cases, there can still be differences of opinion, how do we as a community proceed?

Stalling is not the right option because that would implicitly ignore the voices of those who want to move forward, especially when the majority of the community wants to do so. We also need agility and support for many who value innovation while maintaining high-quality code -- especially in a competitive ML software landscape.

As a result, after acknowledging each others' views and having a constructive conversation, I recommend listening to the majority community's voices -- as it shall be a decision of the community, by the community, and for the community.

When in doubt, we choose community over code.

[areusch]

I appreciate you guys bringing this back up, and I appreciate the goal here of establishing a project policy to ensure we're welcoming to new projects as the MLC landscape changes. I still have some reservations about this proposal in its current form, but I would support it if a small number of changes were made. Here is what I think it needs:

1. Approval mechanism. I disagree with @Hzfengsy's statement that "establishing a scoped module is not a typical code change." Because the typical S0 module would reside inside apache/tvm on main, a subproject at the moment is essentially several PRs that add code to the codebase. This is a typical code change, so I don't see a reason to adopt Lazy Majority consensus here. I appreciate that Hive uses Lazy Majority, but Hive has 52 PMCs and 104 committers, and also has a process by which committers and PMCs automatically gain emeritus status after 6 months of inactivity. TVM has a much smaller PMC. I would support this proposal if Lazy Consensus was the voting mechanism.

   One of my practical concerns here is that we lose recourse for preventing overload of the CI. And while I do want us to be welcoming of new projects, I could see the following situation arising that would place committers in an awkward position:

   1. A new S0 module is proposed and a majority of PMC members approve. The S0 module is accepted.
   2. PRs for the new module begin to merge.
   3. At some point, the integration test suite is added, but this involves regression tests that add 1 hour to the CI. The proposer either doesn't have time to rewrite the tests, or it's not practical to do so. Meanwhile a committer more familiar with the CI flags this, but the proposer cites acceptance of the RFC as grounds that the project has chosen to merge the new S0 module and its tests.

   CI is a bit tricky since it's complex enough that not everyone understands it, but it's a shared community resource that underpins the whole project. I think that choosing community over code means that we don't overextend the project so that we can ensure that such shared resources are available to everyone to work towards the project's goals. While I agree the community has been generally involved with keeping the CI green, only 4 of 20 PMCs have made or been involved with changes to Jenkinsfile in the past year (and of those, only 2 made changes other than updating a CI image). Changing Jenkinsfile is the primary way we do things like resharding tests to reduce apparent CI runtime. I'm not sure Lazy majority is the right tool to evaluate this aspect of the decision, considering that adding S0 modules may involve significant changes to the CI.

   To conjure another situation at the other end of the spectrum: Someone new to the project proposes that we add an S0 module, but although they get a few supporters, they don't manage to get majority PMC support. Perhaps they don't know many PMCs, or the PMC doesn't know much about the relevant subject area. Is the proposal rejected? Majority seems like it could also be a high bar here, and if we intentionally adopt a different voting mechanism to accept new projects, the rules seem more confusing for newcomers.

   It doesn't make sense to me to have two different voting mechanisms involved with adding code (one for PRs, and one for RFCs that propose a bunch of PRs to add isolated code). There are many situations today in which committers and PMCs can block changes to the project, and we rely on a sense of community to avoid such deadlock. I continue to think that fostering consensus in the community is the best way to choose community over code. Should we come to a point where the community needs to choose between two paths forward, such as when making an S1 -> S2 change, and consensus cannot be reached, then that is a case where I think Lazy Majority may be a reasonable path to making forward progress.

2. S1 and S2 module lifecycle. This RFC talks a lot about S0 proposals, and it is great that we are creating a welcoming stance towards new contributions. However, I would like to see the proposal rounded out with discussion of how an S0 module should become S1 and how an S1 module can deprecate others (i.e. S1 -> S2). I agree that some of this is covered in the RFC today, but because the RFC doesn't include complete discussions around the S1 and S2 phases, I think it's hard to understand the S1 and S2 concepts. Here's a specific example:

   There can be follow-up steps (S1), such as making a dialect broadly accessible in existing compilation flows.

   This sentence, inside "Background," discusses the existence of S1 steps, giving one example. However, the way it's worded makes it sound like others may exist; however those aren't enumerated there. It doesn't really make sense to enumerate them in the Background section, but without a full S1 section, there isn't a good place to put that info. Here I am thinking of new contributors and wondering what a new contributor would do should they want to understand the full process. Where could they look?

   Some specific questions I can imagine readers having that I think should be answered in the same place as theSS0 proposal:

   • How should someone propose to make a module S1? Is it an RFC? What voting mechanism is used?
   • Are there additional roadmap or documentation requirements placed on modules that become S1?
   • At what point do we decide to deprecate modules (i.e. move other modules to S2 state)? What voting mechanism is used for that decision? How long is the typical notice period?

3. What is "stable" in TVM and how long is it stable for? I'm asking the specific questions in the previous point because industry contributors need to have an understanding of such things for project planning purposes. Is GraphExecutor stable? It's still in contrib years after being established as the reference for many unittests.

   I would like to see us be okay with making it explicit which modules are S0, S1, and S2 in the TVM codebase. Doing this is not intended to make any judgement on such code, nor is it intended to imply the quality is poor--it is merely intended to guide newcomers so that they understand which paths have been well-established in TVM and which are under development.

I'd be happy to discuss any of this further. I think these are the same three points I've been asking for in the Discuss post and initially on this review thread, and I appreciate that you guys have been iterating on the RFC. Hopefully this clarifies my sticking points.

[tqchen]

Thank you @areusch .

First of all, @Hzfengsy has already addressed many points you bought up in the past, including:

• Putting the CI/testing as part of the important factors to consider
• Clarifying the definition of S0
• Clarifying n how deprecation of S0 happens

I would like to acknowledge that you have made your point clearly in your last point. Most of those points are disagreement on how to do things. In light of such disagreement, it is important to acknowledge each others’ view, reevaluate, and come back to also put the community's voices into consideration.

Let me first try to address some of the cases you bought up:

My read is that the decision is around “module-establishment” but not blank approval of all code that comes. All the subsequent code changes are still subject to the normal review process.

CI is indeed one important area of the codebase. It is also equally important in other areas, such as community member contributing code, document, importer, different modules. No area is certain superior to another one. We are working together to empower each other.

One thing to keep in mind though is that CI/testing is only good if we have community members who come and use them. It is a great thing to have comprehensive tests, and we work collectively to resolve issues in testing, CI, system design, documentation and other places.

Remember we are fostering a culture that empowers each other. Let us see how the two scenarios you mentioned will play out in the empowering culture (which I believe we all strive to do):

Imagine at some point, the integration test suite is added, but this involves regression tests that add 1 hour to the CI.

• That particular code is receiving the normal review process, and will likely be blocked, which requires the contributing team to be mindful (either move integration, or come and help work CI to be able to support such cases).
• When some regression happens after the code gets merged, open-minded team members can always come and revert(or disable) the tests, and work together to improve them.
• If it gets down to fixing Jenkinsfile, people volunteer to help, just like in other areas. I have personally helped maintain Jenkinsfile for the first few years and would be more than happy to help improve Jenkinsfile for a module that gets supported by the majority of community members. My belief is other members will also do so.
• Bottom-line, despite some hiccups, we bring a module that welcomes more community members, while still keeping other modules to support other community members.

Remember that problems do arise in the community and we work collectively to address them. We also have tools in our hands and multiple decision points(including reverting) to iteratively improve. If we simply block the gate, we of course simply will be free of those "troubles". Unfortunately nor will we empower or even have those community members around who would otherwise be part of the community.

It is great to us collaboratively working together iteratively – which means indeed a few disruptions, but we also welcome more people to work together to bring in improvements. We all pay our credits to the community in one way or another. While not everyone might help in CI, and reverting some of their PR might cause a bit of burden on others, they help reviewing code in other modules, fixing regression tests, writing docs, and bringing in architectures that reduces burdens.

In the case of someone proposing a new S0 module and not getting majority support.

That would entail that there is at least one -1 from the PMC members. Which also means that it would get rejected at any other possible mechanisms. Not knowing PMC won’t be a problem at all because many of us are eager to review, and empower good proposals from community members. I also do not think relevant subject areas would be a problem since we have some of the best experts in ML compilation. Again you will find how much of an empowerment mindset helps greatly in here. I have welcomed many modules from people that I don;t personally know of. A lot of people loves TVM community becomes of our open mindset.

Establishing a module indeed has a lot of factors to consider, there are more negative considerations from the PMC, likely they have already considered the code and community factors. In such cases, I would still expect us to see constructive conversation and reasoning.

OSS development is not a single bundled thing.

• We establish module
• We add PR changes
• We revert patches that brings regression, CI issue or other cases

Each step has their own contexts, set of stakeholders. They can come from different people. Some volunteers can take over others. While it is important to set marks through RFC, roadmap, signaling, and strategy. It is also important to be able to facilitate and empower each other through each of those steps.

The same thing applies to process RFCs. We bring in the process RFC to address things one step at a time. While clarifying S1, S2 is indeed important, which can be a topic of future discussions. Same would go for improving the stability of TVM overall – which I believe we all are working hard as a community. Sometimes bringing new modules is part of that process.

I don’t think S0 has to do with stability btw, it only has things to do with the impact scope of the project – one module can be very stable and remains in S0. On the other hand, we might choose to refactor a S1 module that touches many areas (in that sense S1 might become unstable). They all go through the same process of code changes, documentation, release notes that signal changes.

It is natural that some parts of the codebase will be under more active development than others, which also comes with clear documentation, release notes that signal these changes. There will also be roadmaps as part of the RFC that tracks what they propose to do, as well as follow ups RFCs(and different conversations) that track additional changes. All of this information will help to inform our community members – academic and industry alike.

It is natural that different players in the community have different styles, and as a result different needs to do planning. Different scenarios can occur depends on the scope of impact:

• Deprecation of a module A certainly would impact all the members that use and invested in A. The team usually would take a heavier amount of consideration.
• Adding a module B that still allows all the features of A continues to function is likely going to be less disruptive. Because the members can continue to use A as they are. While other members(if they choose to) can start to leverage B.

It is also natural that different members have their own preference of planning, and how much lockdown we have on the plan.

• Some (industry) contributors would prefer a very detailed planning of every possible future in a longer time horizon.
• Some industry contributors would favor agility in ability to deliver new features in new modules, while continuing to leverage existing longer established modules. We have also seen many of those in the community as well.
• Some industry contributors value timely landing of capabilities so they can use them to unlock their use cases.
• Volunteer contributors cannot afford to overcome a large hurdle of contributing features while still listing out all the possible future plans.
• The success of ML software, even in industry settings, largely depends on the ability to iterate, have agility in bringing in quality advances, while allowing long standing parts to continue to remain stable for users who need them. Without agility to react to incoming needs, likely the project won’t survive the competition.

These are all important factors to consider. In the meantime, we cannot simply take the interest of one or a few players. Apache TVM is a neutral OSS and that is what attracts our members in the first place. The decision process requires us to wear an Apache TVM hat. Let us see the considerations from the project’s point of view:

• Regardless of preference, we should be a community that brings people along. Enabling different modules is a great way to do so. So members who favor stability can lean more heavily toward longer established modules; the introduction of new modules won’t change the way these modules are used, and enables the (industry or academic) player to leverage new modules in a timely fashion. I understand this may not be exactly what every member would like if it is software development in their own internal setting. But that is an approach that brings people along. We see examples of that in PyTorch and other ML ecosystem
• ML is a fast evolving field. Users do not simply go what our plan tells them to do. Instead, emerging needs come, users and community members need to be empowered. If we do not bring people long, people simply go elsewhere to find other solutions/communities. The best interest of our members is to make the project competitive, welcoming and better. In order to do that we need to bring people along.

Bringing people along is important and that is what makes a lot of OSS projects successful. As we make room for each other, support each other in the community over code principle.

[tqchen]

I want to come back and encourage that we take fine-grained steps. As @Hzfengsy mentioned, many community members, including himself, had to overcome a large language barrier to be able to come out and help facilitate this process. Having a bundled conversation reduces the ability for volunteers to participate, which goes against one core principle of ASF – which is empowering everyone, including volunteers. We should reduce such burden by having well scoped proposals, knowing that with a growing community, the conversation always continues.

One observation is that there can be a disagreement on how much bundling we have in making decisions. I would like to come back and acknowledge what @areusch stated, you have clearly made points– requiring all approvals, requiring S1 and S2 discussions, and bundling stability discussion to be addressed is a valid position on how we do things to establish this process.

It is also clear that there is a different position, that emphasizes on empowering the majority of the community, more fine-grained decision making so we don’t have to bundle all these conversations together and continue to empower each other.

They are different valid opinions on how we shall operate. After acknowledging each other’s views, take these into context, and have a constructive conversation, I recommend listening to the majority community's voices and empowering the community.

[mbaret]

Having a dragging conversation simply due to bundling reduces the ability for volunteers to participate

I'm sorry but having a 'dragging conversation' is entirely the point of a RFC. It's not a rubber-stamping process, nor should it be. And this RFC fundamentally changes the governance model of the project.

I recommend listening to the majority community's voices and empowering the community.

Firstly, there aren't many community members even present commenting here, let alone a majority, so I'm not sure what's meant by this. But more importantly, and with my Apache hat on, I fundamentally disagree with the characterisation of those who are not unconditionally supportive of this RFC as 'anti-community'. In my view this is both unnecessarily antagonistic and indeed wrong.

There are multiple different stakeholders that make up our community, I'll enumerate 3 types below:

• Feature developers
  They contribute new capabilities to TVM to ensure it keeps pace with ML landscape.
• Maintainers
  They maintain the code to ensure existing capabilities don't regress and fix bugs when they arise.
• Users
  They actually apply TVM to real problems so that the project provides value to the world.

As drafted, S0 modules seek only to empower feature developers at the cost of maintainers. Users wouldn't even see significant benefits as S0 modules would necessarily need to be hidden from the majority of users, otherwise they'd become de facto S1 modules. I therefore consider this proposal, in its current form, as hostile towards those wishing to contribute towards the maintenance of TVM.

I would further question whether we have substantial evidence that S0 modules are routinely blocked. We have 3 different auto-tuners, 3 executors, 12 backend targets, 14 BYOC targets and 11 frontends too. The vast majority of RFCs are approved, as are the vast majority of pull requests. These all point towards a community that is already extremely accommodating of new features, perhaps to a fault given our limited capacity to maintain these different components.

However, the S1 and S2 mechanisms would be of enormous value to maintainers (and users!) by strictly defining the maintenance surface (S1) and allowing for that maintenance surface to shrink as well as grow (S2). This would finally allow us to start reducing the complexity in the code base and aim to provide a rock-solid user experience where TVM could compete with PyTorch/TensorFlow/ONNX on robustness.

Could we therefore please stop pretending this RFC 'empowers the majority of the community' and say what we mean: it empowers the PMC - the composition of which currently doesn't closely reflect the active contributors in the community. There's a clear and obvious route to evolving this proposal into something that benefits everyone via S1/S2 processes, but for an unknown reason that's being refused.

[tqchen]

@mbaret first I want to apologize about my wording. Indeed constructive conversation is important -- that is why we are having this conversation for about several months now already.

I would like to point out the language barrier factor, the need to empower our community members who have put energies into this process. They are important factors that we should take into consideration.

First of all, there is no intention to indicate that those who speak against the RFC as "anti-community", I want to apologize if the text was interpreted in that way. Instead, we summarize the case there is a disagreement on hows, and disagreement on how usually are addressed through procedural approaches.

I also would like to point out that I disagree with some of your choice of wording specifically. But let us first get into some of the detailed parts:

Firstly, there aren't many community members even present commenting here

We have observed at least community contributors from more than eight organizations supporting the proposal in this RFC and the original thread. It is very rare to see vocal support of such proposals

Three different personas

Thank you for laying out the personas. I agree with some of those categorizations. I would like to point out however, that almost every one of us in the community are wearing multiple hats. I personally contributed features, maintained the modules myself and others contributed, fixes test-cases, bring refactoring, write documents.

It is unfair to use simplified arguments to categorize any S0 proposer as only feature developer. This is some form of labeling that some of us unconsciously apply in our conversations, and they are not helpful for our conversations nor being fair to many community members here, please stop doing so.
In each proposal however, we would certainly evaluate the scenarios, and if such cases do occur – where the proposer ignores the maintenance duty, please let us know and we would certainly take these into strong consideration and resolve the case. The proposal also have an explicit deprecation trigger that makes suh case clear.

As drafted, S0 modules seek only to empower feature developers at the cost of maintainers.

Let's come back to think about bringing people along on the journey. Please remember that a set of “maintainers” are not fixed in the community.

• Bringing additional contributors along would mean we would get more people maintaining the code together. This has been the case in a lot of our past empowerment. The proposal text explicitly suggested that the proposer would need to maintain the module.
• Additionally, we enable a broader set of people who will help contribute to maintenance of existing modules. These are the people who otherwise will not be given opportunity or motivation to contribute.
• It is indeed true that there will always be part of impact, and everyone in the community is supposed to take these into consideration when evaluating the factors.

S0 modules would necessarily need to be hidden from the majority of users

I do not think the proposal text implies that S0 needs to be hidden. Nor suggest that they should not be maintained. The proposal also did not suggest that S0 should be outside of the maintenance surface.

The definition of S0 only has to do with scope of impact. It is great to have something that has a clear scope of impact while benefiting a lot of users. The difference in terms of scope of impact would merit different levels of considerations in open mindedness.

There is also no disagreement on our need to reduce the overall complexity, enhance robustness of the code. We are all working together toward that goal.

Imagine a case when a S0 module is being needed by more than eight organizations -- many of them come from different backgrounds(including industry), but they are not needed by some other members. Shall we continue to hide them from the majority of users?

Of course not, we work to present these modules to the users along with other modules. When evaluating S0 modules, we should of course also evaluate all the code factors, including robustness, testing and other things being listed.
My read is that people might have a different opinion on what S0/S1 should be. Which is totally fine – again that is a different opinion on how we should do things. We are discussing the definition as being presented here, which only has to do with scope of impact to other modules, I think it is a clean way to isolate out other factors, which are also important, and we can also have more followup separate conversations.

Could we therefore please stop pretending this RFC 'empowers the majority of the community' and say what we mean: it empowers the PMC - the composition of which currently doesn't closely reflect the active contributors in the community.

Apache TVM community operates under the apache way. We all contribute in different ways at different stages. Some of us contribute code, some of us offer project and community development inputs, some of us write documents. Apache principle means that we encourage volunteer contributions where our contributions past or recent are equally valued. This is necessary so the project is not being dominated by single or a few organizations and remains neutral.

Of course the PMC cares about the current stake of the project, and its futures. Our PMC members are also valuable members of the community, we wear an Apache hat and empower the community, in a lot of cases over code. In this case community members from more than eight organizations(industry and academic) spoke up. I do not see any problem in the discussions. I would like to additionally note that theI do want to note that empowering and bringing people along – give them the opportunity to contribute and become active contributors. Many of whom will become maintainers.

What I read again a lot of things can be disagreement on how we should address things – in bundled or unbundled fashion. I want to come back and encourage that we take fine-grained steps.

[mbaret]

Imagine a case when a module is being needed by more than eight organizations -- many of them are industry players, but not all other members.

In such a situation, I would expect that with the combined resources of those eight organizations, they could address any concerns raised by the community through the standard process. That's the behaviour I'd anticpate of any new organization looking to contribute to an OSS project - they'd first and foremost prioritize listening to and respecting the existing community.

I'd expect this S0 process to more commonly apply to smaller contributions where the contributor doesn't necessarily have the bandwidth to make long term commitments nor to continue parallel (forked) development. In those cases, I can see that a higher barrier to making a contribution (like requiring roadmaps, further technical documentation, tutorials, and extensive review) might exceed what is reasonable to ask of an individual/small team. Industrial contributors, however, should easily have the resource to clear that barrier, as it would almost certainly already form part of their internal process.

The only exception I can see to this would be where a proposal is being maliciously vetoed (e.g. with entirely unreasonable requests or no justification at all). I've personally never seen this happen in the TVM community, and hope things remain that way.

[tqchen]

Thanks @mbaret. First of all, I have updated my last post and please read it through again.

In such a situation, I would expect that with the combined resources of those eight organizations, they could address any concerns raised by the community through the standard process. they'd first and foremost prioritize listening to and respecting the existing community.

These people are part of the existing community. And existing community as a whole should work on listening and empowering each other. People contribute in different ways, code, documents, advocation, no one means would be superior to another one. I would avoid the default assumption that many of cases are not -- instead, i would always welcome, and assume many of them are already part of the community.

new organization looking to contribute to an OSS project

As we evaluate contributions to Apache projects, we evaluate the contributions from individuals. While we indeed tend to think people as organizations -- people move across organizations and many are contributing in their volunteering basis.

It is also important to say that when considering contributions or decisions, we do not weight by resources, or interest by organizations. It is totally fine for our members to make statement that are reasoned from the benefit of their own internal organizational interest -- they just do not carry binding values.

It would be dangerous though if we are only making reasoning on some part of the factors, such as code, but not the community. That goes against our overall principle, and as members of community we need to constantly remind each other that community empowerment is key here.

Resources and ability to go through process

A lot of our discussions are about how we should operate as a community.

Shall we always disallow new modules that have relevant features to coexist with existing ones? Should we outright reject the phased approach and always ask for a detailed plan before accepting a module? After considering the arguments, shall we empower community members so some of them can enjoy the new module, and help us to grow our developer community?

There is no single perfect answer here, and we all need to evaluate things by considering all the factors. First of all, it is important to acknowledge that each of our views is both valid in many cases. It is very important to understand each others' view, and re-evaluate the proposal under such a context.

• So while some of ways of operation suggested by some of our members(including yours) are indeed valid, and it could be one way on how we operate things -- we should totally acknowledge that.
• The other ways of operating things are also valid and we should also acknowledging that as well.

While both ways of how are valid and they can be ways of how we operate -- there is no assumption of ill-intention on either ways of operation. After having constructive discussions, clearly acknowledging the opinions. This is the time when we needs to lean into the community to collectively come up with how we operate. There is already establish ways in ASF and our community to do so. We are following that process.

[mbaret]

First of all, I have updated my last post and please read it through again.

Extensively editing your reply after people reply to it creates a discussion that's essentially impossible to follow. If you want to raise new points, and in the interest of maintaining the public record of our discussions, I'd appreciate a separate reply. Editing should be for making small corrections or minor additions, not completely re-drafting a response.

Shall we always disallow new modules that have relevant features to coexist with existing ones?

It's getting a bit tiresome pretending this RFC isn't in direct response to what's happened with Relax, as no concrete examples are being cited and the mysterious 'hypothetical' happens to coincide precisely with the Relax situation.

If the request on the Relax RFC was 'scrap this entirely, rewrite it some other way', I'd have absolute sympathy. But let's recall what was actually requested on that RFC:

• Explanation of why the technical approach taken we chosen
• Roadmap of features that were going to implemented (not explicit timelines)
• Roadmap of how someone currently using Relay could migrate to Relax

These were questions asked in good faith by an organisation that actually wanted to migrate to Relax, so it's unbelievable to me how they were treated. I personally spent many many hours of my own time trying to find a resolution to that RFC so that Relax could progress. And what I concluded was that positive, constructive and polite engagement would have entirely avoided the situation we're now in, it's completely unrelated to process. If we really want to improve our community, we should all reflect on how that RFC (and the following vote) was handled and what lessons were learnt.

I'm concerned that instead of being genuinely interested in improving the contribution lifecycle in TVM (an issue I care about and am very interested in), this RFC is instead just about amending the rules to override a specific veto. Could you reaffirm your original commitment that this new process will not apply retroactively, including to Relax?

[tqchen]

I'd appreciate a separate reply.

Here they are:

Thank you for laying out the personas. I agree with some of those categorizations. I would like to point out however, that almost every one of us in the community are wearing multiple hats. I personally contributed features, maintained the modules myself and others contributed, fixes test-cases, bring refactoring, write documents.

It is unfair to use simplified arguments to categorize any S0 proposer as only feature developer. This is some form of labeling that some of us unconsciously apply in our conversations, and they are not helpful for our conversations nor being fair to many community members here, please stop doing so.

Let's come back to think about bringing people along on the journey. Please remember that a set of “maintainers” are not fixed in the community.

Bringing additional contributors along would mean we would get more people maintaining the code together. This has been the case in a lot of our past empowerment. The proposal text explicitly suggested that the proposer would need to maintain the module.
Additionally, we enable a broader set of people who will help contribute to maintenance of existing modules. These are the people who otherwise will not be given opportunity or motivation to contribute.
It is indeed true that there will always be part of impact, and everyone in the community is supposed to take these into consideration when evaluating the factors.

In each proposal however, we would certainly evaluate the scenarios, and if such cases do occur – where the proposer ignores the maintenance duty, please let us know and we would certainly take these into strong consideration and resolve the case. The proposal also have an explicit deprecation trigger that makes suh case clear.

Relax RFC treatment

I would suggest we move the conversation of relax RFC treatment to the particular thread. But to elaborate a bit, there are indeed constructive explainations of technical approach were taken, features that is going to be implemented. The questions were being genuinely answered through that thread.

There was a disagreement on whether or not deprecation roadmap should be included as a gating criteria of that particular RFC that proposes module co-exist -- this is again a disagreement on how we should do things -- which should be resolved through community conversations. I would expect continued conversation, and will work wearing my Apache hat to help resolve that conversation separately from this process RFC and empower the community members. Please share followup thoughts on that thread.

Could you reaffirm you're original commitment that this new process will not apply retroactively, including to Relax?

I have stated that and I am reaffirming that the new process will not apply retroactively to existing RFCs, that would include the current relax RFC.

It is more important on how the community should move forward in the future. I would love us to be in a world where community's voices are being heard, and we collectively working to decide on how we do things through the defined process .

I would love us to be in a world where community's voices are being considered as an important factor besides code -- to be very honest this is what Apache community should be. As someone who created the project, works together with many community members, and deeply invested into it.

I am very concerned about the needs of community empowerment especially when it comes to allowing relevant modules to coexists to cater to different members' needs. Without such empowerment and consideration, we could be very well on the code for a while, but we will simply not get there as a community that empowers each other.

Thank you for having this conversation. I know that we have disagreements on hows. I would like to acknowledge again:

• The approaches(on must laying out X, Y Z) being described in your post are completely valid and possible approaches on how to do things in Apache TVM. There is nothing wrong in stating these positions. There is nothing wrong for you to make these suggestions on how we should do things.
• There are also different valid opinions on how to do do things. Which are also not wrong. Especially when it comes to putting community factor into consideration. Please look into those messages and re-evaluate under these context.

In the end, after having conversations constructively, acknowledging different approaches on how to do things. We need to collectively work together on which approach to go as a community.

[Hzfengsy]

I withdraw this proposal in favor of the simpler and better process in #102