[Feature] Command Line tool support

[simonsssu]

Motivation

Currently, we can only access tables in the cluster through the Flink SQL Client. However, this approach has several limitations. On one hand, users need to be familiar with Flink SQL to perform queries and management operations. On the other hand, the metadata about these tables is restricted by the capabilities of Flink SQL, which cannot meet the demands of more complex scenarios.

We can provide a set of CLI tools ( Like kafka CLI tools ) for users to admin their resources in the cluster. e.g. users can get a table's distribution among tablet server.

Solution

https://docs.google.com/document/d/1_XorPkTfq3ig3O-PkfylxXsyw3aYGBSkqitslSkpmyc/edit?tab=t.0

Tasks

• Add command line tool related script. #1015
• Introduce Command Line tool framework #1013
• Implement table sub command line tool #1012
• Implement cluster sub command line tool #1014
• Implement produce sub command line tool #1016
• Implement consume sub command line tool #1017

and more , e.g. acl, perf etc

[simonsssu]

This can be devided into several tasks for more CLI tools.

[wuchong]

+1 for this

What CLI tools are high-priority to support?

[simonsssu]

@wuchong IMO，phase1 we can do those CLIs for log table.

1. Table Management
   • list the tables in a cluster.
   • describe detail configs of a table. e.g. bucket distribution among the cluster, table's configuration.
2. Offset management
   • consumer status of current table. ( Like a kafka-consumer-group.sh )
3. Table interactive producer/consumer for debugging.
   • verify produce and consume for a logTable.
4. Cluster admin
   • list the table's disk usage
   • list the topology of a cluster.
   • list / dynamic change the configs of a cluster.

[polyzos]

+1 for this, sounds really good, and we could also create afterwards for Flink procedures.
Maybe we can also draw ideas from Kafka-tools like dumping log segment file internals.

It would be great to create a list, and break it down into concrete rockets and steps

[simonsssu]

this is the doc for each phase:phase plan

[wuchong]

Thanks for sharing the design documentation, @simonsssu! I’m really looking forward to this exciting feature.

I was wondering if we could consolidate all CLI tools into a single command with different actions, similar to how Flink CLI and RocketMQ CLI are designed:

• Flink CLI: CLI Actions
• RocketMQ CLI: Admin Tool

The current design provides separate CLI tools for each use case, which I find complex and inconsistent. Instead, we could streamline the CLI by grouping all actions under a single command, like so:

./bin/fluss list tables
./bin/fluss desc <table_name>
./bin/fluss list servers
./bin/fluss metadata --directory tmp/kraft-combined-logs/_cluster-metadata-0/ 

This approach would simplify the user experience, improve consistency, and make the CLI easier to maintain and extend in the future. What are your thoughts?

[simonsssu]

Agree, this would simplify the user experience and users don't need to know so many different scripts. I will update the docs to add more details.

[wuchong]

@simonsssu thanks, could you draft a new design for CLI based on the simplified single command?

[wuchong]

I found that IGGY has a good CLI design that we can learn from:

https://github.com/apache/iggy?tab=readme-ov-file#cli
https://iggy.apache.org/docs/cli/iggy-cli

[simonsssu]

We've made a simple poc via JCommander, and we can disscuss after I updated the docs. also we can learn from IGGY. cc @wuchong

[wuchong]

Thanks @simonsssu , looking forward to the design doc!

[polyzos]

@simonsssu There are lots of benefits in this work for users, and IGGY also seems really nice... Let me know if you need help with anything.

[simonsssu]

Hi all, here is the brief doc, Brief design doc cc @wuchong @XuQianJin-Stars Could you please help review and give some comments ?