libgemma API refactor - decouple from interactive repl demo specifics, add hello world example using libgemma #82
Conversation
… for intermediate computations, add `model` to EOT token
…s to remove InferenceArgs. TODO: update hello_world example after git commit hash is available for fetching
…LoaderArgs validation, revert ReplGemma EOT token behavior
Separate KV cache from GemmaImpl
…pdate git hash in CMakeLists.txt of hello world after push
…le, add convenience version of constructor (no uncompressed weights)
austinvhuang
changed the title
| // - Bundle runtime parameters as RuntimeConfig | ||
| // - No threadpools within threadpools (inner_pool = dummy) | ||
| // - All tokens accepted | ||
| void GenerateGemma(Gemma& gemma, RuntimeConfig runtime_config, |
There was a problem hiding this comment.
A simplified interface seems useful. Maybe we should also move stream_token and gen inside RuntimeConfig? (Which we should pass by ref = RuntimeConfig&)
There was a problem hiding this comment.
I was thinking of this too, but decided for now to make stateful things more explicit for now. For example, say a user who's new to the code wants to make the randomness reproducible - it's implied what to do from this interface - seed the RNG that gets passed in.
Packing it inside RuntimeConfig adds a level of indirection + opacity - first they have to know or guess that there's an rng that is getting passed in (and not say, initialized somewhere internally), then know to check the internals of RuntimeConfig to find it, then find the place in application codebase where RuntimeConfig is constructed, and finally make the change.
Granted for us w'ere able to track this without too much cognitive overhead, but I want to try to keep things flatly legible as much as we can until other tradeoffs make it obviously needed+beneficial.
There was a problem hiding this comment.
Seems reasonable and fine for now :)
austinvhuang
added
the
copybara-import
austinvhuang
added
copybara-import
4 participants
This PR refactors the library interfaces to facilitate multiple frontends / apps to make better use of libgemma. It removes coupling between the specific interactive interface of
run.cc, it's set of CLI arguments and the core library API ingemma.handgemma.cc. As a test of this library use case, we add a minimal usage example inexamples/hello_world/with minimal setup to interact with the library interfaces.Minor:
See also: