So I created Jacq, a DSL and compiler. A jacq file describes a model’s configuration, weights, layers, and pipeline. Jacq turns it into a standalone CUDA inference runtime and a Python weight exporter. Kernel descriptions can also generate CUDA harnesses.
The easiest way to try it is the TinyLlama example in the README. Building, validating, visualizing, and generating CUDA only require Rust. Of course, running inference does require an NVIDIA GPU, CUDA, Python dependencies, and weights. Jacq doesn't support other GPU vendors (AMD, etc) or Metal, yet (PR's welcome!)
There's also an experimental vLLM patch to compile and patch your jacq file into a vLLM build, as well.
Feedback very welcome!