New in January 2024

Explore a curated collection of the best open source ML models

In January, we published a new model library to better present the best open source machine learning models.

One focus in redesigning the model library was making it easier to navigate the taxonomy of open source models. A foundation model like Stable Diffusion comes in different versions, variants, and sizes, with different implementations such as model server implementation and additional ControlNet image pipelines (which themselves come in different variants like Depth and Canny).

The new model library categorizes models by task (text to image, text generation, text to audio, etc), by family (Stable Diffusion, Mistral, Audiogen, etc), and by publisher (Stability AI, Mistral AI, Meta, etc), then provides detailed information about each model’s version, variant, size, optimizations, license, and other essential properties.

Try the model library for yourself today, or learn more about what’s new in the changelog announcement.

General availability for L4 GPUs

The NVIDIA L4 GPU, based on NVIDIA’s latest Ada Lovelace architecture, is now available for model inference on Baseten starting at $0.8484/hour — about 70% of the base cost of an A10G-based instance.

The stat breakdown for L4 vs A10G shows some interesting tradeoffs:

• The L4 has 121 TFLOPS of FP16 Tensor Compute (vs 70 TFLOPS on A10G)

• The L4 has 24 GB of VRAM (matching 24GB on A10G)

• The L4 has 300 GB/s of memory bandwidth (vs 600 GB/s on A10G)

In summary, the L4 has almost twice the compute power, but only half the memory bandwidth of the other 24-gigabyte GPU, the A10G. Most model inference for LLMs and other autoregressive transformers models tends to be memory-bound, meaning that the A10G is still a better pick for tasks like LLM chat. But the L4 is useful for cheaper inference on compute-bound workloads.

Check out our announcement of L4 availability for more details on these new GPUs!

Benchmarking model inference

We’re always working to make model inference as fast and cheap as possible. But that requires a nuanced understanding of what “fast” means, along with a clear picture of the different factors that affect inference speed and cost. These details come from running performance benchmarks for model inference.

Benchmarking considerations vary somewhat from model to model. We wrote guides for setting up and interpreting benchmarks for two of the most widely used types of models:

• Performance benchmarks for large language models

• Performance benchmarks for image generation models like Stable Diffusion

One major lever for improving model performance is quantization. Quantization lets you run large models on smaller or fewer GPUs by reducing the size of model weights (e.g. from 16-bit floating point numbers to 8-bit integers). But quantization can be a tricky process and has the potential to severely worsen model output quality.

Our new introduction to quantizing ML models breaks down the advantages and risks of quantizing models for inference, based in part on our work quantizing Mixtral 8x7B for more efficient inference.

We’ll be back next month with more from the world of open source ML!

Thanks for reading,

— The team at Baseten