6 posts tagged with "training"

Transformer Lab now has built-in support for dstack. If your lab has GPUs scattered across clouds, on-prem boxes, or a mix of both, this improves how you run jobs.

For many looking to experiment with machine learning, the biggest barrier to entry is access to hardware. GPUs are expensive, hard to find, and even harder to share across a team. Big cloud hosting providers have complex interfaces, pricing models, and try to lock you into their ecosystem and tooling.

Using Runpod with Transformer Lab changes that. Now you can spin up GPU-backed experiments quickly from the comfort of your own system.

Our open source research initiative recently closed a new round of funding and today, we launched the public beta for Transformer Lab for Teams: a modern operating system for AI research labs. It’s open source and free to use.

🎉 Transformer Lab just expanded beyond image diffusion! We're thrilled to announce text diffusion model support so you can train, evaluate, and interact with cutting-edge text diffusion architectures like BERT, Dream, and LLaDA directly in Transformer Lab.

What's included in this release​

• 🚀 Text Diffusion Server for interactive generation with BERT, Dream, and LLaDA models
• 🏋️ Text Diffusion Trainer for fine-tuning with masked-language and diffusion-style alignment workflows
• 📊 Text Diffusion Evaluator for benchmarking with the EleutherAI LM Evaluation Harness

Transformer Lab is excited to announce robust multi-GPU support for fine-tuning large language models. This update allows users to leverage all available GPUs in their system, dramatically reducing training times and enabling work with larger models and datasets.

This post details our journey to fine-tune smolLM 135M, a compact language model, for Python code completion.

We chose smolLM 135M for its size, which allows for rapid iteration. Instead of full fine-tuning, we employed LoRA (Low-Rank Adaptation), a technique that introduces trainable "adapter" matrices into the transformer layers. This provides a good balance between parameter efficiency and achieving solid results on the downstream task (code completion).

Transformer Lab handled the training, evaluation, and inference, abstracting away much of the underlying complexity. We used the flytech/python codes-25k dataset, consisting of 25,000 Python code snippets, without any specific pre-processing. Our training setup involved a constant learning rate, a batch size of 4, and an NVIDIA RTX 4060 GPU.

The Iterative Fine-tuning Process: Nine Runs to Success​

The core of this project was an iterative refinement of LoRA hyperparameters and training duration. We tracked both the training loss and conducted qualitative assessments of the generated code (our "vibe check") to judge its syntactic correctness and logical coherence. This combination of quantitative and qualitative feedback proved crucial in guiding our parameter adjustments.