MiniMind

MiniMind: Train a 26M-Parameter GPT from Scratch in Just 2 Hours!

MiniMind is an open-source project aimed at lowering the barrier to entry for training large language models (LLMs). With just a minimal cost of around 3 RMB and a training time of 2 hours on a single NVIDIA 3090 GPU, users can train a lightweight 26M-parameter GPT model from scratch.

Key Features:

• Lightweight Model: The smallest version of MiniMind is only 25.8M parameters, making it accessible for personal GPUs.
• Comprehensive Training Process: The project includes detailed code for pre-training, supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and model distillation.
• Open Source: All core algorithms are implemented from scratch using PyTorch, without relying on third-party libraries.
• Multi-Modal Capabilities: MiniMind has been extended to support visual multi-modal tasks with MiniMind-V.
• User-Friendly: The project serves as a tutorial for beginners in LLMs, providing a hands-on experience in training and understanding the underlying mechanisms of large models.

Benefits:

• Cost-Effective: Users can experience the entire process of building a language model for less than 3 RMB.
• Educational Resource: Ideal for those looking to learn about LLMs and their training processes.
• Community Driven: Encourages contributions and improvements from the community, fostering a collaborative environment for AI development.

Highlights:

• Supports single and multi-GPU training.
• Compatible with popular frameworks like transformers and trl.
• Provides a simple API for integration with third-party applications.

Join the MiniMind community and start your journey in AI model training today!