Vibe Coding Discoverawesome-llm-pretraining
View on GitHubAwesome LLM pre-training resources, including data, frameworks, and methods.
Curated awesome list of LLM pre-training resources: technical reports from Llama, Qwen, DeepSeek and others, training frameworks (Megatron-LM, DeepEP, DeepGEMM), open datasets and data-filtering methods. Useful as a reading and reference index for anyone pre-training or studying LLM training pipelines.
Use Cases
Finding LLM pre-training technical reportsChoosing a pre-training framework like Megatron-LMSelecting open-source pre-training datasetsLearning MoE training techniquesStudying data curation and filtering methodsComparing dense vs hybrid/Mamba architecturesReproducing open-data LLM training recipesSurveying scaling and learning-rate annealing strategiesAccelerating training kernels (FP8 GEMM, expert parallelism)
Built With
- Frameworks
- Megatron-LM · DeepEP · DeepGEMM · Liger Kernel · Comet
Tags
awesome-list · llm-pretraining · training · datasets · data-curation · moe · model-architecture · scaling · training-strategies · technical-reports · pretraining-data · resource-collection · llm · research