research-papers

Deep Learning & Machine Learning Research Paper Collection

📌 Overview

This repository is a comprehensive collection of influential research papers in Deep Learning (DL), Machine Learning (ML), Artificial Intelligence (AI), Generative AI (GenAI), CUDA/Triton, and other related fields. The goal is to provide a structured approach to understanding the evolution, core concepts, and practical implementations of these fields.

⚠️ Disclaimer

This is a personal learning project. The implementations and notes may contain errors or simplifications. Use with caution and always refer to the original papers.

🎯 Project Goals

• Implement all these research papers from scratch.
• Provide scratch implementations to aid learning and understanding.
• Build a structured, categorized, and well-maintained repository for reference.

🌟 Inspiration

Inspired by @saurabhaloneai and expanded with additional research papers and implementations.

📂 Repository Structure

├── Foundational Deep Neural Networks
├── Optimization & Regularization
├── Sequence Modeling
├── Language Modeling
├── Open Source LLMs & Implementation
├── Architecture Innovations
├── Training Methodologies
├── Image Generative Modeling
├── Deep Reinforcement Learning
├── General Machine Learning Papers
├── CUDA & Triton Optimization Papers
├── Generative AI (GenAI)
├── Scaling & Model Optimization
├── Reasoning & Capabilities
├── Inference & Efficiency Techniques
├── Fine-tuning & Adaptation
├── Graph Neural Networks
└── Self-Supervised and Few-Shot Learning

---

📚 Research Papers Collection

1️⃣ Foundational Deep Neural Networks

• Learning Internal Representations by Error Propagation (1987) 📄 Paper 💻 Repo
• Learning Representations by Backpropagating Errors (1986) - Rumelhart et al. 📄 Paper 💻 Repo
• Backpropagation Applied to Handwritten Zip Code Recognition (1989) 📄 Paper 💻 Repo
• Gradient-Based Learning Applied to Document Recognition (1998) - Yann LeCun et al. 📄 Paper 💻 Repo
• ImageNet Classification with Deep Convolutional Networks (2012) - AlexNet 📄 Paper 💻 Repo
• Convolutional Networks for Biomedical Image Segmentation (2015) - U-Net 📄 Paper 💻 Repo
• VGGNet (2014) - Very Deep Convolutional Networks for Large-Scale Image Recognition 📄 Paper 💻 Repo

2️⃣ Optimization & Regularization Techniques

• A Simple Weight Decay Can Improve Generalization (1991) 📄 Paper 💻 Repo
• Deep Sparse Rectified Neural Networks (2011) - ReLU 📄 Paper 💻 Repo
• Deep Residual Learning for Image Recognition (2015) - ResNet 📄 Paper 💻 Repo
• Dropout (2014): Preventing Neural Networks from Overfitting 📄 Paper 💻 Repo
• Batch Normalization (2015): Accelerating Deep Network Training 📄 Paper 💻 Repo
• Layer Normalization (2016) 📄 Paper 💻 Repo
• Gaussian Error Linear Units (2016) - GELU 📄 Paper 💻 Repo
• Adam: A Method for Stochastic Optimization (2014) 📄 Paper 💻 Repo
• Transformers without Normalization (2024) - Yann LeCun’s Dynamic Tanh (DyT) 📄 Paper 💻 Repo

3️⃣ Sequence Modeling

• Continually Running Fully Recurrent Neural Networks (1989) - RNN 📄 Paper 💻 Repo
• Long-Short Term Memory (1997) - LSTM 📄 Paper 💻 Repo
• Gated Recurrent Unit (2014) - GRU 📄 Paper 💻 Repo
• Sequence to Sequence Learning (2014) - Seq2Seq 📄 Paper 💻 Repo
• Neural Machine Translation with Alignment (2014) - Attention 📄 Paper 💻 Repo
• Sparsely-Gated Neural Networks (2017) - Mixture of Experts 📄 Paper 💻 Repo

4️⃣ Language Modeling

• Attention Is All You Need (2017) - Transformer 📄 Paper 💻 Repo
• BERT: Bidirectional Transformers for Language Understanding (2018) 📄 Paper 💻 Repo
• GPT-1 (2018): Improving Language Understanding 📄 Paper 💻 Repo
• GPT-2 (2019): Language Models are Unsupervised Multitask Learners 📄 Paper 💻 Repo
• T5: Unified Text-to-Text Transformer (2019) 📄 Paper 💻 Repo
• Vision Transformer (2020): Image Recognition with Transformers 📄 Paper 💻 Repo
• Llama (2023): Open Foundation Models 📄 Paper 💻 Repo
• Llama 2 (2023): Open Foundation Models 📄 Paper 💻 Repo
• Llama 3 (2024): Open Foundation Models 📄 Paper 💻 Repo
• Mistral (2023): Efficient Transformer Architecture 📄 Paper 💻 Repo
• Mixtral (2023): Sparse Mixture of Experts 📄 Paper 💻 Repo
• Reinforcement Learning from Human Feedback (RLHF) (2022) 📄 Paper 💻 Repo
• Constitutional AI (CAI) (2023) 📄 Paper 💻 Repo
• Direct Preference Optimization (DPO) (2023) 📄 Paper 💻 Repo

5️⃣ Image Generative Modeling

• Generative Adversarial Networks (2014) - GAN 📄 Paper 💻 Repo
• Auto-Encoding Variational Bayes (2013) - VAE 📄 Paper 💻 Repo
• Deep Convolutional GANs (2015) - DCGAN 📄 Paper 💻 Repo
• CycleGAN (2017): Unpaired Image-to-Image Translation 📄 Paper 💻 Repo
• Denoising Diffusion Probabilistic Models (2020) 📄 Paper 💻 Repo
• DALL-E (2021): Text-to-Image Generation 📄 Paper 💻 Repo
• CLIP (2021): Visual Models from Natural Language Supervision 📄 Paper 💻 Repo
• Rotary Position Embeddings (RoPE) (2021) 📄 💻 Repo
• Grouped-Query Attention (GQA) (2023) 📄 💻 Repo
• Multi-Query Attention (MQA) (2019) 📄 💻 Repo
• Sliding Window Attention (2020) 📄 💻 Repo

6️⃣ Deep Reinforcement Learning

• Playing Atari Games (2013) - Deep Q-Learning 📄 Paper 💻 Repo
• Mastering the Game of Go (2016) - AlphaGo 📄 Paper 💻 Repo
• Protein Structure Prediction (2021) - AlphaFold 📄 Paper 💻 Repo
• Proximal Policy Optimization (2017) - PPO 📄 Paper 💻 Repo

7️⃣ General Machine Learning Papers

• Random Forests (2001) - Leo Breiman 📄 Paper 💻 Repo
• XGBoost: A Scalable Tree Boosting System (2016) - Chen and Guestrin 📄 Paper 💻 Repo
• Scaling Laws for Neural Language Models (2020) 📄 Paper 💻 Repo
• DistilBERT (2019): Model Compression 📄 Paper 💻 Repo

8️⃣ CUDA & Triton Optimization Papers

• FlashAttention (2022): Fast and Memory-Efficient Attention 📄 Paper 💻 Repo
• FlashAttention-2 (2023): Faster Attention Mechanism 📄 Paper 💻 Repo
• Triton Compiler (2022): Custom GPU Kernels 📄 Paper 💻 Repo

9️⃣ Generative AI (GenAI)

• InstructGPT (2022): Following Instructions with Human Feedback 📄 Paper 💻 Repo
• LoRA (2021): Low-Rank Adaptation of Large Language Models 📄 Paper 💻 Repo
• Direct Preference Optimization (DPO) (2023) 📄 Paper 💻 Repo
• DeepSeek LLM (2024): Scaling Open-Source Language Models 📄 Paper 💻 Repo

🔟 Scaling & Model Optimization

• Chinchilla (2022): Training Compute-Optimal Large Language Models 📄 Paper 💻 Repo
• Switch Transformers (2022): Scaling to Trillion Parameter Models 📄 Paper 💻 Repo
• GLaM (2021): Efficient Scaling with Mixture of Experts 📄 Paper 💻 Repo

1️⃣1️⃣ Reasoning & Capabilities

• Chain of Thought Prompting (2022): Reasoning with Language Models 📄 Paper 💻 Repo
• Tree of Thoughts (2023): Deliberate Problem Solving 📄 Paper 💻 Repo

1️⃣2️⃣ Inference & Efficiency Techniques

• QLoRA (2023): Efficient Fine-Tuning of Quantized Models 📄 Paper 💻 Repo
• Knowledge Distillation (2022): Comprehensive Survey 📄 Paper 💻 Repo
• Pruning and Quantization Techniques (2022): Model Compression Survey 📄 Paper 💻 Repo

1️⃣3️⃣ Fine-tuning & Adaptation

• P-Tuning (2021): Prompt Tuning with Soft Prompts 📄 Paper 💻 Repo
• Prefix-Tuning (2021): Optimizing Continuous Prompts 📄 Paper 💻 Repo
• AdaLoRA (2023): Adaptive Low-Rank Adaptation 📄 Paper 💻 Repo

1️⃣4️⃣ Graph Neural Networks

• Graph Convolutional Networks (2016) - GCN 📄 Paper 💻 Repo
• Graph Attention Networks (2018) - GAT 📄 Paper 💻 Repo

1️⃣5️⃣ Self-Supervised and Few-Shot Learning

• Bootstrap Your Own Latent (2020) - BYOL 📄 Paper 💻 Repo
• Prototypical Networks (2017) 📄 Paper 💻 Repo

---

🛠 Implementation Guidelines

1. Start Simple: Begin by implementing models with easy-to-use libraries like Scikit-learn or TensorFlow.
2. Replicate Results: Try to reproduce the results from the paper before adding your tweaks.
3. Analyze: Evaluate your implementation with proper metrics and compare with the paper’s results.
4. Visualize: Use plots to understand model performance and feature importance.

---

📥 How to Use

1. Navigate to the relevant category in the repository.
2. Read the research papers and access the implementations.
3. Check the boxes as you complete each implementation.
4. Experiment with implementations to gain deeper insights into each concept.

---

🔗 References & Credits

Inspired by @saurabhaloneai and various research papers from ArXiv, NeurIPS, CVPR, and ICML.

🧠 Maintained by

Maintained by Abinash Pradhan - @abinashpradhan01.