reading list

LLM

• 2401.02412 LLM Augmented LLMs: Expanding Capabilities through Composition
  • Related Reddit: Expanding Capabilities through Composition (CALM) : r/LocalLLaMA
• 2312.15166 SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling
• GitHub - pratyushasharma/laser: The Truth Is In There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction
  • Simultaneously Enhance Performance and Reduce LLM Size with no Additional Training - LASER by Microsoft : r/LocalLLaMA
• mixtral: 2401.04088\ Mixtral of Experts
• Phi model is free ! microsoft/phi-2 · Hugging Face
• LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
• GitHub - pratyushasharma/laser: The Truth Is In There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction
• Paper page - LLaMA Beyond English: An Empirical Study on Language Capability Transfer
• GitHub - microsoft/TransformerCompression: For releasing code related to compression methods for transformers, accompanying our publications
• Evolving New Foundation Models: Unleashing the Power of Automating Model Development

QA

• Paper page - ChatQA: Building GPT-4 Level Conversational QA Models

MoE

• Paper page - Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
• Paper page - Soaring from 4K to 400K: Extending LLM’s Context with Activation Beacon
• Mixture of Experts Explained
• mlabonne/phixtral-4x2_8 · Hugging Face
• Mixture of Experts for Clowns (at a Circus)

RAG

• 대규모 언어 모델을 위한 검색-증강 생성(RAG) 기술 현황 - 2/2편 - 읽을거리&정보공유 - 파이토치 한국 사용자 모임
• Long-Context Retrieval Models with Monarch Mixer · Hazy Research
• RAFT: RAG 기법을 활용한 LLM 검색 증강형 미세조정(RAG + FineTuning) - 읽을거리&정보공유 - 파이토치 한국 사용자 모임

Code

• skypilot/llm/codellama at master · skypilot-org/skypilot · GitHub

Benchmark

• EQ-Bench Leaderboard

LLM Task

• [2402.10171] Data Engineering for Scaling Language Models to 128K Context

CPU

LLaMA Now Goes Faster on CPUs

References

• Daily Papers - Hugging Face