MLG 035 Large Language Models 2

At inference, large language models use in-context learning with zero-, one-, or few-shot examples to perform new tasks without weight updates, and can be grounded with Retrieval Augmented Generation (RAG) by embedding documents into vector databases for real-time factual lookup using cosine similarity. LLM agents autonomously plan, act, and use external tools via orchestrated loops with persistent memory, while recent benchmarks like GPQA (STEM reasoning), SWE Bench (agentic coding), and MMMU (multimodal college-level tasks) test performance alongside prompt engineering techniques such as chain-of-thought reasoning, structured few-shot prompts, positive instruction framing, and iterative self-correction.
Links

• Notes and resources at ocdevel.com/mlg/mlg35 (https://ocdevel.com/mlg/35?utm_source...)

• Build the future of multi-agent software with AGNTCY (https://agntcy.org/?__hstc=105720540....)

• Try a walking desk (https://ocdevel.com/walk?utm_source=p...) stay healthy & sharp while you learn & code
In-Context Learning (ICL)

• Definition: LLMs can perform tasks by learning from examples provided directly in the prompt without updating their parameters.

• Types:

• Zero-shot: Direct query, no examples provided.

• One-shot: Single example provided.

• Few-shot: Multiple examples, balancing quantity with context window limitations.


• Mechanism: ICL works through analogy and Bayesian inference, using examples as semantic priors to activate relevant internal representations.

• Emergent Properties: ICL is an "inference-time training" approach, leveraging the model's pre-trained knowledge without gradient updates; its effectiveness can be enhanced with diverse, non-redundant examples.

Retrieval Augmented Generation (RAG) and Grounding

• Grounding: Connecting LLMs with external knowledge bases to supplement or update static training data.

• Motivation: LLMs' training data becomes outdated or lacks proprietary/specialized knowledge.

• Benefit: Reduces hallucinations and improves factual accuracy by incorporating current or domain-specific information.


• RAG Workflow:

1. Embedding: Documents are converted into vector embeddings (using sentence transformers or representation models).

2. Storage: Vectors are stored in a vector database (e.g., FAISS, ChromaDB, Qdrant).

3. Retrieval: When a query is made, relevant chunks are extracted based on similarity, possibly with re-ranking or additional query processing.

4. Augmentation: Retrieved chunks are added to the prompt to provide up-to-date context for generation.

5. Generation: The LLM generates responses informed by the augmented context.


• Advanced RAG: Includes agentic approaches—self-correction, aggregation, or multi-agent contribution to source ingestion, and can integrate external document sources (e.g., web search for real-time info, or custom datasets for private knowledge).

LLM Agents

• Overview: Agents extend LLMs by providing goal-oriented, iterative problem-solving through interaction, memory, planning, and tool usage.

• Key Components:

• Reasoning Engine (LLM Core): Interprets goals, states, and makes decisions.

• Planning Module: Breaks down complex tasks using strategies such as Chain of Thought or ReAct; can incorporate reflection and adjustment.

• Memory: Short-term via context window; long-term via persistent storage like RAG-integrated databases or special memory systems.

• Tools and APIs: Agents select and use external functions—file manipulation, browser control, code execution, database queries, or invoking smaller/fine-tuned models.


• Capabilities: Support self-evaluation, correction, and multi-step planning; allow integration with other agents (multi-agent systems); face limitations in memory continuity, adaptivity, and controllability.

• Current Trends: Research and development are shifting toward these agentic paradigms as LLM core scaling saturates.
Multimodal Large Language Models (MLLMs)

• Definition: Models capable of ingesting and generating across different modalities (text, image, audio, video).

• Architecture:

• Modality-Specific Encoders: Convert raw modalities (text, image, audio) into numeric ...