Skip to main content

Learning path

AI Application Development learning path: everything to study, topic by topic

Comprehensive assessment of AI application development skills covering LLM fundamentals, RAG systems, agent frameworks, evaluation and safety, production deployment, and advanced AI engineering including fine-tuning and RLHF.

Short answer

To learn AI Application Development, work through 7 modules and 27 topics in the order they build on each other. It starts with LLM Fundamentals, then moves on to RAG & Knowledge Systems and Agent Frameworks. Use the map as a checklist, or measure your level and study only what is missing.

7
modules
27
topics
60
min to assess

What you will study

Topics are grouped by module. Open one to see its objectives and key concepts.

1 · LLM Fundamentals4 topics
  • Tokenization & Embeddings

    How LLMs tokenize text, BPE/SentencePiece algorithms, embedding spaces, semantic similarity, and vector representations of language

    Objectives

    • Explain how tokenization affects model input limits
    • Use embeddings for semantic similarity tasks

    Key concepts

    • BPE tokenization
    • SentencePiece
    • token limits
    • embedding dimensions
    • cosine similarity
    • semantic search
    • token counting
  • Prompt Design Basics

    Crafting effective prompts, system/user/assistant roles, few-shot examples, chain-of-thought prompting, and prompt templates

    Objectives

    • Design prompts that produce consistent, high-quality outputs
    • Apply chain-of-thought for complex reasoning tasks

    Key concepts

    • system prompts
    • few-shot prompting
    • chain-of-thought
    • zero-shot prompting
    • role assignment
    • output formatting
    • prompt injection awareness
  • Model Selection

    Choosing the right LLM for a task — comparing capabilities, cost, latency, context windows, and specialization across providers

    Objectives

    • Select appropriate models based on task requirements and constraints
    • Compare trade-offs between model size, cost, and capability

    Key concepts

    • context window
    • parameter count
    • inference cost
    • model benchmarks
    • open-source vs proprietary
    • task-specific models
    • multimodal capabilities
  • API Integration Patterns

    Working with LLM provider APIs — authentication, request/response formats, streaming, rate limiting, error handling, and SDK usage

    Objectives

    • Integrate LLM APIs with proper error handling and retries
    • Implement streaming responses for real-time user experience

    Key concepts

    • chat completions API
    • streaming responses
    • rate limiting
    • retry strategies
    • API key management
    • structured output
    • function calling basics
2 · RAG & Knowledge Systems4 topics
  • Vector Databases

    Storing and querying embeddings in vector databases — Pinecone, Weaviate, Qdrant, pgvector, indexing algorithms, and distance metrics

    Objectives

    • Choose the right vector database and index type for a given workload
    • Configure distance metrics and understand their trade-offs

    Key concepts

    • HNSW index
    • IVF index
    • cosine distance
    • L2 distance
    • metadata filtering
    • namespace isolation
    • index tuning
  • Chunking Strategies

    Splitting documents for effective retrieval — fixed-size, recursive, semantic, and document-structure-aware chunking approaches

    Objectives

    • Select chunking strategies based on document type and retrieval needs
    • Tune chunk size and overlap for optimal retrieval quality

    Key concepts

    • fixed-size chunking
    • recursive character splitting
    • semantic chunking
    • chunk overlap
    • document structure parsing
    • chunk size optimization
    • parent-child chunking
  • Retrieval Pipelines

    Building end-to-end retrieval pipelines — query preprocessing, embedding, top-k retrieval, re-ranking, and context window assembly

    Objectives

    • Design a complete RAG pipeline from document ingestion to response generation
    • Apply re-ranking to improve retrieval precision

    Key concepts

    • query expansion
    • hypothetical document embeddings
    • re-ranking models
    • top-k retrieval
    • context window packing
    • citation tracking
    • retrieval-augmented generation pipeline
  • Hybrid Search

    Combining dense and sparse retrieval — BM25 + vector search, reciprocal rank fusion, keyword-semantic blending, and query routing

    Objectives

    • Implement hybrid search combining semantic and keyword-based retrieval
    • Configure reciprocal rank fusion weights for domain-specific needs

    Key concepts

    • BM25
    • sparse embeddings
    • reciprocal rank fusion
    • query routing
    • dense-sparse combination
    • keyword fallback
    • domain-specific lexicon
3 · Agent Frameworks4 topics
  • Agent Architectures

    Designing LLM-powered agents — ReAct, plan-and-execute, reflection loops, state machines, and workflow orchestration patterns

    Objectives

    • Design agent architectures for different task complexities
    • Implement ReAct and plan-and-execute patterns

    Key concepts

    • ReAct pattern
    • plan-and-execute
    • reflection loops
    • state machines
    • agent memory
    • agent loops
    • orchestration frameworks
  • Tool Use & Function Calling

    Enabling LLMs to use external tools — function calling APIs, tool schema definition, result parsing, and error recovery

    Objectives

    • Define tool schemas that LLMs can reliably use
    • Handle tool execution errors gracefully in agent loops

    Key concepts

    • function calling schema
    • tool description design
    • parameter validation
    • parallel tool calls
    • tool result injection
    • error recovery
    • tool selection strategies
  • Multi-Agent Systems

    Coordinating multiple AI agents — delegation, supervisor patterns, communication protocols, shared memory, and conflict resolution

    Objectives

    • Design multi-agent systems with clear delegation and communication patterns
    • Choose between orchestration and choreography for agent coordination

    Key concepts

    • supervisor agent
    • delegation patterns
    • agent-to-agent communication
    • shared memory
    • consensus mechanisms
    • specialized agents
    • orchestration vs choreography
  • Planning & Reasoning

    Advanced agent reasoning — task decomposition, self-critique, tree-of-thought, backtracking, and dynamic replanning strategies

    Objectives

    • Implement task decomposition strategies for complex multi-step problems
    • Design self-critique mechanisms that improve agent reliability

    Key concepts

    • task decomposition
    • tree-of-thought
    • self-critique
    • backtracking
    • dynamic replanning
    • goal verification
    • chain-of-thought reasoning
4 · Evaluation & Safety4 topics
  • LLM Evaluation Metrics

    Measuring LLM output quality — automated metrics (BLEU, ROUGE, BERTScore), LLM-as-judge, human evaluation, and A/B testing frameworks

    Objectives

    • Select appropriate evaluation metrics for different LLM tasks
    • Design LLM-as-judge evaluation pipelines

    Key concepts

    • BLEU score
    • ROUGE score
    • BERTScore
    • LLM-as-judge
    • human evaluation rubrics
    • inter-annotator agreement
    • evaluation dataset curation
  • Hallucination Detection

    Identifying and mitigating LLM hallucinations — factual grounding, citation verification, self-consistency checks, and confidence calibration

    Objectives

    • Implement hallucination detection mechanisms in production systems
    • Design grounding strategies that reduce confabulation

    Key concepts

    • factual grounding
    • citation verification
    • self-consistency
    • confidence scores
    • knowledge conflicts
    • abstention strategies
    • retrieval-augmented fact checking
  • Content Safety

    Building safe AI applications — content moderation, output filtering, PII detection, bias mitigation, and compliance frameworks

    Objectives

    • Implement multi-layered content safety systems
    • Design guardrails that balance safety with usability

    Key concepts

    • content moderation APIs
    • output filtering
    • PII detection
    • bias detection
    • toxicity classification
    • guardrails frameworks
    • compliance requirements
  • Red-Teaming

    Adversarial testing of AI systems — prompt injection attacks, jailbreaking, data extraction, and systematic vulnerability assessment

    Objectives

    • Conduct systematic red-teaming exercises against LLM applications
    • Design defense-in-depth strategies against adversarial attacks

    Key concepts

    • prompt injection
    • jailbreaking techniques
    • data extraction attacks
    • indirect prompt injection
    • automated red-teaming
    • vulnerability taxonomies
    • defense-in-depth
5 · Production AI Systems4 topics
  • Latency Optimization

    Reducing response times in AI systems — streaming, parallel processing, speculative decoding, prompt compression, and edge deployment

    Objectives

    • Optimize end-to-end latency in LLM-powered applications
    • Apply prompt compression and model quantization techniques

    Key concepts

    • streaming responses
    • prompt compression
    • speculative decoding
    • parallel API calls
    • edge inference
    • model quantization
    • batching strategies
  • Cost Management

    Controlling AI infrastructure costs — token optimization, model tiering, prompt caching, usage monitoring, and budget alerting

    Objectives

    • Design cost-efficient AI architectures with model tiering
    • Implement usage monitoring and budget alerting systems

    Key concepts

    • token optimization
    • model tiering
    • prompt caching
    • usage metering
    • budget alerts
    • cost-per-query analysis
    • batch vs real-time trade-offs
  • Caching Strategies

    Caching for AI applications — semantic caching, exact-match caching, embedding-based cache lookup, TTL policies, and cache invalidation

    Objectives

    • Implement semantic caching to reduce redundant LLM calls
    • Design cache invalidation strategies for dynamic knowledge bases

    Key concepts

    • semantic caching
    • exact-match caching
    • embedding similarity threshold
    • cache TTL policies
    • cache invalidation
    • cache warming
    • multi-tier caching
  • Observability for AI

    Monitoring AI systems in production — tracing LLM calls, logging prompts and completions, quality metrics dashboards, and drift detection

    Objectives

    • Build observability pipelines for LLM-powered applications
    • Detect quality degradation and model drift in production

    Key concepts

    • LLM call tracing
    • prompt/completion logging
    • quality dashboards
    • latency percentiles
    • token usage tracking
    • model drift detection
    • feedback loop integration
6 · Advanced AI Engineering2 topics
  • Fine-Tuning Pipelines

    Adapting foundation models — data preparation, LoRA/QLoRA, training loops, hyperparameter tuning, evaluation, and model distillation

    Objectives

    • Design end-to-end fine-tuning pipelines with proper evaluation
    • Apply LoRA/QLoRA for parameter-efficient fine-tuning

    Key concepts

    • LoRA adapters
    • QLoRA
    • training data curation
    • hyperparameter tuning
    • catastrophic forgetting
    • model distillation
    • evaluation during training
  • RLHF & DPO

    Aligning models with human preferences — reward modeling, PPO training, DPO, preference data collection, and alignment evaluation

    Objectives

    • Design preference data collection workflows for alignment
    • Compare RLHF and DPO approaches for model alignment

    Key concepts

    • reward modeling
    • PPO training
    • DPO
    • preference datasets
    • KL divergence penalty
    • alignment tax
    • RLHF vs DPO trade-offs
7 · Expert AI Application Engineering5 topics
  • RAG Architecture Patterns

    Designing production RAG systems: chunking strategies, embedding models, retrieval optimization, reranking, and evaluation

    Objectives

    • Design a production RAG system optimized for retrieval quality and latency
    • Evaluate RAG pipelines using RAGAS and identify failure modes

    Key concepts

    • chunking strategies
    • embedding models
    • vector search
    • hybrid retrieval
    • reranking
    • RAG evaluation (RAGAS)
    • context window management
  • AI Agent Design

    Building production AI agents with tool use, memory management, planning loops, and multi-agent orchestration

    Objectives

    • Design production AI agents with appropriate memory and tool architectures
    • Evaluate and ensure safety of AI agents in production environments

    Key concepts

    • ReAct pattern
    • tool calling
    • memory types (episodic/semantic)
    • agent planning
    • multi-agent frameworks
    • agent evaluation
    • safety guardrails
  • Multi-Modal AI Applications

    Building applications that combine text, vision, audio, and structured data using multi-modal foundation models

    Objectives

    • Design multi-modal AI pipelines for document understanding and visual QA
    • Evaluate trade-offs between single-modal and multi-modal approaches

    Key concepts

    • vision-language models
    • document understanding
    • image generation integration
    • audio transcription
    • multi-modal retrieval
    • modality fusion
  • Production LLM Reliability

    Building reliable LLM-powered applications: fallbacks, retry strategies, rate limiting, cost management, and output validation

    Objectives

    • Design LLM application reliability patterns including fallbacks and circuit breakers
    • Implement output validation strategies for production LLM applications

    Key concepts

    • fallback models
    • retry with backoff
    • rate limit handling
    • token cost management
    • output validation
    • structured outputs
    • circuit breakers for LLMs
  • AI Gateway & Routing Patterns

    Implementing AI gateways for centralized model routing, cost attribution, semantic caching, and observability

    Objectives

    • Design an AI gateway that centralizes LLM routing, caching, and observability
    • Implement model routing strategies based on cost, latency, and capability

    Key concepts

    • AI gateway
    • model routing
    • semantic caching
    • cost attribution
    • latency routing
    • LiteLLM
    • Portkey
    • observability for LLMs

Levels: Junior, Mid, Senior, and Expert

Explain it in your own words

Graded by AI · no account · 1 question

No options to guess from: you write, the AI grades against the topic's rubric and tells you what level your answer reads at.

0 / 600

Already studying AI Application Development?

The assessment measures topic by topic, and your plan skips what you already know.

Measure my level

Open to read. An account is only needed to be measured.

Frequently asked questions

Where should I start studying AI Application Development?

Start with LLM Fundamentals: Tokenization & Embeddings, Prompt Design Basics, and Model Selection. The map orders topics by what they depend on, so each one prepares the next.

How long does a full AI Application Development assessment take?

A full assessment of this path takes about 60 minutes. Study time depends on where you start; a diagnostic shows which topics you can skip.

Do I need an account to use this page?

No. The map, objectives and key concepts are open. You only need an account to be measured, get your gap by topic and receive a plan.