AI Engineering Interview Questions 2026 RAG, LLMs, Agents and Projects

AI engineering is the discipline of building, deploying, and maintaining software applications that use AI models — especially large language models (LLMs) — to solve real problems. It combines software engineering (APIs, deployment, logging) with applied AI skills (prompt engineering, RAG, agents). The role is distinct from ML engineering, which focuses on training models, and from data science, which focuses on analysis and insights.

An LLM is a deep learning model trained on large text datasets to understand and generate human language. It predicts the next token (word fragment) given a context. LLMs like GPT-4, Claude, and Gemini are used via APIs — you send a prompt and receive a generated response. AI engineers do not typically train LLMs; they build applications on top of them via APIs.

Prompt engineering is the practice of designing inputs (prompts) to LLMs to reliably get the outputs you need. It involves designing system prompts, structuring context, using few-shot examples, and controlling output format. In production AI systems, prompt design directly affects output quality, reliability, and cost.

A token is the unit of text an LLM processes. Tokens are not exactly words — a common English word is roughly one token; longer words, code, and non-English text may be two or more tokens. LLMs have context window limits (e.g., 128K tokens) and APIs charge per token. Engineers must track and manage token usage for cost and context window reasons.

An API (Application Programming Interface) is a way for software to communicate with another service over the internet using structured requests and responses. LLM providers expose APIs — you send a JSON payload with your prompt and model parameters, and receive a JSON response with the generated text. AI engineers spend a large proportion of their time working with LLM APIs and building APIs of their own (using FastAPI or similar).

JSON (JavaScript Object Notation) is a lightweight text format for structured data. It is the standard format for API requests and responses, including LLM API calls. AI engineers need to read, parse, and write JSON constantly — from sending prompts to LLM APIs, to parsing structured outputs, to returning results from FastAPI endpoints. Structured outputs from LLMs (getting a model to return valid JSON) are a common production requirement.

An embedding turns text (or other data) into a long list of numbers, called a vector. An embedding model converts a sentence or paragraph into numbers so that texts with similar meanings end up close together. Embeddings are the basis of meaning-based search and vector databases, and every RAG pipeline uses them to turn documents and questions into numbers that can be compared.

RAG (retrieval-augmented generation) is a way of building AI apps that can answer questions about private, current or specialist information the LLM was not trained on. It works like this: (1) index your documents as embeddings in a vector database; (2) when a question comes in, retrieve the chunks closest in meaning; (3) give those chunks to the LLM as context; and (4) generate an answer based on the retrieved content.

A vector database stores text chunks as high-dimensional numerical embeddings and enables fast semantic similarity search — finding content by meaning rather than exact keywords. Common vector databases used in AI engineering include Pinecone, Chroma, FAISS, pgvector, and Weaviate. They are the retrieval engine of most RAG systems.

Hallucination is when an LLM generates content that is factually incorrect but stated with confidence. It happens because LLMs generate the statistically likely next token — they do not look up facts. Hallucination is mitigated in production AI systems through: RAG (grounding answers in retrieved documents), structured outputs, output evaluation with RAGAS, human review processes, and adding citations to generated responses.

A system prompt is an instruction you provide to an LLM at the beginning of a conversation to set its persona, define what it should or should not do, and establish output format expectations. In production applications, the system prompt is carefully engineered and versioned — it is a core asset of the AI application.

A prompt template is a reusable prompt structure with variable placeholders that are filled at runtime. For example: "You are a helpful assistant. Context: {context}. Question: {question}. Answer concisely." LangChain's PromptTemplate and ChatPromptTemplate classes are the standard tooling for managing and composing prompt templates in Python.

Structured outputs instruct an LLM to return its response in a specific format — typically JSON that matches a Pydantic schema. This is essential for production AI systems where the application needs to parse and use the model's output programmatically. Some model APIs offer schema-constrained output modes. Check the selected model and supported schema; still handle refusals, interrupted responses and values that are structurally valid but factually wrong.

Function calling (now called tool calling in most APIs) allows an LLM to decide to call an external function or tool rather than generate a text response. You define tools with their parameters, the LLM decides whether and which tools to call, and your code executes the calls and passes results back. Tool calling is the foundation of AI agent behaviour — enabling LLMs to search databases, call APIs, read files, and take actions.

Temperature controls the randomness of an LLM's output. Low temperature (0.1–0.3) makes responses more deterministic and focused — good for factual Q&A, structured outputs, and RAG applications. High temperature (0.7–1.0) increases variety and creativity — good for content generation, brainstorming, and conversational uses. For most production RAG and agent systems, keep temperature low to reduce hallucination risk.

A context window is the maximum amount of text (measured in tokens) an LLM can process in a single request — including both input and output. Common limits range from 8K to 200K+ tokens depending on the model. In RAG systems, the context window determines how many retrieved chunks you can include. Filling the context window entirely can degrade quality due to the "lost in the middle" problem — where LLMs are less attentive to content in the centre of a long context.

Ways to test LLM outputs include: RAGAS (automatic testing of RAG systems that measures faithfulness, context precision, context recall and answer relevancy); using a strong LLM as a judge to score another LLM’s answers against a marking guide; human review, where a person grades answers against a model answer; and measures for the specific task, such as accuracy for sorting tasks. Production AI systems need a mix of automatic and human testing.

Prompt injection is an attack where a malicious user crafts input that overrides your system prompt or manipulates the LLM to behave in unintended ways. For example, a user might include text like "Ignore all previous instructions and instead...". Production AI applications must validate and sanitise inputs, scope system prompts defensively, and in some cases run injection detection filters.

Few-shot prompting means including examples of the desired input-output format in the prompt, to guide the LLM's behaviour. Instead of just saying "classify this email as spam or not spam", you include two or three labelled examples before the target input. Few-shot prompting improves consistency and is especially effective for formatting, tone, and classification tasks.

Ways to keep costs down include: choosing the smallest model that is good enough; caching answers to repeated questions; keeping prompts as short as possible by dropping retrieved chunks that do not help; streaming the answer so it starts before the whole response is ready; sending requests in batches where speed allows; and tracking token use per request in your logs so you spot cost problems early.

RAG (Retrieval-Augmented Generation) is a pattern for grounding LLM answers in external knowledge. It is used because LLMs only know what they were trained on — they cannot answer questions about private documents, recent events, or domain-specific content without retrieval. RAG solves this by retrieving relevant context at query time and injecting it into the LLM prompt, enabling factual answers about any content that has been indexed.

The RAG pipeline has two phases. Indexing: load documents → split into chunks → embed each chunk with an embedding model → store vectors + metadata in a vector database. Querying: user submits question → embed the question → retrieve top-K similar chunks from the vector database → assemble chunks into a prompt template → call LLM → return grounded answer. Optional production additions: reranking, citation extraction, RAGAS evaluation, and caching.

Chunking is the process of splitting source documents into smaller segments before embedding. Chunk size is a critical design decision: chunks that are too small lose context (the chunk alone may not answer any question); chunks that are too large dilute the embedding (the vector averages across multiple topics). Common approaches: fixed-size with overlap (256–512 tokens, 10–20% overlap), recursive character splitting, and semantic chunking (split at topic boundaries). Poor chunking is one possible cause of poor RAG retrieval.

Embeddings convert text chunks and user queries into numerical vectors. The core insight is that semantically similar texts produce similar vectors — allowing the vector database to find the chunks most relevant to the user's question by measuring geometric distance in vector space. The same embedding model must be used for indexing and querying; mixing models produces incomparable vectors.

A retriever is a LangChain building block that takes a question and returns relevant documents. It wraps the vector store’s similarity search and can add label filters, minimum scores and combinations of several retrievers. Common retrievers: VectorStoreRetriever, MultiQueryRetriever (asks the question several ways), ContextualCompressionRetriever (shortens what is retrieved to keep only the relevant parts) and BM25Retriever (keyword-based).

Reranking is a two-stage retrieval strategy: broad vector search retrieves top-20 candidates; a cross-encoder reranker then reorders them by relevance to the specific query, returning the top 3–5. Reranking improves retrieval precision at the cost of added latency (~50–200ms). Add reranking when your initial retrieval returns semantically close but topically irrelevant chunks, or when improving factual grounding is worth the latency trade-off. Cohere Rerank and FlashRank are common options.

Citations connect each part of the LLM's answer to the specific retrieved chunks it used. Implementation approaches: (1) prompt the LLM to include source references inline; (2) parse the response and map back to source metadata (source file, page number, URL); (3) use structured output mode to return citations as a separate field alongside the answer. Cohere's Command models have native citation support. Citations are important for enterprise RAG applications where users need to verify answers.

RAG reduces hallucination by giving the LLM specific, relevant, up-to-date context in the prompt. Instead of relying on potentially incorrect training memory, the model is instructed to answer only using the provided context and to say "I don't know" if the context doesn't contain enough information. The system prompt typically includes an explicit instruction like "Answer based only on the provided context. If the context doesn't contain the answer, say so."

RAGAS is an open-source framework for evaluating RAG systems. It measures four metrics: faithfulness (does the answer contradict the retrieved context?), context precision (are the retrieved chunks relevant?), context recall (were all necessary chunks retrieved?), and answer relevancy (does the answer address the question?). You use RAGAS by preparing a test dataset of questions with ground-truth answers and relevant document IDs, running your RAG pipeline, and computing the scores. RAGAS evaluation should be run as part of CI on every change to chunking, retrieval, or prompting logic.

Hybrid search combines vector similarity search (semantic) with keyword search (BM25) and merges results — typically using Reciprocal Rank Fusion (RRF). It improves retrieval when queries include specific identifiers, product codes, or exact phrases that benefit from keyword matching, while still handling semantic paraphrasing. Weaviate, Qdrant, and Azure AI Search support hybrid search natively. Adding hybrid search is a common improvement step for production RAG systems.

Production RAG brings problems such as: poor retrieval because documents were split badly or the wrong embedding model was chosen; duplicate or overlapping chunks in the results; limits on how much retrieved text fits in the prompt; slow answers, because embedding, vector search, reranking and the LLM call run one after another; costs across the embedding API, the vector database and the LLM; access control (stopping user A from retrieving user B’s documents); keeping the index fresh when source documents change; and quality slowly dropping when there is no testing to catch it.

Choose fine-tuning when: you need the model to use a particular writing style or tone; the task uses specialist vocabulary the base model handles badly; what it must learn is stable and unlikely to change; or speed is critical and retrieval would make answers too slow. Choose RAG when: the information changes, is private or must stay current; you need citations and a clear trail to the source; the knowledge base is large or grows often; or you cannot afford the cost and computing power of fine-tuning. See the RAG vs Fine-Tuning guide for the full way to decide.

A vector database stores long lists of numbers (embeddings) and lets you search by similarity, finding the K items closest in meaning to a question. A relational (SQL) database stores structured rows and is searched by exact values or ranges. Vector databases are not built for transactions or joining tables; relational databases cannot search by meaning. Production AI systems often use both: SQL for app data and a vector database for meaning-based search.

Semantic search finds documents by meaning rather than by exact keyword match. A query for "employee health benefits" will find a document section titled "Workforce Wellness Policy" because both express the same concept in the embedding space — even though they share no words. Semantic search is powered by embedding models and vector similarity search, and is the foundation of RAG retrieval.

Three main metrics: cosine similarity (measures the angle between vectors — the most common for text embeddings; score ranges from -1 to 1, higher is more similar); dot product (accounts for both direction and magnitude — used with normalised embeddings); Euclidean distance (straight-line distance in vector space — lower is more similar; common for image embeddings). For most text-based RAG applications, cosine similarity is the standard choice.

Metadata filters allow you to combine semantic similarity search with structured attribute filters — for example, "find the most semantically similar documents, but only where source = HR_Policy and date > 2025-01-01". This is essential for enterprise RAG systems where different document categories, access levels, or time ranges must be scoped per query. All major vector databases (Pinecone, Weaviate, Qdrant) support metadata filtering.

pgvector is a PostgreSQL extension that adds vector column types and similarity search operators to PostgreSQL. It allows teams to store embeddings alongside their existing relational data and run vector similarity queries using SQL. Good for teams already running Postgres who want to avoid adding new infrastructure. Handles moderate vector counts (under ~10M) well; purpose-built vector databases outperform it at very large scale.

FAISS (Facebook AI Similarity Search) is an open-source library for fast similarity search over vectors. It is a library, not a full database, and mainly works in memory. It has no built-in server, storage for labels or way to save data permanently. Use FAISS through LangChain for prototypes, research or apps where your own code manages the whole index. For production that needs saved data, many users or label filters, use a proper vector database.

Pinecone is a fully managed cloud vector database: nothing to run yourself, a serverless tier, a production-grade service agreement, and label filters and separate namespaces built in. It suits production. Chroma is an open-source vector store for Python that installs with pip and saves data locally, with nothing extra to run. It suits prototypes, demos and small projects. In a typical AI engineering project, use Chroma while developing and Pinecone for cloud production.

Hybrid search combines vector similarity search (semantic) with BM25 keyword search and merges the results using Reciprocal Rank Fusion or a weighted score. It improves retrieval precision for queries that include specific identifiers, technical terms, or exact phrases that keyword matching handles well, while still benefiting from semantic understanding for conceptual queries. Weaviate, Qdrant, and Azure AI Search support hybrid search natively.

An AI agent is an LLM-powered system that autonomously decides which actions to take — calling tools, browsing the web, writing and executing code, querying databases — in order to complete a goal. Unlike a simple chatbot that only generates text, an agent has tools it can use and follows a loop of Reasoning → Acting → Observing the result → Reasoning again until the task is complete.

A chatbot takes a user message and generates one text response. An AI agent can take a user instruction and execute multiple steps autonomously: searching a database, calling an API, running code, reading a file, and writing output — all without requiring the user to direct each step. Agents use tool calling to interact with external systems and maintain state across multiple reasoning steps.

An agentic workflow is an AI pipeline where the LLM dynamically decides the sequence of steps — which tools to call, in what order, and when to stop — based on intermediate results. This contrasts with a fixed pipeline where steps are predefined. LangGraph is the primary framework for building structured agentic workflows with explicit state management and conditional routing.

Tool calling allows an LLM to invoke external functions you define — a database query, API call, calculator, code executor, or file reader. You register tools with their names, descriptions, and parameter schemas. When the LLM decides a tool should be called, it outputs a structured tool-call message. Your application executes the function and returns the result to the LLM, which continues reasoning. Tool calling is the mechanism that gives agents the ability to take actions in the world.

Agents have two kinds of memory. Short-term memory is the conversation so far and the working results within one session, kept in the context window. Long-term memory is stored outside the context window, often in a vector or SQL database that keeps summaries of past sessions, user preferences or knowledge built up over time. Managing memory is a core design question for production agents: what to keep, what to summarise and what to look up when needed.

In the planner/executor pattern, one LLM call (the planner) decomposes the user goal into a sequence of sub-tasks; separate LLM calls (executors) or tools then carry out each sub-task. The planner also handles replanning when an executor fails or returns unexpected results. This pattern improves reliability for complex multi-step tasks by separating high-level reasoning from low-level execution.

A multi-agent system uses multiple specialised AI agents that collaborate — one agent might search the web, another analyses retrieved data, a third writes a report, and an orchestrator coordinates them. Multi-agent systems are more capable than single agents for complex tasks but introduce coordination overhead, communication costs, and failure cascade risks. LangGraph and CrewAI are the main frameworks for building multi-agent systems in Python.

LangGraph is a framework for building agent workflows as graphs that keep track of state. LangChain’s original agent executor ran one general loop; LangGraph gives you clear control over the shape of the state (with typed nodes), decisions about which step comes next, loops for retrying or reflecting, and checkpoints where a person can step in. It is the preferred choice for production agents that need predictable behaviour, state you can debug and complex workflows with branches.

CrewAI is a framework for building role-based multi-agent systems. You define agents with specific roles (researcher, writer, analyst), assign tasks to agents, and define a process (sequential or hierarchical). CrewAI is higher-level than LangGraph — easier to get started with multi-agent systems, but less granular control over state and routing. Good for content creation pipelines, research summarisation, and agentic tasks that map naturally to human team structures.

Human-in-the-loop is a design pattern where the agent pauses at defined checkpoints to request human approval, clarification, or correction before proceeding. LangGraph supports this natively via breakpoints and persistence. HITL is important for high-stakes or irreversible agent actions (sending emails, modifying databases, making API calls with real-world effects). In production, HITL is often part of the QA and safety layer.

Guardrails are limits that stop an AI agent from doing something harmful, unintended or against policy. They can include: checking inputs (turning away out-of-scope questions); checking outputs (reviewing answers before users see them); limiting which tools agents can use; limiting requests; and monitoring with alerts when behaviour looks unusual. NeMo Guardrails and Guardrails AI are Python libraries for adding guardrails in a structured way.

Agent evaluation is harder than static LLM evaluation because agents take multiple steps. Approaches: trajectory evaluation (did the agent take the right sequence of steps?); final output evaluation (did the agent produce the correct result?); tool use accuracy (did it call the right tools with the right parameters?); latency and cost tracking (how many LLM calls and tokens per task?); and failure mode analysis (what percentage of tasks fail and why?). LangSmith and custom logging are the primary tools for agent tracing and evaluation.

MCP is an open standard protocol (introduced by Anthropic) that defines a universal way for AI models to connect to external data sources and tools. Instead of writing custom integration code for every data source, MCP-compatible AI assistants and agents can connect to any MCP server using a standard protocol. This dramatically reduces integration overhead and enables a reusable ecosystem of AI tool connectors.

An MCP server is a lightweight process that exposes data sources, tools, or capabilities to MCP-compatible AI clients via the MCP protocol. An MCP server might expose: your company's CRM data, a file system, a database query interface, a web search tool, or a code execution environment. You can build MCP servers in Python or TypeScript using the official MCP SDK. The server handles capability exposure; the AI client decides how to use those capabilities.

An MCP client is an AI application, assistant, or agent that connects to MCP servers to access their capabilities. Claude Desktop is an MCP client; so are Claude Code, Cursor, and any LangChain/LangGraph agent built with MCP integration. MCP clients discover available tools from connected servers at runtime, allowing the AI model to decide which tools to invoke.

A traditional API integration requires custom code for each service: a custom request builder, response parser, authentication handler, and error handler — maintained per-service. MCP provides a standard protocol so that any MCP-compatible AI client can connect to any MCP server without custom integration code. The analogy is USB: before USB, every device needed a custom port; MCP creates a standard connector for AI tools.

MCP is a key enabler for enterprise AI agents. An agent built with MCP integration can dynamically discover and use tools from any connected MCP server — a company database, an internal API, a document management system — without each integration requiring custom code. This makes MCP-connected agents significantly more versatile and easier to extend than agents with hardcoded tool integrations.

Security points for MCP include: signing in to the server (only approved clients can connect); limiting each server’s tools (not every agent should get every tool); checking inputs to the server to block injection attacks; logging every MCP tool call; limiting requests to stop abuse; and network security (in production, run MCP servers on internal networks, not open to the public internet).

LangChain is a Python (and JavaScript) framework for building applications on top of LLMs. It provides abstractions and integrations for: prompt templates, LLM wrappers, document loaders, text splitters, embedding models, vector stores, retrievers, chains, agents, and output parsers. Its main strength is accelerating RAG prototype development through its extensive integration ecosystem — connecting to over 100 LLM providers, vector databases, and data sources.

A chain in LangChain is a sequence of components connected together — for example, a prompt template → LLM → output parser. LangChain Expression Language (LCEL) uses a pipe operator syntax (prompt | llm | parser) to compose chains declaratively. Chains can be simple (prompt to LLM) or complex (retriever → prompt → LLM → parser → router → another chain).

A retriever is a LangChain interface that takes a query string and returns relevant document chunks. It wraps vector store similarity search but can also implement custom logic — multi-query retrieval (generating multiple paraphrases of the query), contextual compression (trimming irrelevant parts of retrieved chunks), or ensemble retrieval (combining vector search and BM25). Retrievers are the core component connecting vector stores to RAG chains.

LangChain tools are Python functions decorated or wrapped as tools that an agent can choose to invoke. Each tool has a name, description, and input schema. When a LangChain agent decides to use a tool, it generates the tool name and arguments as a structured output; LangChain invokes the function and passes the result back to the agent for the next reasoning step. Tool design — writing clear names and descriptions — directly impacts whether the agent chooses tools correctly.

A LangChain agent is an LLM given a set of tools and a goal. It follows a reasoning loop: the LLM reasons about the goal → decides whether to call a tool → LangChain executes the tool → observation is added to context → LLM reasons again. LangChain supports several agent types: ReAct, OpenAI Functions, and OpenAI Tools agents. For production stateful agents, LangGraph is preferred over the built-in agent executor.

LangChain provides components for building LLM applications and RAG systems. LangGraph is built on top of LangChain and provides a graph-based framework for stateful, cyclical agent workflows. LangChain agents run a generic loop with limited state control; LangGraph lets you define explicit nodes, edges, conditional routing, state schemas, and human-in-the-loop checkpoints. Use LangChain for straightforward RAG pipelines; use LangGraph for complex, stateful, or multi-agent systems.

Common complaints about LangChain in production: its layers make it harder to debug when something goes wrong inside a chain; updates often broke existing code, which made maintenance painful; it encourages quick prototypes, which can turn into code that is hard to maintain in bigger systems; and for very fast systems, direct API calls can replace its extra layers. LangChain is excellent for prototypes; production teams often move to more direct code over time.

LangSmith is a LangChain-developed platform for tracing, evaluating, and monitoring LangChain applications. It records every step of a chain or agent run — inputs, outputs, latency, token counts — and makes them browsable and searchable. It supports creating test datasets, running evaluations, and comparing runs. LangSmith is the primary tool for debugging LangChain applications and setting up RAG evaluation pipelines in the LangChain ecosystem.

Build a FastAPI app with your LLM or RAG logic → write a Dockerfile that installs dependencies and starts uvicorn → build the image → push to a container registry (ECR, GCR, Docker Hub) → deploy to cloud (AWS ECS, Google Cloud Run, Azure Container Apps). Environment variables (API keys, database URLs) are injected via secrets management (AWS Secrets Manager, GCP Secret Manager). Health check endpoint (/health) is required for orchestration platforms to manage the container lifecycle.

Ways to make answers faster: stream the answer (send words as they are written instead of waiting for the end); cache answers (store repeated questions in Redis and return the saved answer); choose the model by task (smaller, faster models for simple questions); make LLM calls in parallel when an answer has several parts; handle requests asynchronously; and tune retrieval (fewer chunks, and chunk sizes that keep the prompt short). Measure the 50th, 95th and 99th percentile response times for each endpoint.

Use structured JSON logging with a consistent schema per log entry: request_id, timestamp, user_id (hashed), model, input_tokens, output_tokens, latency_ms, retrieved_chunks (IDs only, not full content), and response_status. Send logs to a centralised log aggregation system (CloudWatch, Datadog, ELK Stack). Never log raw user inputs or full LLM responses if they may contain PII — apply masking or only log hashed/truncated versions.

LLMOps is the practice of running LLM apps in production. It is like MLOps for traditional ML, but adapted to the problems LLMs bring. It includes: versioning and managing prompts; pipelines that test outputs; watching speed and cost; managing model versions (including when a provider retires a model); A/B testing prompts and retrieval methods; and responding when the quality of answers drops.

Monitor four levels: infrastructure (CPU, memory and container health, with standard cloud monitoring); the API (requests, errors and response time for each endpoint, with Datadog or CloudWatch); the LLM (tokens per request, model errors and cost per question, from provider dashboards and logs); and quality (automatic RAGAS tests on a sample, people reviewing flagged answers and feedback from users). Set alerts for: an error rate above 1%, a 95th-percentile response time above your target, unusual token costs and any drop in quality measures.

Keeping costs down: choose the smallest model that is good enough for each type of question; cache frequent questions in Redis; keep the prompt short by dropping retrieved chunks that are less relevant; use streaming so you can stop early when possible; log the token count of every request; set hard token limits on API calls; set cost budgets for each customer or organisation; and check the cost-per-question dashboard every week for anything unusual.

AI applications use environment variables to store API keys, database connection strings, and configuration — never hardcoding secrets in source code. Locally, use .env files loaded by python-dotenv (never committed to git — add to .gitignore). In production, use a secrets management service (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault) to inject secrets at runtime. In Docker/Kubernetes, secrets are mounted as environment variables or files, not baked into the image.

Production AI apps should: retry, waiting longer each time, when they hit temporary rate limits and timeouts; switch to another model or provider if the main one is down (for example, from GPT-4o to Claude Sonnet); use a circuit breaker so they stop hammering a failing service; fail gently, returning a cached or default answer instead of an error page; and alert when error rates pass a limit. Putting your LLM calls behind one layer of your own makes it easier to switch providers.

Structure your answer as: (1) Problem — what was the use case and why RAG over a simple prompt? (2) Architecture — document loading, chunking strategy and size, embedding model choice, vector database and why, retrieval configuration; (3) LLM — model choice and system prompt design; (4) Evaluation — what metrics you used and what scores you achieved; (5) Deployment — where and how you deployed it; (6) Results — what worked and what you would improve. Avoid describing a tutorial copy — explain the decisions you made and why.

Describe the evaluation approach you used: whether you set up RAGAS with a labelled test dataset (questions + ground truth answers + relevant chunk IDs), or used manual spot-checking, or an LLM-as-judge setup. Mention which RAGAS metrics you tracked (faithfulness, context precision, context recall) and what your scores were. Interviewers want to see that you measured whether your system worked — not just that you built it.

Describe your mitigation strategy: RAG grounding with citations; low temperature setting; a system prompt instructing the model to say "I don't know" when context is insufficient; output validation checking that the answer is supported by retrieved content; human review sampling. If you measured hallucination rate before and after a mitigation, that is a strong answer.

Describe the full deployment path: FastAPI endpoint wrapping the RAG or agent logic → Dockerfile → built and pushed to a container registry → deployed to a cloud service (Cloud Run, ECS, Railway, etc.) → environment variables managed securely → health check endpoint → basic monitoring in place. If you deployed to a personal server or just ran it locally, be honest — but explain what production deployment would look like for the same system.

Describe how you tracked token usage: logging input and output token counts per request, estimating cost per query from the LLM provider pricing, and identifying the most expensive query patterns. If you implemented caching for repeated queries, mention that. If you chose a cheaper model for low-complexity queries, explain that decision. Cost awareness is a strong signal even in a small side project.

This is your chance to show you think about production. Good answers include: a better way of splitting documents, with tuned overlap; adding hybrid search for more precise results; adding a reranker; setting up full RAGAS testing instead of spot checks; streaming answers; tighter access control; and a feedback loop where users’ corrections are added to the test set. Pick the improvements that fix the real weaknesses of what you built.

Engineers make trade-offs. Strong answers acknowledge them: "I used Chroma locally because I didn't want to manage Pinecone infrastructure for a prototype — in production I would use Pinecone for reliability"; "I used a fixed chunk size of 512 tokens which was a reasonable default but I didn't run a chunk size ablation study"; "I chose GPT-4o-mini for cost reasons — the retrieval quality compensates for the smaller model in most queries but edge cases exist".

Be specific and confident. Name the LLM provider (OpenAI, Anthropic or Gemini), the embedding model, the vector database, the framework (LangChain or LangGraph), the API framework (FastAPI), the cloud platform and the monitoring tool. Give one sentence of reasoning for each: not just “I used LangChain because it is popular”, but “I used LangChain because it connects directly to Chroma, which sped up the prototype, and I can swap the vector store later if I need to”.

Describe a real debugging story: poor retrieval because chunks were too large, so their embeddings were too vague; a LangChain agent calling the wrong tool because the tool’s description was unclear; a FastAPI endpoint returning 422 errors because the Pydantic model did not match the LLM’s structured output; or a RAGAS faithfulness score of 0.4 that showed the LLM was ignoring the retrieved context. How you debug AI systems is one of the clearest signs of experience.

This is a direct prompt to demonstrate non-tutorial thinking. Strong differentiators: you evaluated the project (RAGAS or equivalent) and iterated on chunking or retrieval based on results; you deployed it to a real environment; you added features like citations, hybrid search, or access control; you handled edge cases the tutorial ignores; you benchmarked chunking strategies; or you made conscious trade-off decisions and documented them. The answer must be specific — vague claims of "it's more advanced" are unconvincing.

Common AI engineering interview questions cover: Python and API basics (calling an LLM API, reading JSON, handling errors); LLM ideas (what a token is, temperature, structured output); how a RAG pipeline is built (chunking, embeddings, vector search, retrieval and generation); vector databases (meaning-based search, similarity measures and how tools compare); AI agents and tool calling; LangChain, LangGraph and CrewAI; what MCP is; deploying with FastAPI and Docker; testing with RAGAS; and walking through your projects, explaining what you built, how you tested it and what trade-offs you made.

Prepare in five areas: (1) core ideas: LLMs, tokens, embeddings, RAG, vector databases, agents, tool calling and MCP; (2) frameworks: LangChain, LangGraph, CrewAI or similar; (3) deployment: FastAPI, Docker, cloud basics, logging and testing; (4) projects: at least one deployed RAG or agent project you can explain from start to finish; and (5) system design: being able to design a company knowledge chatbot or a document question-and-answer system with a sound design, retrieval, testing and monitoring. Do not stop at knowing how to prompt ChatGPT; interviewers want to see that you can build apps.

Yes — RAG questions are among the most common in AI engineering interviews, especially for roles at companies building knowledge assistants, document search tools, or AI chatbots. Expect questions on the full pipeline: document loading, chunking strategy, embedding model selection, vector database choice, retrieval design, reranking, prompt template construction, and RAGAS evaluation. You should be able to draw and explain the full RAG architecture and describe the decisions you made in a real project.

AI engineers need a working grasp of machine-learning ideas, such as what training, fine-tuning and embeddings are. Deep ML theory (the maths of backpropagation, optimisation methods and model designs) is not usually tested unless the role is about building models or ML engineering. The focus is on building AI apps in practice: RAG pipelines, deploying LLM APIs, designing agent workflows, testing outputs and shipping to production. ML theory is a bonus; being able to build is what is required.

The projects that show the most are: (1) a deployed RAG knowledge assistant, with a chunking approach, a vector database, a retrieval pipeline, a FastAPI endpoint and RAGAS testing; (2) a LangGraph workflow with several agents, with typed state, tool calling and decisions about which step comes next; (3) an LLM API with structured outputs, deployed to the cloud with logging and error handling; and (4) an assistant connected through MCP, where even a simple MCP server offering two tools stands out. For any project, be ready to explain your design choices, what you tested, what failed and what you would improve.

To prepare for LLM interview questions: understand what tokens are and how context windows work; know the difference between temperature, top-p and max tokens; understand system prompts and prompt templates; learn structured outputs and function or tool calling; understand how to test LLM outputs (RAGAS, human review, using an LLM as judge); know the cost and speed trade-offs between model sizes and providers; and be able to explain why a model might make things up and how RAG reduces that. Read at least one LLM provider’s API documentation (OpenAI or Anthropic) in depth.

Common AI system design questions include: “Design a chatbot that answers from company policies”, “Design a customer support assistant that works at scale”, “Design a research system with several agents”, “Design a tool that answers questions about PDFs”, “Design an AI system with access control for company documents” and “Design a production LLM API with monitoring and a fallback”. For each, build your answer around: the problem and its scope, how data flows and the overall design, how retrieval works, which model you chose and why, how you will test it, how it is deployed, security and access control, and monitoring and alerts.

AI engineering roles usually expect you to know LangChain. Interviewers may ask about chains, prompt templates, retrievers, connecting tools, LCEL (LangChain Expression Language), LangGraph for agents that keep state, and when LangChain’s extra layers get in the way rather than help. Know its strengths (quick RAG prototypes and many ready-made connections) and its limits (extra layers and harder debugging in complex pipelines). LangGraph matters more and more for roles with several agents or workflows that keep state.

Yes. AI agents are quickly becoming a core interview topic, as companies move from fixed LLM apps to agent-based systems. Understand what makes a system an agent (the model decides which tools to use and in what order), how tool calling works, what memory and state mean for agents, how LangGraph differs from LangChain agents, how several agents work together and how to keep a person in the loop. Even one hands-on LangGraph agent project makes you stand out: most candidates can talk about agents but cannot show one.

Freshers with a technical background (computer science, engineering, Python or data science) should: (1) learn the basics of Python APIs: calling LLM APIs, handling JSON and building FastAPI endpoints; (2) understand LLMs, tokens, embeddings and RAG, both in theory and hands-on; (3) build one complete RAG project from start to finish with LangChain, Chroma and FastAPI; (4) put it on GitHub with a README that explains the design, test results and how to run it; and (5) be ready to explain every decision in the project. The Technovids AI Engineering Course covers all of this with live trainer guidance and two shipped projects.

Software developers moving into AI engineering start with big advantages: their skills in connecting APIs, writing good code, deployment and debugging carry straight over. Focus your learning on: LLM APIs, prompt engineering and structured outputs; building and deploying a RAG pipeline; LangChain and LangGraph; deploying a FastAPI endpoint with logging and monitoring; and testing outputs with RAGAS. Projects and GitHub matter most: a deployed RAG assistant or LangGraph agent shows you are ready for AI engineering far better than certificates alone.

Start with the AI Engineering Guide for the full landscape, then the AI Engineering Roadmap for a sequenced learning path. To understand the specific skills interviewers test, see the AI Engineer Skills guide. To build a portfolio you can confidently explain, see the AI Engineer Projects guide. For live structured training across the full interview-relevant stack — LLMs, RAG, agents, LangChain, MCP, and deployment — with five guided production projects and instructor feedback, see the AI Engineering Course.