AI Without the Hype: A Working Engineer's Guide

Most AI content online falls into two camps: doom about superintelligence, or breathless claims that a chatbot will replace your job by Tuesday. Neither helps you ship anything. This guide skips both and walks through how to actually use the current generation of AI tools in real projects, with the limits made explicit.

What an LLM actually does

A large language model like GPT-4 or Llama 3 predicts the next token in a sequence, trained on huge amounts of text. That's it. There's no internal fact-checker, no persistent memory between sessions (unless you build one), and no understanding in the way a person understands. When you ask it a question, it's generating a statistically plausible continuation of your prompt.

This matters practically: the model will confidently generate a Python function that calls a library method that doesn't exist, because that method sounds like something the library would have. Always run the code. Always check the API reference. Treat model output as a first draft from a fast, well-read intern who sometimes lies without knowing it.

A real workflow: using an LLM for code

Here's a pattern that works instead of just asking ChatGPT to "build me an app":

1. Write the function signature and docstring yourself, specifying types and edge cases.
2. Ask the model to implement it against that exact spec.
3. Write your own tests separately — don't ask the model to write tests for code it just wrote, it will tend to write tests that pass trivially.
4. Run the tests. Feed failures back in as new prompts, not vague "it doesn't work" statements.

def parse_duration(text: str) -> int:
    """
    Parse strings like '1h30m', '45s', '2d' into total seconds.
    Raise ValueError on invalid input.
    """

Giving the model this exact contract produces far better output than a loose description, and it gives you something concrete to test against.

Retrieval-augmented generation, in plain terms

RAG gets thrown around as a buzzword but the mechanism is simple: instead of relying on what the model memorized during training, you fetch relevant documents at query time and stuff them into the prompt.

A basic setup:

1. Chunk your documents (500-1000 tokens each is a common starting point).
2. Embed each chunk with a model like text-embedding-3-small or an open model like bge-small-en.
3. Store vectors in something like Postgres with pgvector, or a dedicated store like Qdrant.
4. At query time, embed the user's question, run a similarity search (cosine distance is standard), and pull the top-k chunks into the prompt alongside the question.

SELECT content FROM docs
ORDER BY embedding <=> '[0.012, -0.045, ...]'
LIMIT 5;

This is why a chatbot trained on data up to a cutoff date can still answer questions about your internal wiki from last week. It's not reasoning about your company — it's reading your documents and summarizing them.

Where classic ML still wins

Not every problem needs a transformer. If you're predicting churn from structured tabular data — account age, usage frequency, support tickets — a gradient-boosted tree model like XGBoost or LightGBM will usually outperform an LLM-based approach, train in minutes instead of hours, and cost a fraction to run. Reach for scikit-learn's RandomForestClassifier or XGBoost before reaching for an API call, especially when your data fits in a spreadsheet and your target variable is a clean number or category.

Cost and latency are design constraints, not afterthoughts

A GPT-4-class call costs real money per token and takes real seconds to return. If you're building a feature that runs on every page load for every user, that adds up fast and the latency will be visible. Cache aggressively, use a smaller model like GPT-4o-mini or a local Llama 3 8B for anything that doesn't need top-tier reasoning, and reserve the expensive model calls for the parts of your pipeline where quality actually matters.

The failure mode nobody warns you about

Models hallucinate more, not less, when you ask them for things slightly outside their training distribution — obscure library versions, internal company jargon, recent CVE numbers. If the answer needs to be exactly right (a security advisory, a legal citation, a medical dose), don't trust generation alone. Verify against a primary source every time, and build that verification step into your pipeline rather than trusting it to human review after the fact.

AI tooling is genuinely useful once you stop expecting it to think and start treating it as a fast pattern-matcher you have to check. Build the verification step in from day one and you'll get real value out of it instead of a stream of confident nonsense.

If you want to go further, check out Korra Studio's Python and Data Science tracks for the fundamentals that make working with these tools actually productive.