Skip to content
Pratim/Dasude
02/Agentic AI3 min read

Wardrobe Recommendation Engine

Outcome

Outfits are built from retrieved wardrobe items and validated in code before anything is shown.

Agentic AIRAGMultimodalFashion

GPT-4.1-mini, CLIP, pgvector, BM25, Python

View on GitHub
A grid of twenty garment photos from the wardrobe dataset: shirts, jackets, trousers, shorts and topswardrobe
A sample of the wardrobe items the engine retrieves from, taken from the project repository.

Draft write-up

A personal stylist you talk to in plain English. Describe an occasion, point at a garment, or share an outside photo, and the system returns complete, wearable outfits assembled only from items in the wardrobe. The language model is never allowed to invent clothing: the engine retrieves real items first, asks the model to compose outfits strictly from those, and validates every outfit in code before it is shown.

Source on GitHub ↗

01Problem

A language model asked to style an outfit will happily recommend clothes you do not own. For a wardrobe assistant that makes every answer useless. The task needed retrieval over the real closet, outfit rules the model cannot break, and memory so follow-ups like "smart clean look" attach to the earlier request.

02Approach

  1. 01

    Understand the request

    A language model turns the raw message into structured intent: occasion, vibe, item id, image path and whether a follow-up question is needed. A rule-based layer repairs edge cases and stops stale conversation context from leaking into a fresh request.

  2. 02

    Remember the session

    Per-session memory merges new information with earlier turns, so a short follow-up attaches to the right occasion.

  3. 03

    Retrieve with hybrid search

    For the chosen garment, compatible candidates are pulled per role (top, bottom, outerwear) by combining semantic embeddings in pgvector, a metadata bonus for color and style, and BM25 keyword scoring.

  4. 04

    Compose, then validate in code

    The model assembles three outfits (minimal, layered, alternative) strictly from the retrieved item identifiers. Incomplete or rule-breaking outfits are removed in code instead of being trusted to the model, and two more calls write the intro and a final stylist note.

03Architecture

A retrieval-first pipeline: the model parses and composes, but the closet, the rules and the final check all live outside the model.

01MessageTEXT, ITEM ID OR OUTSIDE PHOTO02Query understandingLLM + RULE REPAIR03Chat memorySESSION CONTEXT04Hybrid retrievalPGVECTOR + METADATA + BM2505Outfit composerGPT-4.1-MINI06ValidationCODE

  1. 01Message

    Text, item id or outside photo

    Three ways in: describe an occasion, style a specific garment, or pair an outside photo with the wardrobe.

  2. 02Query understanding

    LLM + rule repair

    Structured intent, with rules that fix edge cases and block stale context.

  3. 03Chat memory

    Session context

    Occasion, vibe, item id, image and history carried across turns.

  4. 04Hybrid retrieval

    pgvector + metadata + BM25

    Cosine similarity on embeddings, a color and style bonus, then exact keyword reranking.

  5. 05Outfit composer

    GPT-4.1-mini

    Three outfits built only from retrieved identifiers, following hard pairing rules.

  6. 06Validation

    Code

    Outfits that are incomplete or break the rules are dropped before display.

  • Three full outfits per request, each with a reason, styling notes and a confidence level
  • Style around one specific garment, with alternatives from similar items already owned
  • Outside photos are captioned and embedded with CLIP, then treated as a temporary garment
  • "Find similar items" through CLIP image similarity

04Challenges

  • The model could recommend clothes that are not in the wardrobe.

    FixRetrieval comes first, composition uses only retrieved identifiers, and every outfit is validated in code.

  • Old conversation context leaked into new requests.

    FixPer-session memory merges follow-ups deliberately, and a rule-based layer resets context for a fresh request.

05Learnings

Outcomes

  • A working RAG pipeline for fashion with grounded, validated outfits
  • Every recommendation carries a reason, styling notes and a confidence level
  • A written production and scaling roadmap in the repository

What I learned

  • Do the checking in code. A prompt asking the model to behave is not a guarantee.
  • Hybrid retrieval keeps exact attributes that embeddings alone blur.

06Next

  • 01Follow the production and scaling roadmap in the repository
  • 02Add an offline evaluation set of requests with expected outfits