AI · Retrieval Systems · BIM
BIMtrieval
BIMtrieval helps architecture teams understand complex BIM information through natural-language questions, hybrid SQL / RAG / graph retrieval, grounded answers, and synchronized visual evidence.
Overview
A BIM (Building Information Model) is a digital building, not just a 3D mesh. An IFC file
stores typed objects — walls, doors, spaces, storeys — each with properties,
quantities, materials, and relationships to other objects, spread across a graph of references
that can run to hundreds of thousands of records, and two authoring tools can export the same
real-world idea in completely different structures.
In an architectural office that one model is read by people with very different
responsibilities and technical depth, who do not share the same understanding of what it
contains or how reliable each element is. Practitioners describe the familiar version: a
visually detailed model that one person reads as complete enough for its intended use and
another reads as missing the data that use requires — a disagreement that surfaces as a
scope and schedule argument rather than an information one.
Five roles, one model · swipe to see the full map on a narrow screen
Each arrow is a translation step, and the Job Captain and BIM Coordinator carry most of that
burden: they are the people who have to restate what is in the model in terms of design, scope,
cost, and schedule. The more an answer depends on inspecting the model directly, the more it
depends on whoever happens to know how to interrogate it.
Research question
Can natural-language access to a building model reduce that translation burden, by letting
each role ask the same model a direct question and receive an answer that is grounded,
bounded, and visible in place?
BIMtrieval is this project’s answer: a backend-centred BIM information and explanation
system. Someone should be able to ask “which floor has the most doors?” without
first learning the IFC schema, and should see the objects behind that answer highlighted in the
model. The retrieval and validation pipeline is the substance of the project; the frontend
exists so a person can inspect what it decided. No single technique handles a BIM question well
alone, so the work is divided:
- SQL is authoritative for exact filters, counts, grouping, and aggregation.
- RAG (retrieval-augmented generation) carries semantic intent and terminology, and finds related evidence when the wording of a question and the model’s own vocabulary do not match.
- Graph search follows IFC relationships, containment, filling, voiding, aggregation, material association, and spatial membership.
- LLMs interpret the question and write the final explanation, but deterministic code stays responsible for validation and execution.
A second goal shapes that same division of labour: minimizing LLM token usage. Every
step deterministic code can own is a step the model never has to read, so the raw IFC file, the
full object tables, and the untrimmed retrieval results stay out of the prompt entirely. What
this project proposes is therefore a lightweight BIM searching model — one that
answers from a bounded, adjudicated packet on a small model roster, rather than one that
scales by feeding a larger model more of the building.
Guiding principle
The LLM decides what the question means; deterministic code decides what the data proves.
The LLM never queries the raw IFC file, never writes the SQL that runs in production, and
similarity is never treated as proof. This is a research prototype: the intended benefit is
clearer communication about a model, not a measured productivity result.
Pipeline Evolution
The backend went through three architectures, and the difference between them is really one
question: how much evidence should reach the final LLM, and who decides what is relevant?
The same 42-question benchmark was carried through every iteration.
v1Broad retrieval, late narrowing
An LLM chose the retrieval route, the backend gathered broad evidence groups, and the final
LLM received group summaries plus up to fifty sample rows and had to work out which of them
answered the question.
Weakness: too much responsibility landed on the final LLM. Constraints
could be lost in the noise, or a broad group relabelled as the requested concept.
v2Strict early planning
Planning moved to the front. A deterministic slate proposed a small bounded set of subject,
property, value, and location candidates; the binder could select only from that slate, each
typed answer part was executed once, and the final LLM wrote from a compact packet.
Weakness: the overload was gone, but so was information. A bounded
slate could omit a valid concept entirely, leaving the final LLM with no way to answer
flexibly — only with what the slate had already allowed.
final v3Complete semantics, bounded evidence
Ingestion writes one complete, validated semantic manifest per model, and the binder sees all
of it — nothing is pre-filtered away from interpretation. What is bounded instead is the
evidence: a typed constraint ledger records every requirement in the question, a
deterministic gate proves the binding covers them, and only adjudicated results reach the
final LLM. Enough context for a flexible answer; few enough facts to stay checkable.
Cost of the middle ground: the complete manifest is a large prompt.
It is sent as a cacheable prefix and paired with a small binder model, which is how the
accuracy machinery survived the move to a cost-reduced roster.
Later experimental directions were built and then discarded; the project kept an updated v3.
The Final v3 Pipeline
The chart below is the explanation; these notes only frame it. Ingestion runs once per
model: a SHA-256 fingerprint makes it idempotent, IfcOpenShell parses objects, properties,
quantities, materials, and relationships into PostgreSQL, a validated semantic manifest
describes what is queryable, deterministic templates become RAG documents embedded into
pgvector, and the geometry is converted once into viewer fragments — with a model
accepting questions only when all four artifacts match its identity. Ingestion also preserves
typed IFC measurements and their source units, so supported numeric filters and aggregations
can be executed deterministically.
Per question, the backend runs as deterministic checkpoints. Interpretation turns the
question and its inherited context into a typed constraint ledger; the binder maps it onto the
complete manifest as typed answer parts, without writing SQL. A validation gate proves the
binding covers the ledger before anything executes, allowing at most one focused correction.
Retrieval and execution then run each part once by the method that owns it, and the results are
adjudicated as
exact, zero, partial,
unavailable, or ambiguous. Only that compact packet reaches the answer
writer, whose prose is checked back against it before the viewer presents the same executed
predicate as a highlight.
Zero is a result, not a default
A zero is reported as real only when the target, the filters, and the coverage were actually
executed and proven; a failed semantic search, an unavailable field, or a scope the model does
not contain returns an honest unavailable or a clarification.
Full information flow
One connected graph across three regions, carrying data and control from the original IFC file
through to the grounded answer and the synchronized 3D viewer.
SQL, RAG, and graph paths are labeled, not colour-only.
LLM usage
A successful query uses two generative calls plus a local embedding model, each with a narrow,
bounded role. Every model name is configuration-driven.
| Role | Model | Reasoning | Responsibility & hard limits |
|---|---|---|---|
| Call 1, semantic binder | gpt-5.4-nano |
medium | Maps the question and its ledger requirements to semantic IDs and typed answer parts. It cannot write raw SQL, invent access paths, or decide that similarity proves a fact. |
| Optional corrective call | gpt-5-mini |
high | Used only for a proven, recoverable binding gap, and receives the failed ledger items and focused candidates rather than a duplicated universe. At most one correction. |
| Call 2, grounded answer writer | gpt-5.4-mini |
low | Turns the compact adjudicated packet into readable prose. A deterministic validator rejects unsupported counts, claims, or citations and can replace the text with a factual fallback. |
| Embedding model (not a generative call) | BAAI/bge-m3 |
— | A local model producing normalized 1,024-dimensional vectors for stored RAG documents and query text, supporting semantic and multilingual recall. Similarity is a candidate signal, never proof. |
Call budget
Normal successful active-model query: 2 LLM calls.
Proven recoverable binding gap: maximum 3 total calls.
Evaluation
The benchmark is 42 human-assessed questions carried unchanged through every iteration: exact
counts, floor scoping, compound constraints, qualitative questions, deliberately unsupported
questions, conversation follow-ups, prompt injection, malformed input, and a non-English query.
Verdicts were assigned against database ground truth; a clarification passes only when ambiguity
genuinely prevents a reliable answer, and a confident but wrong answer always fails.
PASS
PARTIAL
FAIL
Raw verdicts, PASS / PARTIAL / FAIL out of 42: v1 14 / 7 / 21, v2 21 / 3 / 18,
final v3 30 / 3 / 9. The v3 figures are the latest run, captured live on 2026-07-27 with
the cost-reduced roster. Only recorded, comparable verdicts are compared here — the three
runs did not record cost, latency, and token figures on the same basis.
The shape of the improvement matters more than the number. v1 and v2 sit on opposite sides of
one tradeoff: v1 gave the final LLM too much and it selected badly; v2 gave it too little and it
could not answer questions the data would have supported. v3’s remaining failures are
informative rather than random — a binding correction that did not recover, a wall count
returned where a thermal property was asked for, and 45 storey entities reported as 45 floors.
The Application
The frontend is the delivery surface, not the source of truth: a React / TypeScript / Vite
application using That Open Components and Fragments with Three.js, talking only to the backend
HTTP API. The browser never connects to PostgreSQL or to OpenAI, and one prepared model is
active at a time.
3D viewer and query-result highlighting
Query-result geometry is highlighted while the rest is dimmed rather than hidden, so the answer
keeps its spatial context, and a truncated highlight set is disclosed rather than implied away.
A compact Fine / Standard / Fast control changes only the visibility and edge-detail
thresholds, so a large model stays navigable without being reloaded or reconverted.
Chat panel
A floating conversational panel carries the question, the concise prose answer, the exact total,
and a compact class summary. The exact total, the viewer highlight count, and the bounded
evidence list stay three independent numbers. Evidence disclosure is collapsed by default, and
clicking an entity citation selects that object in the viewer without asking the model
anything.
Synchronized Query Explanation panel
When a completed result is worth inspecting, a Query Explanation card opens above the chat and
stays synchronized with what is highlighted in the viewer. The backend chooses the presentation
deterministically from the accepted operation and its already-computed result — a bounded
table, a horizontal bar chart, or a grouped relationship diagram — so the frontend never
infers a visualization from prose, and a scalar answer opens no panel at all. Selecting a group
re-highlights only that subgroup, with an All results action to restore the full answer;
neither issues a query or an LLM call.
Component-detail panel
Clicking an object, or a citation in an answer, opens a component card with an isolated preview
of that instance and a bounded, read-only detail list — class, name, type, storey,
materials, allowlisted properties — with absent fields omitted rather than shown empty.
Same type and Same family highlight every exactly matching object; when the source
IFC supplies no type data those actions are disabled with the concrete reason instead of
guessing a grouping from names.
IfcCovering instance: the bounded detail list is all the source file supplies,
so Same type and Same family are disabled with that reason stated.
Floor-plan mode
The viewer can switch from perspective 3D to a top-down orthographic plan of one logical floor,
cut about 1.2 m above it and bounded below so lower floors do not show through openings. The
floor buttons come from the same elevation-band clustering that resolves “the second
floor” in a question, so the control and the language can never disagree — which is
why a 45-storey file offers 9 floor buttons. A 3D button restores the exact camera view
that was active before, and model, selection, highlights, and conversation are all preserved.
Outcome and Limitations
The project shows that a large, messy, inconsistently structured BIM model can be opened to
plain-language questions. Roughly seven of ten benchmark questions return a correct grounded
answer.
- The intended benefit is communication: someone who cannot interrogate the model directly can ask it a question and see the objects behind the answer. That benefit is argued from the practitioner accounts above, not measured — no user study was run.
- Answer quality is bounded by the source file. When a model does not carry the data a question needs, the honest response is unavailable, not an estimate.
- Roughly half the remaining failures are visible — a binding gap the single correction could not repair, ending as a clarification. The other half are confident but wrong, and those are the ones that matter.
- It is a local, single-user prototype: no IFC upload, authentication, or model editing.
Technical Stack
| Source format | IFC2X3 |
| IFC parsing | Python 3.11, IfcOpenShell |
| Structured storage | PostgreSQL, SQLAlchemy, psycopg2 |
| Vector retrieval | pgvector with cosine / HNSW search |
| Embeddings | BAAI/bge-m3, Sentence Transformers, PyTorch / CUDA, 1,024 dimensions |
| Query backend | FastAPI, Pydantic, parameterized read-only SQL |
| Generative models | gpt-5.4-nano binder, gpt-5-mini correction, gpt-5.4-mini grounded answer |
| Query methods | typed SQL, SQL-scoped RAG, bounded IFC graph traversal |
| Frontend | React 18, TypeScript, Vite 6, Zustand |
| 3D BIM viewer | That Open Components / Fragments 3.4.6, Three.js 0.185.1, web-ifc 0.0.77 |
| Explanation visuals | plain React, CSS, and SVG — no charting dependency |
| Backend tests | pytest, ruff |
| Frontend tests | Vitest, Playwright, ESLint |
| Portfolio diagrams | D3.js v7 |
References
-
OpenIFC Model Repository
openifcmodel.cs.auckland.ac.nz -
BIMData Research and Development, IFC files
github.com/bimdata/BIMData-Research-and-Development · curated IFC file list -
IfcOpenShell
docs.ifcopenshell.org · github.com/IfcOpenShell/IfcOpenShell -
pgvector
github.com/pgvector/pgvector -
BGE-M3
arXiv:2402.03216 · huggingface.co/BAAI/bge-m3 -
That Open Engine
engine_components · engine_fragment -
R2RML and Ontop
W3C R2RML · ontop-vkg.org/guide
BIMtrieval adopts the explicit semantic-to-relational mapping principle, not an RDF / SPARQL stack. -
IRNet / SemQL
Guo et al., “Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation,” ACL 2019. aclanthology.org/P19-1444
Relates to schema linking, a typed intermediate representation, and deterministic SQL generation. -
DIN-SQL
Pourreza and Rafiei, “DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction,” NeurIPS 2023. proceedings.neurips.cc
Relates to stage separation and targeted correction. -
BIRD
Li et al., “Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs,” NeurIPS 2023. proceedings.neurips.cc
Relates to database-value understanding and real / dirty data. -
KG-supplemented RAG
Zhu et al., “Knowledge Graph-Guided Retrieval Augmented Generation,” NAACL 2025. aclanthology.org/2025.naacl-long.449 · GraphRAG local search
Relates to semantic seeds followed by graph expansion and associated text.
These papers and projects informed the architecture. BIMtrieval uses its own IFC relationship
graph, PostgreSQL schema, and RAG documents; it imports no RDF stack, no second graph database,
and no external GraphRAG indexing stack.