Concepts
Model-Driven Data Engineering earns its own words. These are the coined concepts behind the method — one memorable name per idea, so a team can point at it, reuse it, and build on it. A shared language for business, engineers and AI.
The master brand, and the thesis the rest of the corpus elaborates: a data platform's structure and logic are data, not code. Entities, attributes, mappings, filters, time policies, rules — and even the generated SQL — live as rows in a metamodel, and the platform's artifacts (DDL, dbt, pipelines, tests, lineage, documentation, diagrams) are generated views of that metadata. Models are the source of truth; everything downstream is derived from them deterministically. The bet is easy to state and hard to hold: you write the model, not the pipeline. A five-table demo is enough to prove the trick — SQL generated from metadata instead of typed by hand. The full metamodel is 268 tables, and the honest question is why it grows that far. The answer is not sprawl but discipline: repeating groups become tables (an identifier has member columns, in order), declaration and call are separated the way a programming language separates them (a function has declared parameters; each use binds arguments), and capabilities arrive in coherent clusters (Data Vault, fact-oriented modeling, import bridges) you can adopt or ignore. The model is large the way a city is large — districts, not sprawl — and roughly two-thirds of it is optional. What makes the approach trustworthy is that the metadata is executable, not descriptive: it does not document what the platform does, it is what the platform does. Because SQL, tests, lineage and docs are all executions of the same metadata, they cannot drift out of sync — the metadata star is the image, deterministic generation is the mechanism, and the metadata operating system is the engineered whole. MDDE is not a tool; it is the way of thinking the tools implement. The framework even documents itself — its own reference, DDL and diagrams are generated from the metamodel database, because a metadata framework whose documentation is hand-maintained would be a contradiction.
Beneath every MDDE pattern sit a few load-bearing principles. First: the model, not the code, is the source of truth — if code and model disagree, the code is wrong. Second: derive, don't hand-write — anything that can be generated deterministically from the model should be, because generated artifacts cannot drift from their source. Third: make metadata executable, so the platform's description and its behavior are the same thing. Everything else in the concept library is these three principles applied to a specific layer. The foundations matter because they are where implementations quietly fail. Teams adopt a generator but keep hand-patching its output (violating principle one), or generate SQL but hand-write tests and docs (violating principle two) — and within months the platform is model-flavored rather than model-driven. Holding the foundations is what separates Model-Driven Data Engineering as a method from code generation as a tool. They are also where MDDE joins the wider thesis: structure beats magic. The foundations are the structure — and the master line follows from them: the magic isn't in the model, it's in the structure you give it.
The BTM pipeline is the method seen as one flow, from scattered reality to a living, queryable brain. Most of what an organization knows is already written down — but it’s trapped in PowerPoints, Confluence pages, SQL transformations, data models, table structures and reports, where no machine and few humans can reason over it. The pipeline is how that raw material becomes structure. The stages chain together. Capture: scrape existing information into metadata — markdown with frontmatter, folder-per-unit — and analyze the existing SQL and transformations into the same standard format. Mirror: load it as a mirror in a fast database like DuckDB. Enrich: hunt for gaps, inconsistencies and rules, then run AI-driven interviews with the stakeholders the documentation and code check-ins reveal, adding explanatory metadata. Serve: optimize and enrich, build spaced-repetition training so new people gain the skills, expose MCP servers so users and systems can ask the company brain questions — while everything is logged and git-versioned, so you see how the brain develops over time and keep the lineage back to every source. The differentiator is that the whole pipeline can be generated from instructions derived from existing solutions — a reusable, teachable skill set. That is the offering: not a one-off project, but a repeatable method for turning any organization’s buried knowledge into a company brain.
The company brain is what the BTM pipeline builds toward: the organization’s knowledge, structured well enough that you can ask it questions. Not a wiki you scroll and a warehouse you query separately, but one connected, versioned body of metadata — the models, the rules, the transformations, the decisions, the explanations — that both people and systems can reason over. The knowledge that used to live in a few experts’ heads and a thousand documents becomes an asset the whole organization can consult. Two properties make it a brain rather than an archive. It is queryable: via MCP servers, a business user or another system can ask a plain question and get a grounded answer, because the structure underneath is real. And it has memory and provenance: everything is logged and git-versioned, so you can see how the brain developed over time and trace the lineage of every fact back to its source — which document, which SQL, which interview it came from. This is the payoff of treating everything as metadata first. A company brain compounds: every source captured, every gap filled, every interview added makes the next question easier to answer. It is the difference between an organization that has documents and one that owns its knowledge.
Efficiency here is not about working faster for its own sake; it is about refusing to spend where nothing compounds. Too many meetings with no clear result, expensive licenses for a fraction of a tool you actually use, tokens burned on chat with no structured vault behind it — each is a leak. The efficient move is to name the leak and close it: cut the meeting time, drop the bloat, and put the freed resource into the structure that keeps paying back. The multiplier is structure. Model the knowledge once, in the open, and everything downstream — SQL, lineage, docs, answers, the next meeting’s prep — is generated from it instead of rebuilt by hand. That is the difference between effort that evaporates and effort that accumulates: the first is a cost, the second is an asset. Ask the honest question in every meeting and every purchase: what value are we adding, and are we doing it the right way? Efficiency is what you get when the answer is allowed to change what you do.
Continuous learning is the concept that is most often forgotten and matters most. The field is not standing still: AI, generative modeling, new platforms, new methods, the shifting role of the architect — the ground moves every few months. A skill set that was current two years ago is already partly stale. So learning cannot be a one-off training you did once; it has to be continuous, built into how the work runs, or you quietly fall behind while feeling busy. And it is not only the modelers. The whole team has a lot of new things to learn — and so do the stakeholders. People need to learn the new concepts, understand the business model, the glossary and the processes, and get comfortable with using AI and prompt engineering. That is a real curriculum, not a memo. If only the abstract thinkers learn and everyone else stays where they were, the gap between the model and the people who must use it only widens. Bringing the stakeholders along — teaching, not just delivering — is part of the job now. And we should certainly not depend on sending people to a single training or workshop and expecting everything is then done. It isn’t. A one-off session plants a seed at best; without reinforcement it fades within weeks. Real learning is continuous and embedded in the work — revisited, practiced, and supported day to day — not a box ticked once on a calendar. Crucially, continuous learning cannot be left to good intentions — it has to be supported by a method and a structure. The same discipline BTM applies to data applies to learning: a structured place for what the team is learning, curated knowledge that compounds instead of scattering, trainings and workshops with a repeatable shape, and AI to help surface and reinforce what matters. Structure is what turns “we should all keep learning” from a slogan into something that actually happens, week after week. Take it one step further and do the BTM thing: build continuous training into the system, and model the learning itself. Just as we model contracts, deliveries and incidents, we can model the learning — what each person and role needs to know, which concepts and processes are covered, and keep track of knowledge levels over time. When learning is metadata, gaps become visible (“who still needs the new glossary, the new AI workflow?”), onboarding has a defined path, and the training is not a hopeful event but a tracked, improving part of the platform. Model the learning, and you can actually manage it. This is why the trainings and workshops matter, and why the whole approach leans on a growth mindset. Continuous learning turns the pace of change from a threat into an advantage: the teams that keep learning compound their skill the way structured knowledge compounds. Keep learning, together — it is also, genuinely, part of the fun.
Automation amplifies whatever you feed it. Feed a generator inconsistent naming, ad-hoc modeling choices, and undocumented exceptions, and it will manufacture that inconsistency at industrial scale. Data discipline is the unglamorous precondition: consistent naming conventions, one way to model each pattern, rules written down and followed, exceptions made explicit. Not because tidiness is a virtue, but because generation only works on regular input. This inverts the usual pitch about automation. Teams hope tooling will compensate for missing discipline; in practice the dependency runs the other way — discipline is what makes the tooling possible. A template can only generate what conventions make predictable; deterministic SQL generation is only deterministic because the modeling choices upstream are constrained. Every standard you enforce is a class of code you never have to hand-write again. As a signature principle, it is the human half of structure beats magic: structure is what you build, discipline is how you keep building it the same way. And discipline in MDDE is not self-denial forever — the platform progressively takes it over, as conventions become checks and checks become generation constraints, until the disciplined way is simply the only way the platform works.
Data-driven is the practice of letting measurement steer the decision rather than intuition. You instrument the thing, watch what the numbers actually do, and act on that — which sources fail most, where the model drifts, which contracts are chronically late. It is the operating mode that observability, accountability and reality checks all depend on: none of them mean anything unless someone is willing to be moved by what the data says, even when it contradicts the plan. The important extension is that this applies to our own process, including our own learning. If continuous learning is real, it has to be data-driven too: track knowledge levels, see who has picked up the new concepts and who hasn’t, measure whether a training actually changed anything, and improve the program on that evidence — not on the feeling that “we did the workshop, so we’re good.” Dogfood it: model the learning, measure it, and let the data guide what you teach next. Data-driven is the twin of fact-based: fact-based says only verified truth counts, data-driven says let the measurements decide. Structure makes it possible — when everything is metadata, the numbers are already there to steer by.
Decomposition is the move that makes complexity workable. A tangled IT landscape, a sprawling business process, a 268-table metamodel — none of it can be held in one glance. The way through is to break the whole into parts small enough to name, model and reason about on their own: an entity, a mapping, a rule, a concept. Each part is self-contained; together they compose the whole without anyone having to hold the whole in their head at once. This is what thinking in structures actually is. You decompose until each piece has one clear responsibility, then you compose the pieces through explicit relationships — not by blurring them together. It is also what makes the work checkable and reusable: a small, named part can be verified, corrected and reused, where a monolith can only be admired or feared. Decomposition and simplification are a pair. Decomposition splits the whole into the right parts; simplification makes each part as plain as it can be without losing what matters. Do both and the model becomes something a team — and a machine — can actually work with.
Simplification is the discipline of removing everything that isn’t earning its place. Most users of a heavy modeling tool only ever touched a fraction of it; most models carry structure no one uses. The simple move — build on your own requirements, reuse only the fraction you actually need, strip the rest — is not laziness, it is precision. What remains is easier to understand, cheaper to maintain, and far harder to get wrong. The bar is Einstein’s: as simple as possible, but not simpler. Simplification is not dumbing down; it is finding the smallest form that still carries the meaning. Cut a necessary distinction and you have oversimplified; keep an unnecessary one and you have hidden the signal in noise. The skill is knowing which is which — and that comes from having decomposed the problem first. Simplification is the twin of decomposition: break the whole into the right parts, then make each part as plain as it can be. Together they are what lets a complex landscape be modeled clearly instead of merely completely.
Breakthrough Modeling is new, but it is not a blank page. It stands on the shoulders of giants. Decades of hard-won thinking already exist — Data Vault, Kimball and dimensional modeling, Inmon, the Unified Star Schema, universal data models, Alec Sharp’s business-oriented data modeling, FCO-IM, ensemble modeling — and throwing that away to feel original would be its own kind of waste. The people who built these methods solved real problems; the answers are in the literature. So the discipline is to distil the essentials and keep what works. Not adopt one school wholesale, and not reject them because they predate AI — but read across them, take the parts that still hold, and leave the parts the world moved past. Data Vault’s separation of keys and change, Kimball’s grain discipline, FCO-IM’s fact-oriented verbalization, Sharp’s insistence that the business can see itself in the model — these are wheels already built. Reuse them. What is genuinely new is the combination and the medium: those essentials expressed as structured, AI-readable metadata, unified across structure, movement and meaning. Don’t reinvent the wheel — put the good wheels on a new vehicle. Originality is in the synthesis, not in pretending the past didn’t happen.
It is easy to talk about modeling as if it were only rigor and discipline. But let’s not forget the obvious: modeling is fun. Taking a tangled, half-understood domain and finding the structure in it; watching a model generate a working platform; seeing a stakeholder’s face when the picture finally clicks — that is genuinely satisfying work. The discipline is in service of the pleasure, not a replacement for it. And the timing makes it more so. These are exciting times. We are privileged to have this much new technology at our disposal — AI, generation, visualization, structured knowledge — and the possibilities it opens up are real and new. The people who came before us solved hard problems with far less; we get to build on all of it and go further. That is not a burden, it is a gift, and it is worth enjoying. Fun is not the opposite of serious. A team that enjoys the work challenges more freely, experiments more, and sustains the momentum that breakthroughs need. Keep the positive energy — no-nonsense, but genuinely glad to be doing this — and the whole approach stays alive rather than dutiful. Do the disciplined thing, and have fun doing it.
Extreme Modeling is Breakthrough Modeling with the volume turned up to where the work actually lives. The demo version is tidy; the real version runs while new sources arrive, sources change, companies merge, platforms shift under sovereignty rules, teams churn across offshore and near-shore, and the methodology itself is moving toward mesh and data products. Modeling under that — not despite it but with it fully in view — is the extreme in the name. It demands a new way of working: planmatig, structured, intelligent, focused and dedicated. Not heroics, and not ad-hoc firefighting, but a disciplined system that can absorb concurrent change without losing the thread. That is why the attitude concepts matter as much as the modeling ones — extreme modeling is impossible without ownership, urgency, discipline and clarity carrying the load. Extreme Modeling is the honest AKA for the whole approach: not modeling in a vacuum, but modeling at the scale and pace the enterprise really has. Think big, own your part, check it against reality — and keep the structure intact while everything around it moves.
Extreme ownership is taking full responsibility for your part of the chain, without hedging. Not “the source was late,” not “that’s not my system” — but owning the outcome and the interfaces around it. The idea comes from the book Extreme Ownership (Jocko Willink and Leif Babin): leaders own everything in their world, and the moment blame moves outward, improvement stops. In a data landscape of many systems and many teams, that stance is what keeps the seams from becoming everyone-else’s-problem. Ownership, contracts and accountability are one idea seen from three sides. Ownership is the stance; a contract is the explicit, agreed interface; accountability is being measured against it. To own your part credibly you must be in full control of it and able to explain — in machine-readable form — what was agreed and how it is measured. A promise you can’t express as data and monitor is not ownership, it’s hope. That is the BTM version of the idea: not a motivational poster, but ownership made checkable. Structure the agreement, monitor it continuously, and let the record show whether you kept your word. No excuses works only when the evidence is in the open.
Working fact-based means grounding decisions in what is verifiably true, not in the loudest opinion or the most comfortable assumption. In data work this is almost too obvious to say — yet plans are constantly built on “the source is probably fine” or “everyone knows this rule.” Fact-based is the discipline of checking: look at the actual data, run the query, confirm the assumption before you build on it. When a claim can’t be backed by a fact, it is treated as a hypothesis, not a foundation. The sharp part is turning it on ourselves. We should be fact-based about our own work and our own learning too — dogfood the principle. Don’t assume the team understands the new glossary; measure it. Don’t assume the model matches the business; check it against reality. The same rigor we demand of the data, we owe to our own claims about progress, quality and knowledge. Fact-based pairs with data-driven: fact-based is the stance (only verified truth counts), data-driven is the practice (let the measurements steer). Together they replace opinion-led decisions with evidence-led ones — and make it much harder to fool ourselves, which is the most common failure of all.
Fail fast is the discipline that makes experimentation actually pay off. The point isn’t to fail — it’s to find out quickly and cheaply whether an idea holds, so that when it doesn’t, you’ve lost hours instead of months. You design the experiment to give a clear signal early, you read that signal honestly, and then you decide: double down, or drop it. What you don’t do is keep polishing something that, deep down, you already suspect won’t work. The enemy is sunk cost. The longer you’ve invested in an approach, the harder it is to walk away — and that reluctance is exactly what turns a small, useful failure into an expensive, drawn-out one. Fail fast is permission to let go: because the experiment was cheap and quick by design, abandoning it costs almost nothing and frees you to try the next thing. With AI making experiments faster than ever, the ability to stop becomes as important as the ability to start. This is reality check and efficiency applied to your own ideas: confront the experiment with the facts, and don’t waste time on what the facts reject. Try quickly, learn quickly, quit quickly when you must — and spend the saved time on the experiment that works.
Find the essence first is a way of working, and it points in two directions. Outward, in interviews: before you model a single table, understand the data and the bigger picture — what this domain is really about, what matters and what is noise. Grasp the essence, then drill down into the details. Do it the other way around and you build a precise model of the wrong thing, because you never stepped back far enough to see what you were modeling. Backward, toward the giants: the same skill, aimed at the literature. Data Vault, Kimball, Inmon, the Unified Star Schema, Alec Sharp, FCO-IM, ensemble modeling — you don’t have time to adopt each one whole, and you don’t need to. The move is to quickly identify the essentials of each — the one or two ideas that actually work and are worth keeping — and learn from them without drowning in the whole apparatus. Distil, then reuse. This is the top-down half of thinking in structures, and it pairs with decomposition and simplification: find the essence, break it into the right parts, and make each part as plain as it can be. Essence first is what keeps the detail in service of the meaning instead of burying it.
Accountability is the measurable side of ownership. Ownership is the stance you take, a contract is the interface you agree to, and accountability is being held to it — visibly, on evidence, over time. The three belong together: you take responsibility, you write down exactly what was agreed, and you let the metrics say whether it happened. Without the measurement, ownership is a claim; with it, it is a fact. The BTM requirement is specific: you must be able to explain, in machine-readable form, what was agreed and how it is measured. A responsibility that only lives in a slide or a conversation cannot be monitored, so it cannot really be owned. Expressed as metadata — the contract, the metrics, the thresholds — accountability becomes a query anyone can run: is the agreement being met, yes or no, right now? This is what turns good intentions into a system. Accountability continuously monitored means a missed agreement surfaces on its own, in time to fix it, instead of being discovered in a post-mortem. Full control, clearly stated, continuously measured — that is accountability with teeth.
Most modeling failures are not failures of thinking; they are failures of assembly. The source expert is in one meeting, the consumer in another, the rules in a document, the data behind a login, the decision in someone’s inbox — and because nothing is at the table at the same time, the model is stitched together from partial views that never quite line up. Get it all at the table means the opposite: bring every source, stakeholder, system and insight into one shared view, now, all visible and connected, so the whole picture is in front of everyone at once. The two words that carry it are all and now. All — not the convenient subset, but the sources, the consumers, the rules, the real data, the people who own each piece. Now — not spread across a sequence of handovers where context leaks at every step, but present together while the modeling actually happens. When the whole is on the table, the contradictions surface immediately, the assumptions get challenged by the people who know, and the model is built from a complete picture instead of a guess. This is what the collaborative layer and shift-left are for: they exist to get everything and everyone to the table early. Structure and AI make it finally possible — one connected, queryable source everyone can see — but it still takes the deliberate move of pulling it all together. Get it all at the table, and the model reflects reality because reality was in the room.
The engine that makes the pipeline repeatable is its instruction sets — the rules, skills and prompts that tell the AI how to scrape, analyze, interview, enrich and generate. And the sharp move is where those instructions come from: not invented from scratch each time, but generated from existing, proven solutions. Look at a working data platform, a good model, a mature transformation library, and derive the instruction set that reproduces its logic in the BTM way. The best of what already works becomes the recipe for what you build next. This is the concrete offering. Turning a set of existing solutions into a reusable instruction set — one that a team (or an AI) can run to build their own company brain — is a skill, and it is deliverable and paid. It is also testable: run the instruction set against a known solution and check that the resulting metadata, rules and generated artifacts match. That test is what makes it a product rather than a promise. It sits on not reinventing the wheel and feeds vibe-coding: the instruction sets are what let you generate powerful apps and pipelines fast, grounded in proven logic rather than guesswork. Distill the solution, capture the instructions, sell the skill.
Let’s be honest with each other. Organizations want a great deal from AI right now, and the ambitions are high — rightly so, because AI genuinely offers incredible possibilities. But those possibilities come with a price, and pretending otherwise sets everyone up to be disappointed. The tools are not a magic switch you flip; the value on the other side is real, and so is the cost of getting there. And it does not happen overnight. You cannot buy the outcome, run one program, and expect the transformation to have occurred. What it actually takes is a real transformation of the way we work — the structure, the discipline, the redesigned processes, the continuous learning, the people with the right attitude. That is the price: sustained effort and changed habits over time, not a purchase and a press release. Saying this plainly is not pessimism — it is what makes the ambition achievable. Name the price honestly, commit to paying it, and the incredible possibilities become reachable instead of hype. The organizations that accept the deal — big possibilities, real transformation, no overnight shortcut — are the ones that will actually get there.
Challenging is the first move in the arc: impasse → challenging → breakthrough → momentum. It is the deliberate act of putting every assumption back under the light — the tool, the model, the process, the meeting — and asking the hard question the comfortable answer lets you avoid. Nothing changes in a stuck field until someone is willing to challenge what everyone quietly agreed to stop questioning. The important word is positive. This is not cynicism or contrarianism for its own sake. It is challenging aimed at forward motion: question the way we model, the way we spend, the way we meet — not to win an argument, but to open a door. Done right it energizes rather than deflates; people leave a challenging session with a sharper question and more momentum, not a bruise. And it turns inward as much as outward — challenge your own assumptions first, or the challenge to others isn’t credible. A challenging workshop is where this lives: bring the impasse into the room, question it out loud, and turn a stuck conversation into directed forward energy. A challenge without a next step is just noise; a challenge that lands opens the breakthrough.
Let’s be honest about it. This way of working asks a lot. It takes discipline, focus, real new energy, continuous learning, ownership of your part — and, especially inside large organizations, not everyone will find that comfortable. Some people are happy with the old, low-demand rhythm, and this isn’t that. Pretending everyone will love it would be a sales pitch, not the truth, and the truth serves people better. It is probably also true that not everyone will be suited to it. This isn’t about raw talent or seniority — it is about attitude. It needs people with the right positive, professional attitude: willing to challenge and be challenged, to keep learning, to own outcomes, to do the disciplined thing and still enjoy the work. That attitude can be grown, but it has to be present or chosen; it can’t be faked for long. The consequence is quiet and self-correcting: over time, the team composition adjusts itself. The people who embrace the discipline and the learning stay and thrive; others gradually move toward work that fits them better. That is not a threat and not elitism — it is what happens when a way of working genuinely asks something of people. Being upfront about it is itself an act of integrity. And this is, frankly, part of the provocation of BTM. A method that asks for discipline, focus and continuous learning is deliberately uncomfortable — it challenges the low-demand default and the “send them to a workshop and we’re done” habit. Saying out loud that it’s not for everyone is not a hedge; it is the same positive provocation that runs through the whole approach. The breakthrough was never going to come from the comfortable path.
Deliberate is the word that keeps surfacing because it runs underneath everything. Results come from deliberate structure, not luck; a good meeting is deliberately prepared, not improvised; a model is deliberately built from proven patterns, not stumbled into; learning is deliberate practice, not hope. The opposite of deliberate is ad-hoc, accidental, by-default — and by-default is exactly where the old, stuck ways of working came from. To be deliberate is to make the choice on purpose and be able to say why. Why this pattern, why this meeting, why this tool, why now — each answered, not assumed. It is the quiet common thread through the whole vocabulary: structure beats magic because the structure is deliberate, discipline is deliberate habit, challenging is deliberately questioning the default, efficiency is deliberately spending where it compounds. Take deliberateness out and each of them collapses back into chance. This is what separates a system from a scramble. A deliberate team isn’t slower — it wastes less, because it stops doing things out of habit that no longer serve the outcome. Choose on purpose, know why, and let nothing important happen by accident.
Two stances toward the same circumstances. The spectator watches what is happening and explains why it could not have been otherwise. The player steps in and treats the situation as something they are authoring — including the parts they did not cause. The distinction is not optimism versus realism, and it is not about how much authority you actually hold. It is about where you locate the cause of what happens next.
The classic mistake with any new technology is to pave the cow path: take the old, broken process and automate it, so now it fails faster. Process redesign refuses that. When structure and AI change what is possible, the honest move is to step back and redesign the process itself — how we gather requirements, how we model, how we meet, how data is delivered and checked — not to bolt automation onto a workflow that was shaped by the limits of the old tools. The parallel processes that make modeling hard — migrations, mergers, changing sources, volatile teams — are exactly where redesign pays off. Many of them exist in their current painful form because that was the only way to do it before. Prepared meetings instead of status theatre, shift-left checks instead of end-of-line acceptance, systems talking in metadata instead of humans relaying news — each is a redesigned process, not a digitized old one. Process redesign is challenging turned on the way we work: question the workflow, keep what genuinely serves the outcome, and rebuild the rest around what structure and AI now make possible. Automating a bad process is a cost; redesigning it is where the real gain is. Model the process, then have the nerve to redesign it.
Discipline is the unglamorous engine under the whole approach. Not motivation, not a burst of effort — the quiet habit of doing the structured thing every time: writing the precise minutes, versioning the decision, keeping the model in sync, capturing the note before it evaporates. It is boring on any single day and decisive over a year, because structure only pays back if it is maintained without heroics. It is also the precondition for automation. AI can prepare the meeting, generate the SQL and answer the auditor’s question — but only from output that was captured consistently. One reliable habit (minutes always written, decisions always logged) is worth more than one more clever notation, because it is what the machine runs on. Install the habit and the leverage follows; skip it and the smartest tooling has nothing solid to stand on. (See The Power of Habit — the point is to make the structured move automatic, not effortful.) The pair to hold: discipline is what makes integrity real over time, and data discipline is this same habit turned onto the standards and conventions that make generation deterministic.
Radical clarity is clarity taken to the point where nothing important is left implicit. The abstract thinkers can build something correct and unreadable at the same time — a model only they can follow. That is not a model of the business; it is a private artifact. Radical clarity is the discipline of making the work land completely for the people who have to check it and act on it: plain language, sharp definitions, one idea named clearly rather than five ideas blurred together, and the why made as visible as the what. The word radical matters because half-clarity is the usual failure. A diagram that’s roughly right, a rule that’s mostly documented, an agreement that’s understood “well enough” — each leaves a gap where misunderstanding lives. Radical clarity means openheid van zaken: full openness. What is agreed, what is modeled, how it is measured — all explicit, all in the open, none of it hidden inside a tool or a head. It is a two-audience test: the same structure must be readable by a domain expert (visual, in their words) and by a machine (structured markdown, YAML, rules). If either side needs a translator, clarity has failed. Structure is what makes this possible from one source: say it once, clearly, and render it in the register each reader needs. Sharpen how you communicate, not just what you model — a radically clear model invites correction, while a murky one hides its own mistakes.
Integrity is the foundation the whole attitude rests on. It has two faces, and both matter. Integrity with others: say what you’ll do, do what you said, and don’t hide the parts that aren’t working. Integrity with yourself: be honest about your own output, your own assumptions, your own pace — especially when no one is checking. The second is harder than the first, and it is the one that makes the first believable. It is also structural, not just moral. A model, a decision record, a set of minutes — these are only worth anything if they are honest. Structure without integrity is just paperwork: a clean diagram of something that isn’t true, a decision log that records the story people wanted rather than the one that happened. The discipline of writing things down is what makes integrity checkable — you can hold a versioned record to account in a way you can never hold a memory. Say what you’ll do, do what you said, and be honest about the gap when there is one. That is what earns the trust every breakthrough depends on.
A reality check is the habit of holding the model and the plan up against what is actually happening — the real data, the real deliveries, the real behavior of the systems — instead of against the tidy assumption you began with. Models drift from reality quietly: a source changes shape, a delivery arrives half-empty, a business rule turns out to have three exceptions no one mentioned. The check is what catches the drift before it becomes a wrong number in a report. It is the pragmatic partner of thinking big. You can plan for every parallel process in theory, but the plan is only worth something if you keep confronting it with the facts on the ground. Run the query, look at the actual volumes, compare what was promised to what arrived. When the world disagrees with the model, the world wins — and the model changes. This is where observability and plausibility checks earn their keep: they are reality checks made continuous and automatic. Reality check is the attitude; data observability and plausibility checks are how you run it at scale, so the model stays honest against the world it claims to describe.
Recombine is the move that closes the loop. You find the essence, you decompose the whole into clean parts, you distil what works from the giants — and then you put it back together in a new way. The value isn’t in any single borrowed piece; it’s in the combination no one had assembled before. Data Vault’s key discipline, Kimball’s grain, FCO-IM’s verbalization, AI-readable metadata — each is an old wheel; recombining them onto one vehicle is the new thing. This is where originality actually lives. Not in pretending the past didn’t happen and inventing from scratch, but in synthesis: taking well-understood parts and composing them into something that behaves differently as a whole. Breakthrough Modeling is exactly this — structure, movement and meaning were all modeled before, separately; uniting them in one AI-readable model is the recombination that makes it a breakthrough. Recombine is the counterpart to decomposition: break down to understand, combine back to create. A pile of good parts is not a system; the system is in how they are put together.
Sense of urgency is what turns a good idea into a moved needle. The field is stuck, AI is changing the rules right now, and the tools to work differently already exist — so waiting is no longer caution, it is a decision with a cost. Every quarter spent paying for outdated tools, or burning tokens on unstructured chat, is a quarter handed to someone faster. And the fast movers are often the smaller, smarter organizations — the ones without the license contracts and the committee to protect them, free to build on their own requirements and ship. Urgency is the honest recognition that the gap compounds: the longer you wait, the further ahead they get, and the harder it becomes to catch up. The attitude that follows: don’t accept delays, stick to agreements, no excuses. Not panic — directed forward energy. Momentum. Ask “why wait?” and let the answer move you.
Visualization is how complexity gets shared. An IT landscape and its data flows are too tangled to hold in prose or in one person’s head; the way to make them graspable at every level — not just for the abstract thinkers — is to show them. A whiteboard sketch, a generated diagram, a clear picture of the model: these let a room full of people point at the same structure and argue about the right thing. The shift is that the picture no longer has to be hand-maintained. Diagrams generated from the underlying metadata, driven by AI skills and rules, are always current — the moment the model changes, the picture changes. That kills the old rot where the diagram on the wall drifts away from the system it claims to describe. Read the picture from the source; don’t transcribe it. Visualization serves feedback: you can only correct what you can see. Make the structure visible, and the domain experts can do their half of the modeling.
The first principle, and the one everything else rests on: results come from deliberate structure — models, rules, metadata, patterns — not from a clever prompt, a shiny tool, or hope. The tool is never the differentiator; the structure around it is. Two people with the same AI get very different results, and the gap is not talent or luck — it is whether there is a solid, explicit structure for the tool to work against. This is the same thesis that anchors the Structure Beats Magic personal brand, turned onto enterprise data. A model generated from disciplined metadata beats one hand-assembled in a rush; an AI grounded in a versioned, structured vault beats one improvising from chat. The “magic” people chase — the fast answer, the auto-generated pipeline — is real, but it only shows up on top of structure. Skip the structure and the magic is just a demo that breaks in production. So the move is unglamorous and decisive: build the structure first, and let the tools amplify it. Structure is what makes results repeatable instead of lucky — the difference between a trick and a system.
The future you will get if nothing changes. Not a plan, not a forecast — the predictable continuation of what already happens, arriving on its own. It is the past, extrapolated: the same architecture in three years with more tables, the same team dynamics with different names, the same backlog with newer tickets. Its defining property is that nobody chooses it. A decision can be examined; a default cannot, because it never presents itself as a decision. It simply shows up, on schedule, looking like circumstance.
Thinking big is refusing to model as if the feature in front of you were the only thing happening. In any real data landscape there is a great deal going on at once. Beyond building new functionality and adapting existing data flows to new business requirements, there are always major parallel processes you have to account for: new data sources, changing data sources, company mergers (integrating models and sources), migrations, platform changes driven by things like sovereignty requirements, highly volatile teams, offshore and near-shore teams, shifts in methodology, data mesh, data products. A model that ignores them is correct on paper and wrong in practice. Thinking big is not the opposite of decomposition; it is what tells you what to decompose. You hold the whole landscape in view — all the moving processes — and then break the work into parts that still fit that bigger picture. Miss the scale and you optimize one flow while a migration or a merger quietly invalidates it. Handling all of this at once requires being genuinely planmatig — systematic, structured, intelligent, focused and dedicated. That is a different way of working, and it is the reason Breakthrough Modeling is also, honestly, extreme modeling: modeling under real, concurrent complexity rather than in a tidy demo.
Structure has a settled discipline; movement never did. How data actually moves and changes — the transformation — is the other half of modeling, and it is where the classic tools fell short. Data modeling tools drew boxes and lines for structure and then went quiet about the hardest part: the logic that turns sources into targets. So that logic ended up living where it always has — in code. And the code shows the cost of never modeling it. Transformation is usually captured in complex SQL queries, and in practice they are messy in familiar ways: logic copied and pasted from one query to the next until no two agree; data-quality fixes tangled together with functional business logic in the same statement; time logic (bi-temporal, as-of, snapshots) done ad-hoc and differently everywhere; source-specific quirks hard-wired into the transformation; and no clear layering, so staging, cleansing, integration and business rules all blur into one long, unmaintainable query. Change one thing and you can’t tell what breaks, because the intent was never modeled — only the SQL was written. Breakthrough Modeling treats transformation as something to model, not just to code. The mapping, the filters, the joins, the time policy, the rules — each becomes explicit, layered metadata, and the SQL is generated from it deterministically rather than hand-written and copied. That is what closes the gap the tools left open: movement modeled with the same rigor as structure, so the logic is consistent, layered, traceable, and no longer hiding in a thousand bespoke queries.
Extend trust, and check anyway. Not out of suspicion — checking is precisely what makes trust affordable. Without verification, trust is a bet placed on a person's competence and mood on a given afternoon. With it, trust becomes a working assumption you can safely hold at scale. The discipline comes from clinical research, where it is not a slogan but a regulatory reality. A trial result is not believed because the investigator is honest; it is believed because source data verification, monitoring visits, audit trails and independent review make it checkable. Everyone involved is trusted. Everything is verified anyway. Nobody experiences this as an insult, because the check is a property of the system rather than a judgement about the person. That distinction is the whole concept: verification is structural, not personal.
Right now, most people and organizations are still tiptoeing around it. They sense that AI changes everything, that the old way of modeling and working won’t hold, that a transformation is coming — and they circle it, run a pilot, form a committee, wait for it to get clearer. They postpone, because underneath it they’re uncertain — unsure where to start, unsure it’s worth the upheaval, so they delay. Meanwhile the ground is already moving. This is the wake-up call: the change isn’t on the horizon, it’s here, and hesitating is itself a decision with a cost. This is exactly the impasse we started from. People postpone because there is still too much unclear — the field is fragmented, the tools contradict each other, the new roles and methods haven’t settled, and AI has scrambled the rules faster than anyone has answered them. That lack of clarity is not laziness; it is a genuine stuck point. And it is precisely the impasse Breakthrough Modeling exists to break: impasse → wake-up call → breakthrough → momentum. A wake-up call is not panic; it is clarity arriving suddenly. It names what everyone half-knows but hasn’t acted on: the tools are ready, the smaller and smarter organizations are already moving, and the comfortable pace of “we’ll get to it” has quietly become dangerous. The honest response is to stop circling and face it — accept that it comes with a price, that it takes real change, and start now. And here is the turn: Breakthrough Modeling offers the clarity the impasse lacks. Where things are murky, BTM makes them explicit — one model uniting structure, movement and meaning, a shared vocabulary of concepts, everything as open, versioned, readable metadata. Full openness — openheid van zaken — is the answer to “there’s still too much unclear”: not another opaque tool, but a way of working where what is agreed, what is modeled and how it’s measured are all in the open. The wake-up call points at the problem; BTM is the clear way through it. This is urgency with the alarm turned on, and it is provocation pointed at the whole industry’s hesitation. The point isn’t fear — it’s momentum. Wake up, look at what’s actually happening, and move deliberately while there is still room to lead rather than scramble to catch up. And of course — consistent with everything BTM stands for — the insights of BTM itself may be challenged. That’s not a weakness; it’s the point. An approach built on openness and challenging has to hold itself to the same test, and it welcomes it.
Lineage rendered for humans who make decisions — not just a technical dependency graph, but an explanation of where data comes from and what it means.
Modern platforms are full of complex metadata, lineage and technical detail. Business leaders need clarity, not code — this is the practice of delivering it.
Models designed and presented so the business can see its own concepts in them and derive value, rather than treating the model as a purely technical artefact.
The art and practice of generating diagrams from metadata so they are always correct, consistent and readable by both business and technical audiences.
Making the transformation logic between source and target legible to the business — mappings expressed as reviewable, metadata-driven statements of intent instead of opaque code.
Bridging data engineering and business understanding — metadata, lineage, data contracts, mappings and models explained, governed and applied so both sides understand them.
Modeling as a shared, live activity rather than a handover: business experts and engineers build the model together, using business-friendly notation and diagrams generated from metadata, so the model is a conversation both sides own — not a technical artefact thrown over the wall.
Business meaning does not come in isolated entities; it comes in clusters. "Order" only makes sense together with Customer, Product, Delivery, and Invoice — a small constellation of concepts whose relationships carry as much meaning as the concepts themselves. Conceptual constellations is the practice of modeling and presenting these clusters as connected wholes: pick a center of gravity, map the concepts that orbit it, and make the lines between them explicit. The constellation framing does two jobs. For the business, it is a communication device — a one-page star map of "everything around Order" is far easier to validate in a workshop than an entity list or a full ER diagram, which is where business-friendly diagramming meets semantic modeling. For the platform, it is a scoping device: a constellation is a natural unit of delivery, and its edges are candidate integration points with neighboring constellations. Stitched together, the constellations become the map of the enterprise: overlapping clusters that share stars, converging on the enterprise ontology one recognizable neighborhood at a time — bottom-up and business-first, rather than as one intimidating top-down model.
Meaning is usually the accidental byproduct of a platform: it accretes in column names, dashboard labels, and whatever the last consultant wrote in the wiki. Semantic engineering treats it instead as an artifact you build — with requirements, versions, tests, and maintenance. Definitions are authored, reviewed, and released; changes to meaning go through the same rigor as changes to code, because in a model-driven platform a changed definition regenerates real artifacts. The engineering framing brings engineering questions to the semantic layer: What is its interface (which concepts are published, which are internal)? What is its test suite (do generated artifacts still conform to the definitions)? What is its change process (who approves a redefinition of "active customer", and how do consumers get migrated)? Guarding against semantic drift stops being vigilance and becomes regression testing. Within the semantic family, the division of labor: semantic modeling captures the meaning, semantic design shapes it well, semantic engineering keeps it true over time — and semantic data engineering is the closest sibling, the discipline of doing data engineering itself from that semantic layer. At platform scale the output of all four is the unified semantic fabric.
Semantic modeling captures what things mean before deciding how they are stored: the business concepts, how they relate, and the rules that constrain them. A semantic model says "a Customer places Orders; an Order has exactly one billing Agreement" — no tables, no keys, no load strategy. Those come later, and in MDDE they come derived: the physical model is generated from the semantic one plus architectural patterns, not designed by hand a second time. The sequencing is the point. When physical structure comes first, meaning gets reverse-engineered out of column names and tribal memory — the platform runs on guesswork. When meaning comes first, every downstream artifact inherits it: business-friendly data modeling makes the semantic model a medium the business can actually read and challenge, and the enterprise ontology gives concepts a home beyond a single project. Within the semantic family, this concept names the activity: semantic design is the design discipline around it, and semantic engineering is the practice of building and maintaining the resulting layer as an engineered artifact.
A translation boundary (borrowed from DDD, owned here for data platforms) that isolates a clean, governed model from the quirks and corruption of upstream source systems.
Most governance concepts govern data; architectural governance governs the structure the data lives in. Which patterns are sanctioned, what each zone may and may not contain, how layers connect, what a "source access view" or "integration object" must look like — these are architectural rules, and without active governance they erode one pragmatic exception at a time until the platform is an archaeology site of abandoned conventions. The MDDE mechanism is to make the architecture rules part of the model world rather than a slide deck. When architectural patterns are encoded as templates and constraints — the stance of model-driven data architecture — the generator physically cannot emit an artifact that violates the zone rules, and model-level checks can flag a design that puts business logic in the wrong layer before anything is built. Decisions still need human judgment; architectural decision records capture the judgment, and the encoded rules enforce its outcome. This is also how architecture escapes the ivory tower: instead of architects policing compliance by review, the standards are executable and ambient, and the architects' time goes into evolving the patterns rather than defending them.
An integration architecture organised around business entities and domains — the shape of the business, not the accident of which source systems happen to exist.
A diagram is a picture until you can click it. Then it is an interface. SVG is not an image format in the way PNG is. It is markup — every shape is an element, and any element can be wrapped in a link. So a generated diagram can carry, on every box, a route to the thing the box represents: the entity's documentation, its lineage, the mapping that fills it, the decision record that created it, the source table it came from. Nothing about that is new technology. `<a xlink:href="/concepts/impact-analysis.html">` has been in the spec for two decades. What makes it powerful is what sits behind it: when the diagram is generated from metadata, the generator already knows the target of every link.
Systems meet only at agreed interfaces — never in each other's internals. Every system has an inbound side (what it accepts) and an outbound side (what it promises), and between two systems sits a data contract: agreed up front, versioned, and continuously monitored. That last word is what separates decoupling from a diagram. Boundaries drawn on a slide are not boundaries; boundaries that break a build when violated are.
Enterprise Context Architecture (ECA) is the deliberate design and governance of the context an organization feeds to its AI systems — the instructions, memory, retrieval sources, metadata and signals that decide what a model thinks about. It extends the data-governance mindset one layer up: where an Enterprise Ontology governs what data means, ECA governs the context that surrounds that meaning at the moment of use. Context becomes an asset with an owner, a lifecycle, quality checks and access rules — versioned and curated like any other data product, rather than pasted ad-hoc into prompts. The premise, borrowed from "Context Is King": semantics describe, but context decides — so if AI decisions are to be trustworthy and repeatable, the context layer must itself be architected and governed. ECA is the enterprise-scale counterpart of an individual's context-engineered second brain.
A high-level, agreed set of core concepts — Party, Product, Agreement, Event — that everyone means the same thing by. The anchor that the Business-Oriented Logical Data Model elaborates into a business-shaped integration layer capturing enterprise-wide meaning, flexible enough to historize and reuse.
Generate the building blocks, not the picture. A palette — a draw.io shape library, a set of stencils — produced from the model, so every entity, every attribute list, every relationship type already exists as a correct, drag-and-droppable shape. The layout stays human. The content cannot be wrong. This resolves a tension that generated diagrams alone never quite settle.
A named place in the architecture, above the semantic and context work, where the platform stops carrying data and starts concluding from it. Integration makes sources compatible; semantics fix what the data means; context supplies what the situation is. The reasoning layer is where those combine into a decision, a classification, a score, a derived fact — computed once and governed, rather than re-derived ad hoc in every report, dashboard and prompt. It is the layer AI leans on hardest, which is exactly why it needs structure under it: a model asked to reason over raw, unharmonised, meaning-less data is guessing. Treat reasoning as a layer with a boundary — what enters is governed meaning plus context, what leaves is an explainable conclusion with lineage — and AI reasoning stops being an ungoverned black box.
Logical views on top of historized raw source data that rename cryptic columns, hide technical metadata and apply business-friendly filtering, so downstream modeling works against a clean, stable, legible interface instead of raw source quirks. Their design counterpart is the Logical Source Model — a reverse-engineered, business-shaped representation of a physical source system.
Every organisation runs the obvious systems — accounting, CRM, HR, operations — all systems of record: authoritative, transactional, backward-looking. The one almost nobody builds is the layer above them: a system of intelligence that reads across all the records, detects what's changing, validates its own data (flag, don't guess), and senses the world outside (competitors, market). Built from data + AI + rules; rules is the forgotten third. A layer, not another silo — its value is that it's cross-cutting. Missing systems are invisible precisely because nothing breaks; you just never surface the insight, report bad data straight-faced, and notice the competitor's move a quarter late.
The sticky, exec-friendly hook for why reporting/BI isn't intelligence. A system of record is a rear-view mirror: accurate, detailed, essential — and entirely about what's already behind you. Most organisations, for all their expensive systems, steer forward by looking backward. The intelligence layer is the rest of the dashboard: detect + validate are the windscreen (what's ahead), sensing outward adds the side mirrors (what's coming up alongside). You need all three to have a sense of your surroundings rather than a precise picture of the road you've already driven.
You can do everything upstream right — clean pipelines, a governed semantic layer, trusted data — and still ship an AI that gives wrong answers. The reason sits in a layer most teams treat as pipework: retrieval. Retrieval is where the question gets shaped, the right context gets found and filtered, and the answer gets assembled — and every one of those is a design choice, not a technical default. Hand the model a good foundation through a careless retrieval layer and the foundation never reaches the decision. The failure is quiet, which is what makes it dangerous. Retrieval that returns part of what was needed produces an answer that looks right and is wrong — the model fills the gap plausibly, the output is fluent, and nothing flags that a piece was missing. The canonical case is the rule that has an exception: a policy that holds "resolve within 24 hours, except for accounts under $10K, which get 72." Retrieve the rule without its exception and the agent confidently applies the wrong SLA. The rule without its exception isn't a smaller truth — it's an operational risk wearing the costume of a correct answer. Which is why the shape of what gets retrieved is a product decision, owned deliberately, not an infrastructure setting left on its default. It determines whether meaning survives the trip from your governed data to the model's context: does the retrieval carry the rule and its exceptions, the definition and its scope, the number and the caveat that qualifies it? Get that right — the rule and its exception together — and you get trusted output. Get it wrong and no amount of upstream quality saves you, because the model reasons over what it was handed, not over what you stored. For a semantic-first architecture this is the last mile and the one most likely to be skipped. The semantic layer defines meaning; the retrieval layer decides how much of that meaning reaches each decision. Treat it as plumbing and you've built a beautiful foundation that leaks at the tap. Treat it as a product surface — designed, owned, tested against the exception cases — and the foundation finally shows up where it counts.
The loop that makes an intelligence layer trustworthy: rules check the output at input, intermediate and output stages; on a mismatch the system flags rather than guesses; a human resolves the flag; and the rule itself is sharpened. Two things improve together — the data and the rules. Rules are the forgotten third of data + AI + rules; without them you've built something worse than nothing, because it's wrong with confidence.
Every platform accumulates local semantics: this mart's definition of "customer", that pipeline's notion of "active", a third team's version of "revenue". A unified semantic fabric is the architectural counter-move — a single connected layer of concepts, relationships, and definitions that spans sources and domains, so that meaning is resolved once and referenced everywhere. "Fabric" is deliberate: not a monolithic central model that one team owns, but a woven network where domain models connect through shared concepts — the enterprise ontology provides the shared vocabulary, and something like a metadata mesh provides the distribution model. Domains keep autonomy over their detail; the fabric guarantees that where their concepts touch, they agree. In MDDE terms the fabric is not documentation — it is input. Because integration layers, exposure views, and lineage are generated from the semantic layer, consistency is not a review outcome but a structural property: two artifacts derived from the same fabric cannot silently disagree about what a customer is. This is the architecture-scale expression of what semantic design does at model scale.
A demystification: zones are physical/operational locations (Bronze/Silver/Gold, medallion); layers express modeling intent (harmonization, curation, marts). They are orthogonal, and conflating them is why medallion-vs-Data-Vault debates go in circles. Two named layers fall out: the Harmonization Layer (make sources technically and semantically compatible) and the Curation Layer (codify business rules, reference data and complex KPIs).
A pattern where every SQL file is self-contained with all metadata embedded as structured comments — executable on its own, yet deployable into any metadata platform.
Each transformation step is explicitly typed by its concern — source (src_), filter (flt_), aggregate (agg_), deduplicate (dd_) — one CTE, one concern, one label. A signal AI assistants read to know what a step is for, and a rule the SQL Advisor enforces.
A development surface where SQL is built and run one CTE at a time while metadata flows in as you shape it — not only out at deploy. A built-in SQL Advisor does per-CTE static analysis: it flags separation-of-concerns violations, scores complexity, and warns on non-determinism.
Automated CTE-by-CTE comparison of query results before and after a migration — finding the root-cause difference instead of only comparing final output.
Generating SQL that is byte-for-byte reproducible: no non-deterministic functions, no sorting ambiguity, no random tie-breaks. A hard invariant, not a nice-to-have.
The framing of AI assistants as sous-chefs: they don't improvise SQL freehand, they work inside the structured Recipe Cells DSL, reading the concern of each cell and filling in the how. The cookbook (the patterns and rules) constrains the genie, so AI help stays deterministic and reviewable rather than a fresh guess each run.
The architectural principle that rules and metadata should sit next to the data and the engine, not in a separate governance system that drifts out of sync.
A recipe written as declarative cells — SOURCES, CLEANSES, FILTERS, AGGREGATES, SOURCE_MERGES — each compiling to exactly one focused CTE that does one kind of work. You declare intent per step; the engine emits clean, single-concern SQL. The notebook is the kitchen, the SQL file is the dish; a seam date marks where an old source table hands over to a new one.
The counter to one-way generation: as you shape SQL in a notebook or annotate it SQL-first, that work flows back into the metadata, and metadata flows forward into your editing surface. Generation and authoring stay in sync in both directions, so hand-edits are first-class and drift is prevented.
In a SOURCE_MERGES cell, the seam date is the precise pivot point where an old source table stops and a new one takes over, with explicit rules for the overlapping history around the seam. It makes a source migration a controlled, dated handover instead of a silent discontinuity in the data.
SQL files that describe themselves: structured metadata embedded as part of the file so the query explains its own schema, intent and lineage.
Treating SQL itself as the modeling surface: the model lives close to the engine that executes it, avoiding a translation gap between a modeling tool and the running platform.
The plain-language framing of a source-table transition: the ingredients didn't change, but the pantry moved — upstream swapped one table for another partway through history. It frames data migration as a continuity problem (keep cooking the same dish from a relocated pantry), resolved by a seam date in SOURCE_MERGES.
The authoring metaphor inside Recipe Cells: exploratory work happens in the notebook — the kitchen, where things are messy and iterative (the soup) — and only the finished, clean result is written to the SQL file that production consumes (the dish, the serving). Keeps exploration and delivery deliberately separate.
The six conventional slots of a capsule — schema, properties, quality, contracts, classification, lineage — each a well-known place for one kind of metadata.
A harmonised, governed representation of business entities and relationships that acts as the shared handshake between domains and systems.
A CI enforcement mechanism that stages capsule adoption: it requires named metadata slots to be present before allowing a deployment, tightening strictness phase by phase.
A single, versioned, executable unit where data ships with its full metadata — schema, properties, quality, contracts, classification and lineage — in one place.
Traditional metadata is a rear-view mirror: harvested after the fact, describing what the platform already does, drifting the moment anyone changes a pipeline. Executable metadata inverts the relationship — the metadata comes first, and the platform's behavior is executed from it. A mapping definition is not documentation of a load; it is the input that deterministic SQL generation turns into the load. Change the metadata, and the platform changes. Being executable is what makes metadata trustworthy. Descriptive metadata can lie because nothing breaks when it is wrong; executable metadata cannot drift from reality because it is the source of that reality. This is the mechanical heart of the MDDE thesis: documentation, lineage, tests, and SQL all stay consistent because they are all executions of the same metadata. The concept names the property; its siblings name the practices around it. Metadata-as-code gives executable metadata an engineering lifecycle (versioning, review, CI), and self-describing SQL closes the loop from the artifact side, keeping generated code traceable back to the metadata that produced it.
Knowledge representation is the old AI discipline MDDE quietly rests on: expressing what you know — concepts, relationships, rules, constraints — in a structured, typed form that a machine can act on, not just display. A wiki page about "customer churn" is knowledge for humans; a modeled concept with typed relationships and executable rules is knowledge for the platform. This is why the form of metadata matters so much in MDDE. Free-text descriptions cannot drive generation; structured, typed metadata can. Models, an enterprise ontology, mapping specifications, and rule definitions are all knowledge representations — and the richer and more disciplined the representation, the more the platform can derive from it deterministically and the more safely AI can reason over it via a reasoning layer. The concept also explains MDDE's relevance in the AI era: large language models are strong at language and weak at ground truth. A platform whose knowledge is represented as structured metadata gives AI something solid to reason against — the difference between an assistant that guesses at your definitions and one that reads them.
Metadata is a code artifact, not an application's internal state. Definitions, contracts, mappings and models live in version-controlled files: reviewed in pull requests, tested in CI, deployed like anything else. No metadata trapped in a GUI, a wiki, or a vendor's repository. The claim is deceptively small. It is really a claim about who owns the model and what can be done with it — and almost every other MDDE and BTM capability depends on the answer.
Metadata is the whole game. Everything the approach depends on — the model, the mappings, the rules, the time policies, the lineage, the meaning — is metadata: structure and logic captured as data rather than buried in code or slides. Get the metadata right and the platform can be generated from it, the AI can read it, and the auditor can query it. Get it wrong or leave it implicit, and you are back to archaeology. The shift is to stop treating metadata as documentation about the system and start treating it as the substance of the system. Descriptive metadata drifts because nothing breaks when it is wrong; the metadata that matters here is the kind that runs the platform, so it cannot lie. Structured, versioned and open — markdown, YAML, rules in git — it becomes the single source that SQL, tests, docs and answers are all generated from. Metadata is the noun the whole family of concepts hangs off: executable metadata is the property that makes it run, the metadata star is the image of everything orbiting it, and metadata-as-code gives it an engineering lifecycle. Model it well and the leverage follows; ignore it and there is nothing for the tools to stand on. This is where all of it comes together. A great deal has to be modeled — structure, transformations, contracts, deliveries, calendars, incidents, the process, the change over time — and these models keep changing. The result of all that modeling effort is stored as metadata, accessible to the systems, to the AI, and to the MCP servers — and through those MCP servers, to other systems and to stakeholders who can simply ask their questions and get grounded answers. And it is all versioned over time: in git, and as versioned metadata, so you can see not just what the system is now, but how it got there. One evolving, queryable source, open to everyone and everything that needs it.
Data Mesh's decentralisation applied to metadata: each domain owns and publishes its own metadata as a product, while shared standards and a common model let it interoperate across the enterprise — federation without fragmentation.
Once the knowledge is captured as metadata files, you need to be able to query it at speed — and markdown-in-folders isn’t a query engine. The metadata mirror solves this: the files are read into a fast analytical database, typically DuckDB, that reflects their content. The crucial rule is directionality — files stay the source of truth, the database is the mirror. You never edit the database directly; you edit the files (in git, reviewable) and regenerate the reflection. That keeps the human-editable curation layer authoritative while giving everything downstream a fast, structured surface to run against. The mirror is what makes the brain usable. Apps read from it, MCP servers query it, analysts join across it — all at database speed, all from a reflection that was generated, not hand-maintained. Because it’s regenerated from files, it can never silently drift: if the mirror is wrong, the files are wrong, and the files are where you fix it. This is the pattern the whole ecosystem uses (concepts, the metadata model, the demo browser): metadata as files = truth, DuckDB as the queryable spiegel. Structure once, reflect fast, query everything.
A vision where metadata is not documentation on the side but the operating system of the platform: it governs models, lineage, contracts and execution.
Picture the platform as a solar system. At the center sits one body with all the mass: the metadata — models, mappings, rules, contracts. Everything else orbits it and shines with borrowed light: the SQL is generated from it, the tests are derived from it, the lineage is read out of it, the documentation is rendered from it. Nothing in orbit is authored independently; every artifact's position is determined by the star. The image earns its place because it makes the failure mode visible. In most platforms there is no star — SQL, docs, lineage tooling, and test suites are separate bodies on independent trajectories, and keeping them aligned is manual gravity that teams eventually stop supplying. With a metadata star, alignment is not maintained; it is inherited. An artifact cannot drift out of sync with the metadata for the same reason a planet cannot drift out of its orbit unnoticed. Metadata Star is the mental model; the metadata operating system is the same idea as an engineered system, and executable metadata is the property that gives the star its mass. If the metadata were merely descriptive, it would be decoration at the center — it is generation that creates the gravity.
Companion tables in the same schema and dataset that carry lineage, quality checks and masking rules alongside the primary data — metadata that travels with the asset.
The metadata mirror shows the current state: the files as they are now, read into a fast database. But the files live in git, and git already holds every earlier version of every one of them. Time-travel the model turns that latent history into a first-class dimension of the mirror. Instead of loading only the working tree, you walk the git log and load each commit's version of the metadata, tagging every row with the commit it came from and when it landed. The mirror stops being a snapshot and becomes a timeline. That makes a new kind of question answerable in an app: not "what does this table look like" but "what did this table look like on the fifteenth of July, and what changed since." A version column and a point-in-time filter is all it takes — pick a moment, and the mirror returns the state as of that moment, one row per object. It's an SCD2 pattern, except the history isn't hand-maintained: it's reconstructed from git, which was recording it for free the whole time. The symmetry is the point. Git is the version-control layer under the metadata files; model versioning is the concept of tracking how the model itself evolves. Here they become the same thing — git is the model-versioning clock, and the mirror reads its history the way it already reads the files' content. Structure once as files, version for free in git, and the model's whole evolution becomes queryable — you can watch a definition drift, catch when a rule was introduced, and trace, with lineage, the exact commit a change came from.
A catalogue of reusable modeling patterns that make data models consistent and scalable across projects and domains.
A shared, reusable schema for metadata itself — so models, mappings and platforms describe themselves in one consistent way.
Most governance programs fail on ownership: a central committee writes policy, engineering routes around it, and the business never sees either. Collaborative governance flips the model — definitions, rules, and quality expectations are authored together, by the people who own the meaning (business) and the people who make it run (engineering), in a shared, business-friendly medium. In the MDDE frame this is only practical because the model is the meeting point. When business concepts, mappings, and rules live in business-friendly metadata rather than in code, a domain expert can review and co-own them without reading SQL — and the platform still enforces them mechanically, because everything downstream is generated. Collaboration answers who governs; siblings like embedded governance and governance-as-code answer where and in what form. The payoff is legitimacy: rules that both sides wrote are rules both sides defend. It is the working antidote to ivory-tower architecture, and the same collaborative posture that collaborative data modeling brings to model design, applied to the governance layer.
Lineage carried in-band: every row (or asset) leaves a trail of markers — source, run, transformation, timestamp — so you can walk back from any value to its origin without a separate lineage system. Lineage you can follow from the data itself.
Downstream governance reviews what the platform already did; embedded governance shapes what the platform can do. The checks, standards, and controls execute inside the normal flow of work — at modeling time, at generation time, in CI, at deployment — so a non-compliant artifact is never produced in the first place, rather than produced, discovered, and remediated months later. Generation is what makes deep embedding possible. In a hand-coded platform, embedding governance means asking every engineer to remember every rule; in a model-driven one, the generator is a single choke point that all artifacts pass through. Put the controls there and compliance stops being a behavior and becomes a property of the machinery — the shift-left move applied to governance, all the way to the point of creation. Within the governance cluster, embedded governance names the placement of enforcement: inside the flow. Governance DNA supplies what is enforced (traits encoded in the model), embedded metadata governance supplies the carrier (rules riding in the metadata), and governance-as-code supplies the engineering form. This concept is where they all become operational.
Governance usually lives somewhere else — a policy portal here, a catalog there, an access-control system in a third place — and every hop the data makes is a chance for its rules to get left behind. Embedded metadata governance keeps the rules in the metadata that describes the data: classification, ownership, permitted use, and quality expectations are attributes of the dataset's metadata record, carried along whenever the metadata flows to a new tool, team, or platform zone. The design consequence is portability. Because MDDE metadata already travels — it feeds generation, catalogs, lineage tooling, and consuming domains, including across a metadata mesh — embedding governance in it means every consumer receives the rules for free, in the same payload as the structure. There is no separate synchronization problem between "what the data is" and "what you may do with it"; they are one artifact, versioned together under the metadata-as-code discipline. Within the governance cluster, this concept names the carrier: the metadata layer as the vehicle governance rides in. Its sibling embedded governance is about the flow — enforcement wired into the platform's generation and runtime processes. Carrier and flow together make governance ambient rather than administered.
Governance-as-code is the umbrella practice: not one policy in a script, but the entire governance layer — policies, quality checks, data contracts, access rules, standards — living as code and metadata under version control, with the same lifecycle as everything else in the platform. Governance stops being a parallel organization with its own documents and cadence, and becomes a first-class part of the engineering system: reviewed in pull requests, tested in CI, released with the platform. The MDDE twist is that much of this layer does not even need to be written — it is generated. Quality checks derive from model-level rules, contracts derive from interface definitions, lineage-based access checks derive from the mappings. The governance layer inherits the central MDDE property: derived artifacts cannot drift from their source, so the governance you can read is the governance that actually runs. Within the cluster, this concept names the form and scope of the layer. Policy-as-code is its atomic unit (one policy, one versioned rule); governance DNA is the inheritance mechanism that lets generated artifacts carry governance traits; embedded governance describes where enforcement executes. Together they replace the audit-season model of governance with a continuous, mechanical one.
Ask most teams where their data quality lives and they point at the front door: validation at ingestion, checks in the pipeline, a certified layer. All correct, all necessary — and all upstream of the moment that actually matters. Data doesn't cause damage when it enters; it causes damage when it is used. The dashboard that drives a decision, the app that shows a customer a number, the AI agent that reads a value and acts on it — that is where a stale or wrong figure turns into a wrong outcome, and it is precisely where almost no one is checking. The reassuring sentence that hides the gap is "the data passed all its quality checks last quarter." Quality is not a quarterly snapshot; it is a property of the data at the instant of consumption. A row that was valid at ingestion can be stale by the time a dashboard serves it, superseded by the time an agent retrieves it, or correct in isolation but wrong for this decision in this moment. Governance that only guards the entrance certifies that the data was fine when it arrived — a claim that gets weaker every hour it sits. This reframes governance from retrospective to inline. The traditional model is after-the-fact: audit the lineage, review the access logs, run the human review cycles, catch the problem next quarter. The point-of-consumption model enforces at the decision point: policy, access and quality checked at the exact moment the data is read, so a bad value is caught before it becomes a bad action rather than reconstructed afterward. It is the difference between a smoke detector and a fire report. For an AI-native platform this stops being optional. When the consumer is a human analyst, a slightly stale number is usually survivable — they sanity-check, they notice. When the consumer is an autonomous agent, there is no sanity-check between the data and the action; the point of consumption is the last line of defense. Which is the whole argument: build the checks where the data is used, not only where it enters, because that is where the cost lands.
DNA is the right metaphor because of inheritance. When ownership, classification, retention, and quality expectations are properties of the model — attached to the concept, the attribute, the mapping — every artifact generated from that model carries them automatically. Tag an attribute as personal data once, and every generated table, view, test, and documentation page downstream expresses that classification, because they were all synthesized from the same genome. Contrast this with bolt-on governance: a catalog team annotating artifacts after they exist, forever chasing a platform that changes faster than they can tag it. In the DNA model there is nothing to chase — an artifact cannot exist without its governance traits, because generation is the only way artifacts come into being. Coverage is 100% by construction, not by campaign. Within the governance cluster, Governance DNA names where the encoding lives: in the model, upstream of everything. Embedded governance describes the enforcement happening inside the platform flow, and governance-as-code describes the engineering form of the layer as a whole — DNA is the inheritance mechanism underneath both.
Two axes of lineage: horizontal traces data flow from sources to targets; vertical traces drill-down within a single asset, from column to table.
Lineage answers "where did this come from". Impact analysis answers "what breaks if I change it". Same graph, opposite direction — and the reverse question is the one people actually ask before doing anything. Sometimes called reverse lineage, downstream analysis or dependency analysis. The naming varies; the move does not. Start at a node, walk the edges forward instead of back, and collect everything that would be affected.
A perfectly reasonable-looking query, written in good faith, that redefines a metric or entity slightly differently from everyone else's — silently spawning a competing version of the truth. No malice, no obvious bug; just drift nobody notices until the numbers disagree in a meeting. The villain that governed mappings and deterministic generation exist to prevent.
An explorable graph that connects concepts, mappings, data assets and releases into one interactive map of how data flows and depends.
Knowledge you can’t trace is knowledge you can’t trust. Lineage-to-source is the property that every fact in the company brain carries its provenance: which document, which SQL, which interview it came from — and every change is logged and git-versioned, so you can see how the brain developed over time. Ask “where did this definition come from, and who changed it, and when?” and the answer is in the record, not lost. Two kinds of history matter here. Provenance points backward to the origin: this rule was extracted from that transformation, this definition came from that stakeholder interview. Version history shows evolution: because the metadata files live in git, you can watch the brain grow, diff two states, and roll back a mistake. Together they mean the brain is not a black box — it’s an auditable, explainable body of knowledge. This is what makes the whole pipeline accountable. It pairs with model versioning (how the model itself changes) and underpins accountability: you can only stand behind an answer if you can show where it came from. Log everything, version everything, and the company brain stays trustworthy as it grows.
Documentation generated directly from the operational model, metadata and lineage — it evolves with the platform instead of rotting in a wiki.
Nothing erodes trust in a data platform faster than two dashboards disagreeing about revenue. Metrics are where semantic inconsistency becomes visible to executives, because a metric is a definition compressed into a single number — filters, grain, time window, currency handling, exclusions — and every BI tool, notebook, and export is an opportunity to recompute it slightly differently. Metric governance applies the MDDE pattern to this specific, high-stakes domain: the metric's definition lives once, as structured metadata — its formula, grain, filters, and ownership — and every place the metric appears is generated or validated from that definition. Consumption surfaces such as exposure views carry the computed metric so downstream tools inherit rather than reimplement it, and checks flag any artifact that computes a governed metric its own way. Defined in business-friendly metadata, the definition is also something a controller can read and sign off on — which matters, because a metric definition is ultimately a business decision. The result is that "which number is right?" becomes a resolvable question with an owner, a version history, and a single answer — semantic drift contained at the point where it costs the most credibility.
Data-quality checks generated and driven by the model and its metadata rather than hand-written per table.
A policy in a PDF is a wish; a policy as code is a fact about the platform. Policy-as-code takes individual governance policies — "personal data never lands unmasked in the exposure layer", "every table has an owner", "metric definitions require review" — and expresses each one as an executable, versioned rule that the platform evaluates automatically, in CI or at generation time. The unit of the concept is the single policy, and that is what gives it its engineering properties. Each policy gets a lifecycle: it is proposed in a pull request, reviewed by the people accountable for it, tested against known-good and known-bad cases, versioned so you can answer "what did the policy say in March?", and enforced by machinery rather than memos. Exceptions become explicit, reviewable code paths instead of quiet Slack agreements. Closely related to governance-as-code — the sibling that names the whole governance layer built this way; policy-as-code is its atom. And in MDDE the natural home for these rules is the metadata layer itself, following the same versioned-artifact discipline as metadata-as-code: a policy over models and metadata can be checked before anything is generated, which is where enforcement is cheapest.
The book family. Four kinds of structured document, distinguished by what they do to the reader — enforce, direct, show, or execute. Naming them separately matters because organisations collapse them into one undifferentiated "documentation", and then wonder why none of it gets used. | Book | Answers | Binding? | Reader | |---|---|---|---| | Rulebook | What must always be true | Enforced automatically | The platform (and the auditor) | | Playbook | How we handle this situation | Procedural, judgment allowed | A person deciding | | Cookbook | How this is built, in practice | Pattern, freely adapted | A builder, mid-task | | Runbook | What to do, step by step, now | Deterministic, no judgment | Whoever is on duty — or an agent | The distinction is not formality but what happens when you deviate. Break a rulebook and the pipeline fails. Deviate from a playbook and you explain yourself. Ignore a cookbook and you have written something bespoke. Skip a runbook step and the thing does not work.
The gradual erosion of meaning in data when definitions, classifications and business intent are left ungoverned — combated with metadata-driven, text-first modeling.
Modeling is half social — business stakeholders negotiating what things mean — and half technical — engineers encoding that meaning into versioned code and tests. Treat only one half and the model fails: pure-technical models miss the meaning, pure-social ones never ship. The conversation is part of the engineering, not a prelude to it.
Two lenses on lineage: technical lineage shows joins and filters at code level; functional lineage shows business meaning and intent — why code-based lineage alone isn't enough.
"AI governance" and "data governance" are usually run as two programs, by two teams, with two budgets — and treated as if they were separate subjects. They aren't. They are the same chain seen from opposite ends. Data governance protects the enterprise from what goes in: poor-quality, incomplete or unauthorized data entering the system. AI governance protects it from what comes out: opaque decisions, bias, and automation running unchecked. Inputs on one side, outputs on the other, and a set of shared concerns — privacy, lineage, auditability, risk — sitting between them. Seeing the symmetry settles a common confusion, which is the belief that you can bolt AI governance onto an organization that never governed its data. You can write all the model cards, fairness checks and output guardrails you like; if the data feeding the model is ungoverned, you are inspecting the outputs of a process whose inputs you don't control. The output guardrail catches a bad answer after the bad input produced it — useful, but downstream of the real leak. Governing one end while neglecting the other isn't half a solution; it's a solution to the wrong half. The symmetry also locates a layer most models miss: the enforcement in the middle, at runtime. Design-time governance — policies, standards, catalogs — is necessary but static. What agentic systems add is the need for governance that acts in real time: policy checked, access enforced, quality validated at the moment a decision is made, with escalation when something crosses a threshold. That runtime layer is where input-control and output-control actually meet — the point where the two ends of the chain are held together instead of managed apart. The practical consequence for anyone selling or building this: lead with the chain, not the halves. An organization proud of its AI-governance policy but weak on its data foundation is governing the wrong end, and the fix isn't more output controls — it's the missing input discipline underneath. Structure at the source is what makes governance at the surface mean anything.
A historization pattern handling both valid-time and system-time, plus the ability to reprocess late-arriving corrections without losing history.
Data versioning is the discipline of keeping the history of the data itself: not just the current value, but every value it has had, with the dates that make time queryable. The workhorse is bi-temporal SCD2 — two timelines, one for when something was true in the world and one for when the platform knew it — extended with redelivery support so a corrected or re-sent load slots into history without rewriting it. Ask “what did this record say on the first of March, as we understood it then?” and the answer is a query, not an archaeology project. It is the twin of model versioning, and the pair matters. Data versioning captures how the data changed; model versioning captures how the model and platform changed. Confuse the two and history lies: a value that looks like a data change might really be a model change, and only tracking both clocks tells them apart. Together they let you reconstruct not just the numbers but the system that produced them, at any point in time. This is the foundation audits, reconciliations and reality checks stand on. You cannot prove new equals old, or explain why a number moved, without a faithful record of what the data was and when. Version the data, and the past stays answerable.
A historization design using two coordinated SCD2 layers, enabling deterministic integration and fast, reproducible as-of history.
A layer of views over dual SCD2 tables that hides temporal complexity and delivers clean, consumer-ready data to BI and reporting.
Pre-joined point-in-time tables that accelerate 'as-of X' queries on dual SCD2 models, giving fast, reproducible access to history.
A temporal optimisation that consolidates overlapping SCD2 records carrying identical attribute values into a single record, reducing query complexity.
A design philosophy where the metadata and governance infrastructure ships with the data platform itself — pre-configured and ready to deploy.
Making ETL transformations clear and repeatable by treating mappings as recipes: transparent, business-friendly, step-by-step.
Architecture that treats cost as a first-class design concern — efficient, transparent and sustainable platforms.
A data contract is a crystal-clear agreement between the system that produces data and the system that consumes it: the schema, the meaning, the volume, the timing, the quality expected. The point is that it is explicit and machine-readable — not a shared assumption or an email thread, but a specification both sides commit to and a machine can check. When the producer changes something, the contract is what says whether that change is allowed. Contracts only mean something if they are continuously monitored. A contract no one measures is just documentation; a contract with metrics attached becomes a live guarantee — the moment a delivery breaks the agreed shape, volume or freshness, it is visible immediately, on the producer’s side, before it poisons anything downstream. That is the difference between finding out at ingestion and finding out in a board report. Contracts are the interface layer of ownership: ownership is the stance, accountability is being measured, and the contract is the agreed thing you are measured against. Modeled as metadata alongside deliveries and calendars, contracts make the whole hand-off between systems queryable and enforceable. And an agreement is not a static fact — it has a lineage. At every level, a contract needs the same questions answered as data: how did this agreement develop over time, how is it monitored, who owns it, and who are the stakeholders? Version the contract like anything else, and the model can tell you not just what is agreed now, but how it got here, who changed it, and who has a stake in it holding. An agreement without that history is a claim; an agreement with it is governable.
A data delivery is a concrete hand-off: this dataset, arriving from that system, in this shape, at this time. Modeling deliveries means the expectation is explicit — the consumer knows what should arrive and when — so a missing or malformed delivery is a detectable event, not a surprise discovered three steps later. Expected deliveries are the unit that makes “the data is late” or “the data is wrong” a fact the system can state on its own. The important move is that a delivery carries quality metadata from the provider. The producer runs its own data-quality and plausibility checks and ships the results alongside the data: row counts, check outcomes, known gaps, confidence. The consumer no longer has to guess whether the data is trustworthy — the provider says so, in machine-readable form, and stands behind it. Quality becomes part of the delivery, not an afterthought at the receiving end. Deliveries sit inside their contract and along their calendar: the contract says what’s allowed, the calendar says when it’s due, and the delivery is the actual event with its quality attached. Modeled together, the whole flow between systems becomes observable and accountable.
Data observability borrows a hard-won idea from the software-monitoring world and points it at data: don’t wait for a broken report to tell you something went wrong — watch the health of the pipelines continuously and let the problem announce itself. The classic signals are freshness (is the data on time?), volume (did roughly the right amount arrive?), schema (did the shape change?), distribution (do the values still look sane?), and lineage (what does this break affect?). Together they turn silent failures into early warnings. It is a reality check made continuous and automatic. Instead of a human occasionally sampling the data, the platform is always comparing what arrived to what was expected — against the contract, the calendar, the provider’s own quality results — and raising a flag at the boundary the moment reality diverges. That is what makes accountability enforceable: you can only be held to an agreement that is actually being watched. Bring the observability domain’s ideas in, but model them as metadata rather than bolting on a separate tool: the checks, their thresholds, their results, and the incidents they raise all live in the same structured world as the model they protect. Observed, not hoped — and queryable.
Pipelines driven by models and mappings that declare the intended result, leaving the engine to derive the imperative steps — the missing link connecting a model-driven design to actually-running pipelines. You maintain intent; the how is generated.
A delivery calendar models the time rhythm of the data flowing between systems: this feed is daily by 06:00, that one is monthly on the third working day, this other one is event-driven with a maximum gap. Once the expected timing is data rather than tribal knowledge, lateness stops being something a human notices by accident. “Which deliveries are overdue right now?” becomes a query, and an alert can fire the moment a window closes without the data arriving. This is what turns timing from a source of quiet failures into a monitored guarantee. Without a calendar, a feed that silently stops is discovered when a downstream number looks wrong; with a calendar, the absence itself is the signal, caught at the boundary and attributed to the right system. It is the temporal half of a contract: the contract says what and how good, the calendar says when. Calendars pair with deliveries and incident modeling: the calendar sets the expectation, the delivery is the event, and a missed window becomes a tracked incident with an owner. Modeled as metadata, delivery timing is one more thing you can ask the platform about instead of hoping someone is watching.
When a delivery goes wrong — a feed arrives late, a quality check fails, a schema breaks, a plausibility test flags a suspicious jump — the usual response is a ticket in a disconnected system, and the knowledge evaporates once it’s closed. Incident modeling treats the incident as a first-class object in the model: what happened, on which delivery, against which contract, who owns it, what the impact was, how it was resolved. The incident is linked to the exact data flow it concerns, not filed away somewhere else. Modeled this way, incidents stop being noise and start being signal. You can ask the platform which sources fail most, whether a pattern of lateness precedes a migration, how long resolution takes, which contracts are chronically breached. Each incident is also the raw material for improvement: the fix becomes a new rule or a tightened threshold, so the same failure is caught earlier next time. Incident modeling closes the delivery loop: observability detects the problem, the calendar and contract define what “wrong” means, and the incident records it against an owner so accountability is real. Track incidents in the model, and the data supply chain becomes something you can manage instead of merely suffer.
Letting models drive the flow of agile delivery — the backlog and the platform stay in step because both derive from the model.
An architecture practice where models and metadata carry meaning across the enterprise, connecting design to running systems.
Using models and metadata to plan, manage and measure the evolution of a shared data platform — the point where modeling meets delivery.
Controlled migration of data platforms, modeling styles and sources driven by metadata and automation rather than manual rewrites.
Here is the distinction that matters more than ever. Data modeling has traditionally meant two things: the structure (the data model) and the transformations (how data moves and changes, described and explained through lineage). Both are essential and both stay. But they are not enough. We also have to model the whole process around them — the complex data delivery, the integration, the migration, the quality-checking — and the backroom that runs it all. And crucially, we have to model how the system changes over time. A data landscape is never static: new sources appear, sources change, systems migrate, platforms shift, teams turn over. If the model only captures the current structure, it goes stale the moment the landscape moves — and it moves constantly. Modeling the process means the deliveries, contracts, calendars, quality checks and incidents are part of the model, and so is the history of change itself. This is why Breakthrough Modeling is broader than data modeling: structure and transformation are the what, and the process is the how it actually runs and evolves. Model the process — delivery, integration, migration, quality, the backroom, the change over time — and the model finally describes the living system, not just a snapshot of its shape.
In the new era it isn’t just a simple data model anymore. The data models and transformations keep changing — along with sources, platforms and teams — so we have to model the change in the system itself, the way MDDE does with model versioning. We already had data versioning — bi-temporal SCD2, tracking how the data changes over time. Model versioning is its counterpart: tracking how the model and the platform change over time. Two different clocks, both of which have to be captured, or the history is only half told. The reality this exists for is concrete. Not every source can already deliver in the agreed, enterprise-wide integration format — some business entities are still one version behind, and that has to be modeled, not wished away. And outbound interfaces can sit on different versions at a given point in time too. So at any moment the landscape is a mix of versions coexisting: this source on v2, that one still on v1, this consumer expecting the new shape, that one not yet migrated. Model versioning is what makes that mixture explicit and manageable instead of a source of silent breakage. Once versions are modeled, the transitions become governable: you can see who is behind, plan the migration, keep a contract valid across versions, and let systems tell each other which version they speak. Version the model, and the living system — not just a snapshot — is finally something you can reason about.
Pipeline thinking asks "how do we deliver this dataset?"; platform thinking asks "what capability, once built, delivers every dataset of this kind?" It treats the data platform as a product with users (the delivery teams), an interface (models and metadata), and a roadmap of reusable capabilities — loading patterns, history handling, quality checks, exposure — instead of a growing pile of bespoke pipelines that each solve the same problems slightly differently. MDDE is platform thinking taken to its conclusion: if the capability is truly reusable, it can be generated, and the per-dataset work collapses into modeling. The metadata operating system is the platform's kernel; patterns like batteries included describe its standard library. Teams stop paying the one-off tax on every delivery because the platform, not the pipeline, carries the engineering. The economics are the argument. One-off pipelines scale cost linearly with datasets; a platform amortizes: the tenth source costs a fraction of the first, and improvements to a capability upgrade every pipeline that uses it. That is also why platform thinking is a delivery-method concern and not just an architecture concern — it changes what a team commits to per sprint, which is where model-driven agile delivery picks up the thread.
Shift-left is the discipline of moving the check as early as possible in the flow — quality, validation and schema enforcement pulled back from deployment gates to development and modeling time. The cost of a mistake grows the longer it hides: a misread requirement caught while modeling costs a conversation; the same mistake caught in an acceptance environment costs a release; caught in production it costs trust. So you validate the transformation against the expectation, look at the sample results, and confront the model with reality at the moment of modeling, not weeks later. It changes what “done” means. Instead of building for weeks and discovering the misunderstandings at the end, you make the discrepancy visible on day one: the stakeholder sees the newly modeled logic and says “that’s not what I meant” while it’s still a five-minute fix. Early detection isn’t a nice-to-have; it is the single biggest lever on the cost and pace of the whole effort. Shift-left is the principle; the collaborative layer is where it happens, feedback is what flows through it, and a completeness gate is what enforces it automatically. Move the check left, and problems arrive as questions during modeling instead of failures during release.
The usually-invisible work before any mapping, named and made first-class: profile the source, form hypotheses about keys and relationships, test them against real data (fanout / cardinality testing validates FK candidates by join fanout), and document what you found. It turns 'poke around until it works' into a repeatable, discoverable workflow with artefacts, and feeds the Source Access Views.
The hand-off between systems shouldn’t depend on a human relaying news. When systems talk to each other in metadata, a change announces itself: the producer publishes a structure change, the expected deliveries, the calendar, and the quality results of what it is sending — the outcomes of its own data-quality and plausibility checks — and the consumer receives that update automatically. The boundary between two systems becomes a live, machine-readable conversation instead of a support ticket. The payoff is that surprises turn into notifications. A schema change upstream arrives as a metadata update, not as a broken load discovered at 3 a.m.; a quality dip in a delivery is reported by the provider, not reconstructed by the receiver. Each system knows what to expect and is told when the expectation shifts, so the whole chain can adapt in time instead of after the fact. This is the interface that makes contracts, deliveries and calendars operational: they are only enforceable if the two sides actually exchange the metadata that describes them. Systems-talk-to-systems is continuous metadata integration applied across organizational and platform boundaries — the metadata flows so the data can be trusted.
An engagement principle: knowledge is transferred through deliberate pattern selection and shared documentation from day one, never via a late handover.
Autonomous data engineering is the end-state hypothesis: a platform where a model change is the change. New source, new attribute, changed business rule — the platform regenerates its own pipelines, tests, and documentation, and AI handles the judgment calls that templates cannot, all within the guardrails the model defines. Human engineers move from writing pipelines to curating models and rules. The critical distinction from AI hype is the order of operations. Autonomy without structure is vibe-driven AI — plausible output with no contract behind it. MDDE's route to autonomy runs through determinism first: deterministic SQL generation handles everything that can be derived mechanically, and AI is admitted only where the metadata gives it verifiable boundaries. The model is what makes autonomy safe rather than merely impressive. That makes this concept less a technology bet than a maturity ladder: hand-written → generated → self-adapting. Each rung requires richer generative models and a stronger metadata operating system underneath. The horizon is autonomous; the road there is structural.
Not every model can drive a platform. A boxes-and-lines diagram communicates; a generative model produces. The difference is completeness and precision: a generative model carries enough typed detail — entities, attributes, relationships, mappings, history semantics, quality rules — that SQL, tests, lineage, and documentation can be derived from it mechanically, with no human filling in the gaps between model and code. This reframes modeling effort. In documentation-driven shops, model detail is overhead — it will be out of date next quarter anyway. In MDDE, every increment of model richness is capability: add history semantics to the model and deterministic SQL generation can produce the temporal logic; add quality rules and the tests come out with the pipeline. The model's richness sets the ceiling on what the platform can generate — which is the master line in one sentence: the magic isn't in the model, it's in the structure you give it. The concept sits in the AI & innovation category deliberately (and despite sharing a name with generative AI, means almost the opposite — deterministic derivation, not probabilistic output). Generative models are also what make AI safe to add later: a model rich enough to generate from is rich enough to verify against, which is the stepping stone toward autonomous data engineering.
Data modeling isn’t going away — but how we do it is changing fast. AI and generative technologies shift the work from hand-drawing models to directing them: the architect specifies structure, rules and intent, and the machine generates, checks and evolves the model from that. The classic knowledge (Data Vault, dimensional, 3NF, conceptual/logical/physical) still matters — it’s what lets you judge what the AI produces — but it now sits alongside a new set of skills. The sharpest change is the role of the architect. The old output was a PowerPoint that often stayed unread; the new output is participation — supplying rules and decisions in a technical, machine-readable metadata format that actually drives the platform. The architect moves from documenting the system to being an active part of building it. That is a better job, and a more demanding one. Staying ahead means treating AI as a first-class participant in modeling, not a bolt-on: models rich enough to generate from, structure clean enough for AI to read, and a human who curates rather than rubber-stamps. Next-gen data modeling is data modeling with the leverage turned up — for the architects and engineers willing to change how they work.
There is a real and often-heard fear: that leaning on AI makes us dumber — that if the machine does the thinking, the skill atrophies. It is a fair worry, and pretending otherwise would be dishonest. A person who lets AI decide everything, unchecked, does get duller, the same way a muscle you never use gets weaker. The failure is real; the conclusion — avoid AI — is wrong. The answer is to stay the curator and the decision-maker. AI generates, proposes, drafts; you judge, correct and own the result. You keep asking why, you keep checking against the facts, you keep understanding how the answer was reached rather than just accepting it. Used that way, AI is an amplifier of judgment, not a substitute for it — it clears the grunt work so you can spend your attention on the hard, human parts: meaning, trade-offs, what actually matters. But staying sharp can’t be left to good intentions either — it needs a method, a structure, an agreement and the discipline to keep it up. To stay the boss of AI, we need a deliberate way of keeping up: what we must understand ourselves versus what we delegate, the spaced review that keeps the fundamentals fresh, the continuous learning that is now non-negotiable, and the shared commitment to actually do it. Without that structure the fear comes true; with it, we stay in control. And there is a neat symmetry in where the time comes from. We don’t need to find extra hours for learning — we reclaim them. The time now lost to inefficient meetings is exactly the time to invest in staying sharp. Cut the status theatre, structure the work so meetings are shorter and prepared, and put the recovered hours into the spaced review and practice that keep us ahead of the AI rather than behind it. Efficiency and learning are the same move: stop wasting the time, and spend it on getting sharper. This is exactly why continuous learning is non-negotiable now. The way to not get dumb from AI is to keep deliberately sharpening the thinking AI can’t do for you — through practice, feedback and spaced review. It is the human counterpart to vibe-driven AI: don’t let the AI run unspecified and ungated, and don’t let your own mind do the same. Stay sharp, and AI makes you stronger, not softer.
The template paradigm names a break most of the industry took without noticing. Early metadata-driven tooling generated code from modular, composable templates — small building blocks you could combine, override and reason about. Around 2013 the field drifted toward monolithic code generation: big, opaque generators that emit everything at once. It felt more powerful, and it quietly cost the two things that made the earlier approach durable — composability and maintainability. The lesson matters now because AI tempts everyone back into the same trap. A large model can generate a whole pipeline in one shot, which is exactly the monolith again, dressed in new clothes. When the output is one big block, you can’t reuse a piece, can’t override one rule, can’t trace why a line is there — you can only regenerate and hope. Modularity is what keeps generated systems changeable instead of disposable. Breakthrough Modeling takes the older lesson forward: generate from small, composable, named pieces — templates, rules, metadata — so the result stays inspectable and maintainable even when an AI is doing the assembling. Power without composability is just a bigger thing to throw away.
Vibe coding is building software fast with AI, led more by intent and feel than by a full up-front spec — describe what you want, let the AI generate it, shape it as you go. As a way to experiment it is a gift: you can turn an idea into something you can actually click on in an afternoon, see whether it holds, and learn far more from a working prototype than from a document about it. It is design thinking with the build step made cheap. A concrete, high-value use is building nice interfaces on top of the metadata. All the structure lives as metadata already; vibe coding lets you put a human-friendly face on it quickly — a cockpit, a browser, a dashboard — so stakeholders can see and use the model, not just read about it. The Beacon app is the example: friendly views (calendar, comms, practice, a backroom) vibe-coded over an underlying metadata and DuckDB core. The structure is the source of truth; the interface is a fast, disposable, delightful layer on top. And here is where the past pays off. The legacy applications — the mature enterprise modeling tools people spent years in — are full of genuinely good ideas: the useful views, the workflows, the small features that actually helped. We collect those good ideas and reuse them while vibe-coding, rebuilding just the fraction that earned its place, on our own metadata, without the lock-in. Don’t reinvent what already worked — harvest the best of the old tools and vibe-code it fresh. (No need to name names.) The one caution keeps it honest: vibe coding is for the sketchpad and the interface, not for the load-bearing pipeline. Don’t let the unspecified, ungated style become how you ship the data logic — that’s vibe-driven AI, the anti-pattern. Vibe-code the exploration and the experience; keep the core deterministic. Build loosely to learn, and let fail fast tell you which experiments deserve a proper foundation.
Once the gaps and inconsistencies are known, the pipeline closes them by asking the people who know. AI generates the questionnaires and conducts the Q&A — interviews with the relevant stakeholders, who are often already visible in the underlying documentation and the code check-ins. The domain experts fill the holes the documents left; the knowledge workers curate the answers rather than rubber-stamp them; and the results are written back as explanatory metadata. The dependency is the whole point: AI can only interview well once it has data. It needs the captured metadata, the extracted structure, and the detected gaps to know what to ask — and it needs the rules, skills and instructions (themselves metadata: markdown with frontmatter) to know how to ask it. Point an AI at a blank room and it asks generic questions; give it the structured context and it asks the sharp, specific ones that actually move the model forward. Interviews are not step one; they are what becomes possible after capture and gap-hunting. The output feeds the feedback loop: every answer sharpens the metadata, and every sharpened model makes the next interview better. This is understanding before modeling done at scale — AI-generated questions, human-curated answers, all captured as metadata.
The other half of capture is the code. An organization’s real logic lives in its SQL transformations, data models, table structures and reports — and, often, in transformations expressed in other formats too. This is input as much as any document is, and usually the most valuable and most tangled kind. So the pipeline analyzes it extensively: parse the SQL, reconstruct the models and mappings it implies, read the table structures and the reports that consume them, and convert all of it into one standard metadata format so it sits alongside everything else. Conversion is only the start. Raw code tells you what happens, not why — so on top of the extracted structure you add explanatory metadata: the intent behind a transformation, the business meaning of a column, the reason a filter exists. That is what turns a mechanical dump of SQL into something a human or an AI can actually reason about. Code check-ins also reveal who wrote what, which points at the stakeholders to interview later. This feeds straight into hunting gaps and inconsistencies and into transformation modeling — because once the buried SQL is metadata, its copy-pasted logic, tangled DQ fixes and missing layering become visible and fixable rather than hidden.
ADRs: durable records of architecture decisions capturing the context, the options considered and the reasoning — so the 'why' survives the people.
A company brain is only as valuable as it is reachable. Brain-access-via-MCP is the interface layer: MCP servers that sit on top of the structured metadata and let users — and other systems, and other AI tools — ask questions and get grounded answers fast. A business user types a plain question; an auditor asks for evidence; a downstream system requests a definition — and each gets an answer backed by the real structure, not a hallucination, because the MCP server queries the metadata mirror rather than guessing. This is what turns the brain from a store into a conversation. Because the knowledge is structured and versioned, the MCP layer can answer with provenance (“here’s the answer, and here’s the source”) and stay current as the metadata changes. And because MCP is a standard, the same brain is reachable by the whole ecosystem of tools without building a custom integration for each. It’s the delivery end of the pipeline and the practical form of the company brain: systems talking to systems in metadata, with humans able to join the conversation in plain language. Structure it once; let everyone and everything ask.
Feedback is the mechanism that stops the abstract thinkers from confidently building the wrong thing. Because a modeler can produce a clean, internally consistent model of a business they have half-understood, the only reliable correction is the people who actually own the domain — and the only way to get it is to ask for it constantly, not once at the end. A model that has been checked, challenged and corrected along the way is trustworthy for exactly that reason. For feedback to work it has to be cheap to give and easy to act on. That means the deliverable must be understandable on both sides — a visual, plain-language view the expert can react to, backed by the structured markdown, YAML and rules the machine acts on. If the domain expert can’t read it, they can’t correct it, and the feedback loop never closes. Structured well, feedback stops being an event and becomes a loop: capture the correction, fold it back into the source, regenerate everything downstream, show the new version, ask again. The same discipline that lets AI prepare a meeting also lets the meeting produce feedback that lands somewhere durable instead of evaporating in a hallway. Feedback is not politeness; it is how structure stays true to the world it models.
Platforms fragment the way disks do: not through any single bad decision, but through thousands of small writes in convenient places. A business rule copy-pasted into a second pipeline, a filter redefined in a dashboard, a lookup table cloned into another schema — each fragment works locally, and collectively they scatter the platform's logic across places no one can enumerate. Fragmentation is the default direction of entropy; defragmentation has to be a deliberate, continuous practice. The practice is a recurring sweep with a fixed destination: find logic that exists in more than one place — or in the wrong place — and consolidate it into the model, where it becomes one definition that generation propagates everywhere it is needed. The model is what makes defragmentation pay off; consolidating into yet another shared script just moves the fragment. Consolidating into metadata retires it: the duplicates become generated instances of a single source of truth. "Continuous" is the operative word. Done as a one-off cleanup project, defragmentation loses to entropy within a year; done as routine — a candidate list fed by lineage analysis and duplicate detection, worked down every sprint — it holds the line. It is the structural companion to watching for semantic drift: drift is meaning fragmenting, and this is the maintenance loop that pulls the pieces back together.
Continuous Metadata Integration (CMI) does for metadata what CI did for code: every change to a model, mapping, or rule enters a pipeline that validates it, checks its impact, regenerates the affected artifacts, and propagates the result — automatically, on every change, not in quarterly release ceremonies. A modeler merges a changed mapping; minutes later the platform has consistent SQL, tests, lineage, and documentation reflecting it, or a failed check explaining why not. The premise is metadata-as-code: once metadata is versioned in a repository, it can ride the same machinery as software — branches, pull requests, automated checks, promotion through environments. But CMI adds the metadata-specific stages that code CI does not have: semantic validation (is the model internally consistent?), impact analysis via lineage (which downstream artifacts does this change touch?), and regeneration as the deployment step, with the validation loop as the quality gate on what generation produced. CMI is what makes a model-driven platform feel alive rather than ceremonial. Without it, the model is technically the source of truth but practically a bottleneck — changes queue up for the next big regeneration, and people start patching downstream to avoid the wait, undoing the model-driven contract one hotfix at a time. With it, the shortest path to any change runs through the model, which is exactly where you want the habit to form.
An approach where explicit data contracts (schema, semantics, quality, SLAs) come first and the pipeline is generated and validated to satisfy them — producers and consumers agree on the contract, and the platform enforces it. A sibling paradigm to Model-Driven Data Engineering.
Design thinking brings a designer’s stance to modeling and to the way we work: start from the real human need, not the first technical solution; explore several options before committing; prototype and test rather than argue; and iterate on what you learn. It fits Breakthrough Modeling naturally — understand the essence first, question the default, and shape the answer around the people who’ll use it rather than around the tool that happens to be at hand. At the heart of it is experimentation. You don’t decide the perfect model or process in one shot; you try, observe, and refine. And this is where the new era changes everything: with AI you can experiment incredibly fast. Try a new modeling method, build a small application, generate a prototype, test it against real data — in hours, not months. The cost of a failed experiment used to be high enough that people avoided experimenting at all; now it’s cheap enough that not experimenting is the expensive choice. Vibe coding is a form of this experimentation: quickly building something rough with AI to see whether the idea holds, before committing to it properly. As a way to explore, it’s exactly right — the caution is only not to let that same unspecified, ungated style become how you ship (vibe-driven AI is the anti-pattern in production, not in the sketchpad). Experiment loosely; deliver deliberately. The other half of experimenting well is knowing when to stop. Don’t linger too long in something that, in the end, may not work. The whole point of fast, cheap experiments is that you can afford to abandon them — so set them up to fail fast, read the signal honestly, and move on without sunk-cost attachment. That speed turns design thinking from a workshop technique into a daily practice. Run the experiment, keep what works, discard what doesn’t, recombine the good parts — and let the fast feedback loop do the deciding. Design the way through by trying it, quickly, and learning in public.
When we design systems, we obsess over documenting the target — the data model, the pipelines, the platform. We rarely model and document the system we use to design it. That design system is real and it has structure: the features under discussion, the requirements behind them, the decisions and the reasons for them, the questions and their answers, the minutes of every meeting. Model your own systems, not only the client’s — capture that design process with the same discipline you apply to the thing you are building, and the how-we-decided stops living in people’s heads and email threads. The tooling to do this finally exists. AI turns speech into text automatically; meetings become searchable records; requirements and decisions can be captured as structured markdown and YAML the moment they are made. The raw capability is no longer the constraint. The new constraint is the opposite one: not being overwhelmed by the tools themselves. A new tool arrives every week, and it is easy to spend more energy chasing them than doing the work — or to burn tokens chatting with an AI that has no structured record behind it. So the discipline is to use the tools deliberately and keep the technology under control: one structured place where features, requirements, decisions and minutes land, versioned and readable by both people and AI. Control the tools, or the tools control you.
We — modelers, business analysts, knowledge workers, consultants — are the abstract thinkers in the room. That is exactly the risk: we can build a technically correct model of the wrong thing, cleanly and confidently, and never notice. So the discipline that keeps us honest is not more cleverness. It is transparency and feedback. Be open about the methodology and the deliverables, and ask for feedback constantly — not once at the end, but throughout, from the people who own the domain. That only works if the output can actually be understood on both sides of the table. Our deliverables have to be readable by AI and by mere mortals at the same time: the structured markdown, YAML and rules that a machine can act on, and the visual, plain-language version that a domain expert can correct. If only the abstract thinkers can read it, it is not a model of the business — it is a private artifact that happens to be about the business. The good news is that this is finally possible. With the tools now at our disposal — whiteboards, visualization, advanced generated diagrams, and AI — we can show the same structure in the register each audience needs, and keep every view in sync, if we work structured. Structure is what lets one source drive both the machine-readable model and the human-readable picture. Part of dogfooding is having a backroom — a place where your own machinery is on display, not hidden. The Beacon app makes this literal: alongside the everyday views there is a “Backroom” menu option that shows how the system itself is built and run. That is the stance to take with your own work: don’t just deliver the polished frontroom, expose the backroom too — the model, the rules, the decisions behind the deliverable — so others can inspect and correct it. Model your own systems, not only the client’s (document your own system). Dogfood your own method: make your thinking visible, invite the correction, and let the fact that others can check your work be the proof it is right.
The feedback loop is what makes the whole architecture improve itself. Nothing is right on day one; it converges on right. Generate, run, observe — the interview answers, the test results, the gaps that surfaced, the stakeholder’s “that’s not what I meant” — and feed what you learned back into the source. The next cycle starts from a better model, not from patched-over code. What’s specific here is what the feedback changes. It doesn’t just fix a value; on feedback we continuously adjust the metadata, the skills, the rules, the instructions, the structure — and the way of working itself. A rule that proved wrong becomes a corrected rule; a question that landed badly sharpens the instruction set; a recurring gap changes the structure; a friction in the process reshapes the method. Because all of these live as versioned metadata, feedback has a durable place to land instead of evaporating in a meeting. This is the engine behind the pipeline and the human counterpart of constant feedback: constant feedback is the stance (ask, always), the feedback loop is the mechanism (fold it back into the source, every time). Keep the loops short and always terminating in the metadata, and the company brain gets sharper with every pass.
Capturing the existing information is not the same as understanding it. Once documents and SQL are metadata, the pipeline does something most projects skip: it deliberately goes looking for what's wrong. AI helps the modeler understand the structure, then actively hunts for gaps (what should exist but doesn’t), inconsistencies (two sources that contradict each other), and hidden assumptions that no document states. At the same time it extracts the rules buried in the SQL and the docs, making implicit logic explicit. It has two moves, and it’s worth naming both. The gap hunt looks for what is missing — the entity with no owner, the rule referenced but never defined, the report with no source. The consistency sweep looks for what conflicts — two sources that disagree, a definition that drifts between documents, a transformation that contradicts the model it claims to implement. Gap hunt finds the holes; consistency sweep finds the collisions. The word deliberate matters. Neither gaps nor inconsistencies announce themselves; you have to search for them on purpose, because the comfortable path is to assume the captured material is coherent. It rarely is — that’s exactly why the knowledge was stuck. Surfacing the gaps and contradictions early, as data, is what turns a pile of captured documents into a map of what you actually know and what you still need to ask. This is a reality check on the captured knowledge itself, and it feeds directly into AI interviews: the gaps become the questions, the contradictions become the things to resolve with a stakeholder. Find the holes first, then fill them.
A data model is an organisation's "just to be clear" sentence. Everyone works on implicit assumptions — about what a customer is, when an order becomes an order, which definition of revenue counts. It works, right up until someone writes it down. Modelling is the moment you say: just to be clear, this is what we mean.
We still need the classics: data-modeling books, and the hard-won discussions on Data Vault, dimensional modeling, 3NF, and the conceptual, logical and physical layers. That knowledge doesn’t go away. But on its own it is no longer enough. The modeler’s job now spans communication, mental models, GTD, a growth mindset, and plain discipline — the skills that turn a good structural idea into something a team and a machine can both use. The reason is concrete: AI needs our output in a structured way. Effective habits — writing precise minutes of meeting, capturing interview results, keeping structured notes — are not bureaucratic overhead; they are the input format the AI runs on. A habit like reliably producing minutes is worth more than one more notation, because it is what makes the next step possible. (See The Power of Habit, Charles Duhigg — the discipline is a habit you install, not willpower you spend.) Once the record is structured, the payoff compounds. AI can prepare the meeting from precise minutes, interview results, structured notes, and even email and chat conversations — so when people finally sit down to talk, the conversation is focused and optimized, not spent re-establishing what was already decided. No waiting time. The meeting becomes the moment of decision, not the moment of catch-up. And feedback runs through all of it — the loop that keeps the structure honest.
A pattern is a proven shape you reach for instead of inventing one on the spot. Hub-link-satellite, a dimensional star, a bi-temporal history, a point-in-time table — these are solutions the field already worked out, tested against real problems. Naming them as patterns means you can recognize the situation, apply the known-good shape, and spend your thinking on what is genuinely new rather than re-deriving what a giant already solved. Patterns are how not reinventing the wheel becomes practical. The essentials distilled from Data Vault, Kimball, the Unified Star Schema and the rest live on as patterns you can pull off the shelf and compose. And when a pattern is captured as structured metadata rather than tribal knowledge, the AI can apply it too — generating the right shape consistently instead of guessing. Patterns pair with their siblings: rules say what must hold, skills say how to execute, and patterns say what shape to reach for. Compose from proven patterns and the model is sound by construction; ignore them and you rediscover old mistakes at your own expense.
A skill is a packaged unit of how to do one thing well — a naming convention, a review checklist, a modeling step, a generation routine — written down once, versioned, and callable. Instead of the knowledge living in one expert’s head and being re-explained every time, it becomes an asset the whole team, and the AI, can invoke. The value is consistency: the skill is applied the same way on Tuesday as it was last month, whoever (or whatever) runs it. Skills are what make AI dependable rather than improvisational. An AI given a vague prompt guesses; an AI given a skill — explicit steps, grounded in the structure — performs. This is the difference between hoping the model does the right thing and directing it. Skills turn scattered expertise into reusable behavior the platform can lean on. They compound with their siblings: rules say what must hold, patterns say what shape to reach for, and skills say how to get there — all as structured, versioned metadata the AI can read. A skill is the unit that lets good practice scale beyond the person who first figured it out.
Rules are the constraints and logic of a domain, written as data rather than buried in code: a validation rule, a business rule, a quality check, a governance policy. When a rule lives as structured metadata — an expression with a scope and a severity — it can be versioned, tested, enforced automatically, and asked about (“which entities does this rule touch?”). When it lives inside a stored procedure or someone’s head, it drifts and no one can see it. This is the new role of the architect made concrete: instead of writing a PowerPoint that stays unread, you supply the rules in a technical, machine-readable format that actually drives the platform. The rule is the deliverable, and it runs. Governance stops being a document people are supposed to follow and becomes something that travels with the data and holds automatically. Rules pair with their siblings: skills say how to do something, patterns say what shape to reach for, and rules say what must be true — all as versioned metadata the AI can read and enforce. Structure without rules is a shape with no guarantees; rules are what make the model trustworthy.
The first move of the pipeline is capture. A company’s knowledge already exists, but it’s scattered and locked in formats no machine can reason over: PowerPoint decks, Confluence pages, Word documents, spreadsheets, emails. Scrape-to-metadata is the disciplined act of pulling that information out and re-expressing it as metadata files — markdown with frontmatter — organized folder-per-unit, each with a folder note. What was a slide becomes a structured, addressable unit; what was tribal becomes queryable. The format matters. Markdown carries the human-readable text; the frontmatter carries the machine-readable metadata (type, source, owner, tags, relationships). Folder-per-unit with a folder note gives every idea a home you can link to, version and enrich — the same convention used for concepts and articles. It is deliberately boring and deliberately consistent, because consistency is what lets everything downstream — the mirror, the AI enrichment, the brain — actually work. This is where raw reality enters the pipeline and becomes metadata first. It pairs with analyzing the existing SQL: documents and code are both input, both converted to one standard metadata format before anything smart happens on top.
Engineering pipelines around the meaning of data, not only its movement: business semantics, definitions and relationships are first-class metadata that travel with the data and drive generation, validation and lineage. The counter to plumbing-only data engineering.
Semantic design is the discipline of defining what data means once, deliberately and top-down, so that every downstream layer — logical model, physical model, pipeline, report, AI-context — inherits that meaning instead of re-inventing it. Semantics start at the top: the definition, the relationship, the business rule are designed before the plumbing, and everything below refers back to them. The opposite is bottom-up accretion, where meaning is guessed and re-derived at every layer and drifts apart. Where Semantic Data Engineering is the runtime method (pipelines carry semantics) and Enterprise Ontology is the vocabulary (the agreed core concepts), Semantic Design is the design-time act that produces them: decide meaning once, reuse everywhere.
The DevOps discipline applied to semantics: definitions, models and contracts flow through the same version-control, review, automated-test and deploy pipeline as code, with semantic changes gated and validated. Meaning ships on a release train, not by hand. Pairs Metadata as Code with the Completeness Gate.
Spaced repetition is the learning methodology that answers the “one-off training doesn’t stick” problem head-on. The science is old and solid: we forget on a predictable curve, and the fix is to resurface a piece of knowledge just as it’s about to fade, at gradually increasing intervals. Each well-timed review resets the curve and moves the knowledge toward permanent. It is the difference between learning something once and actually retaining it. The BTM move is to build it into the system, not leave it to willpower. The key concepts, the glossary, the business model, the processes, the AI and prompt-engineering skills — all of it can be resurfaced on a schedule: small, spaced prompts that bring a concept back to attention at the right moment. Because this is all structured metadata already, the system knows what each person has seen and when, and AI can deliver the next review automatically. Learning becomes a background process, like a daily delivery, rather than an event on a calendar. Concretely, we continuously create and update flash cards — kept as metadata and linked to the things they teach: systems, sources, methodologies, AI concepts, prompt engineering, the data model, source-specific knowledge. Because the cards live in the same structured world as the model, they stay current as the model changes, and a card can point straight back to the concept or source it came from. And we keep training ourselves on them, to keep the team sharp. This is not overhead bolted onto the real work — it is part of the job of the knowledge worker. And because the results are stored, we can prove that everyone actually stays up to date. The spaced-repetition outcomes are kept as data, so staying sharp stops being a claim and becomes evidence — and that is the agreement everyone commits to and holds to, the same accountability and discipline we apply to everything else. It is a proven approach, which is why it is no longer optional. Keeping yourself and the team current in a field that changes this fast is not a personal hobby you may or may not pursue; it is a non-negotiable part of the role, the way writing minutes or versioning a decision is. We really do have to adapt to the new times — and prove that we have. Spaced repetition is continuous learning made mechanical, and it is naturally data-driven: track what was reviewed, what stuck, and where knowledge levels are slipping, then adjust the schedule. Structure the review, and retention stops being luck.
The collaborative layer is a place where stakeholders can look at the work while it is being done. The moment you model a new transformation, or pull in data from a new source, they can see the newly modeled or remodeled logic and the actual results — not a slide about it, the real thing. It turns modeling from a private activity into a shared, live workspace where the domain experts are present as you build, not summoned at the end. The reason this matters is timing. Today, discrepancies between what the business expected and what the model actually produces are often discovered late — when a release is finally pushed to an acceptance environment and someone notices the numbers are wrong. By then the misunderstanding is expensive to unwind. A collaborative layer lets those gaps surface much earlier: the stakeholder sees the transformation logic and the sample results as they take shape, and says “that’s not what I meant” on day one instead of week ten. It is where shift-left and feedback become concrete. The layer is the shared surface; shift-left is the principle of moving the check earlier; feedback is what flows through it. Build the collaborative layer and correction happens continuously, against real logic and real data, long before acceptance.
There is an order-of-operations error at the root of most data models, and Ronald Ross names it cleanly: data modeling is an engineering concern — it asks how records should be organized in a system — while conceptualization asks what the business actually means before any of that. Start at storage and you get a model of tables. Start at meaning and you get a model of the business that tables can then serve. Both produce a diagram; only one of them models the right thing. The sharpest way to see the difference is what data modeling routinely discards: the verbs. A table headed customer, order, location is silent on the relationships that give those nouns their point — does the customer place the order, cancel it, dispute it? Those verb-concepts are the chassis of meaning, and a model that drops them is communicatively impotent: the lines between boxes say "related, somehow" and leave both a human and a machine exactly where they started. Conceptualization keeps the verbs because the verbs are the meaning; the nouns are just where it attaches. This is also why a concept model is not a "conceptual data model." A conceptual data model is still a data model — entities and attributes, one abstraction level up from physical. A concept model operates in a different domain entirely: structured meaning, business vocabulary, and the disambiguation of what words mean, before the question of how to store any of it arises. Calling the first one "conceptual" is how organizations convince themselves they've done conceptualization when they've only done early-stage engineering. The payoff for getting the order right is durability. Concepts are more stable than any technology — they reflect how people think, not how machines store — which is why a well-built concept model outlives the platforms beneath it (one held up at ~90% accuracy twelve years on, while the databases under it were replaced twice). And it's the precondition for everything downstream: you cannot resolve data integration, or connect meaningfully to GenAI, using the same storage-first techniques that produced the fragmentation in the first place. Meaning first, then the model, then the machine — in that order, or you industrialize the wrong thing.
There are two ways a semantic layer dies, and they are opposites. Semantic drift is death by neglect: meaning erodes because no one governs it. The blackhole is death by accumulation: meaning is governed so thoroughly, so exhaustively, that the layer collapses under its own mass. Every entity, every dimension, every measure, every hierarchy, every glossary term, every access rule — each one added for a good reason, all of them together forming something no human can hold and no one dares prune. Past a certain weight it stops being a map of the business and becomes a second business to maintain. The mechanism is gravitational, which is why the metaphor fits. Each new definition pulls in the next: a metric needs a dimension, the dimension needs a hierarchy, the hierarchy needs a glossary entry, the entry needs an owner and an access rule. Nothing is wrong locally — every addition is defensible in isolation — but the layer's total mass grows faster than anyone's ability to reason across it. The tell is when people stop consulting the semantic layer and start rebuilding the definition they need in their own query, because finding the sanctioned one costs more than redefining it. At that point the layer has passed its event horizon: it still exists, it is still "governed", and it no longer governs anything. The trap is that the discipline which was supposed to prevent chaos becomes the thing that produces it. A team that has been burned by drift over-corrects toward completeness — model everything, define everything, govern everything — and mistakes volume for rigor. But a semantic layer's value is not proportional to how much it contains; it is proportional to how much of it people actually use and trust. A complete layer nobody can navigate is worth less than a small one everyone reaches for. The escape is not less governance but governed restraint: define only where meaning genuinely diverges from the obvious (you don't redefine every term, only the ones your business bends — the discipline of semantic design), start from the decisions that actually need it rather than from an urge to be exhaustive, and treat the layer's size as a cost to be justified, not a completeness score to be maximized. The question that keeps a semantic layer out of the blackhole is not "what else could we define?" but "what here has earned its place — and what is mass we are paying to carry?"
Nobody decided to couple these systems. Nothing prevented it. A report reads a source table directly because it was there. A pipeline joins another team's staging area because it was convenient that sprint. A dashboard depends on a column whose owner has never heard of the dashboard. The dependency is real, load-bearing, and undeclared — which means it is invisible until it breaks, and it breaks at the worst possible moment: when the owner of the upstream, acting entirely reasonably, changes something they had every right to change.
You know what was decided. You do not know why, or what else was considered. The record says "we use Kafka", "customer is defined as X", "the boundary sits here" — and nothing about the alternatives that lost, the constraints that applied, or who would need to be convinced to change it. Six months later the decision cannot be revisited, only reopened. And reopening means redoing the whole analysis, because the analysis was never the artifact — only its conclusion was.
Documentation written to be produced, not to be read. A wiki that satisfies an audit. A template filled in because the gate requires it. A landscape deck built for a steering committee and never opened again. It is complete, it is compliant, and everyone quietly knows to ask a person instead. The tell is the question "is it documented?" — which is about the existence of an artifact, not about whether anyone can act on it. Theatre passes that question every time.
One method, several incompatible orthodoxies. A discipline matures, gains a following, and then splits — into interpretations, certifications, tool-specific dialects and camps, each convinced it holds the correct reading. Practitioners stop asking what does this problem need and start asking what does my school permit. The cost is not the disagreement. It is that the disagreement becomes the work.
Understanding exists, but you have to get past someone to reach it. A person who must be asked. A licence that must be bought. A tool whose model is only legible inside the tool. A login that a contractor will not be given. The knowledge is not missing — that would be Tribal Knowledge. It is written down, structured, often excellent. It simply sits behind a gate, and the gate is not there because the knowledge is sensitive. It is there because of how the system happened to be built, or because someone benefits from the gate.
Every part is neat; the whole is unknowable. Each folder has a sensible name. Each repo has a README. Each schema follows the standard. Zoom out one level and nobody can tell you what the system is, what depends on what, or where a given fact actually comes from. This is the failure mode that hides best, because it passes every local inspection. Audits look at parts. Reviews look at parts. The problem only appears when someone new arrives, or when something has to change across a boundary — and then it appears all at once.
The model is treated as a picture. Boxes and lines, arranged until they look right, exported to PNG, pasted into a deck. It may be accurate. It is not usable: nothing is generated from it, nothing is validated against it, and the reasoning behind every choice lives outside it. The giveaway is what happens when reality changes. A drawing must be redrawn — by hand, by whoever owns the file, if they remember. A model gets updated once and everything downstream follows.
Every method has a technical cost that gets estimated, and a social cost that does not. The social cost is what the method does to the people around it: who can join the conversation, who goes quiet, and where the line falls between the initiated and everyone else. It is invisible in a tooling comparison and decisive in delivery.
The conceptual model isn't handed down; it's drawn together. Sitting with a stakeholder who has never read a line of SQL and agreeing what a 'customer' is and how it connects to an 'order' happens by sketching it on a shared surface — a whiteboard, a Miro board, an Excalidraw canvas — until the picture matches how they actually think. That co-drawing IS the modelling at the altitude where humans agree; it's not preparation for the real work, it's where buy-in and shared understanding are made. This is the personal/visual counterpart to MDDE's collaborative data modelling, and it connects to the wider field of visual facilitation — getting a room to a shared picture of a contested idea. The rule: the diagram is a conversation both sides own, never a technical artefact thrown over the wall.
The visual counterpart to 'a brain that publishes itself': because the structure is real, machine-readable metadata — entities, links, typed fields — you don't draw the picture and maintain it by hand, you generate it. A diagram derived from the source is always correct, because it's read straight from the truth, not transcribed. Change the underlying files and regenerate; the picture is current. The alternative is the stale architecture diagram rotting in a folder, three reorganisations out of date — accurate the day it was drawn, quietly lying ever since. Living diagrams stay honest on their own; stale diagrams are a maintenance debt you always lose. The vault is the source of its own diagrams. Honest limit: a generated diagram is only as good as the structure beneath it and only as readable as the human judgement on top — auto-drawing ten thousand notes gives a hairball, not clarity; knowing what to leave out is the taste a generator can't supply.
A Map of Content is a curated hub for one theme: it does not hold the content, it holds the routes to the content. One file that answers "what belongs to this subject, and how does it hang together" — maintained by a human, readable by a machine. Borrowed vocabulary (Nick Milo / LYT), used deliberately: the term is established in the PKM field, so it needs no explanation. What BTM adds is the second reader.
Every layer of the system carries a map of itself. Not one index at the top, not a wiki someone maintains on Fridays — a map at each level, generated or curated where that level lives, and readable by both a person and a machine. A Map of Content is the building block: one hub for one theme. Maps Everywhere is the principle: apply it at every layer, until there is nowhere in the system you can stand and not know where you are.
The personal counterpart — knowledge management for the AI era — lives at structurebeatsmagic.com.
Get in touch