schema

schema is the library for schema documents: YAML files that declare the shape of data once, so the types that hold it, the rules that normalize and validate it, and the version it is written under all come from one declaration. It parses a schema document, resolves what the document references, generates Python types from it, and classifies how a document changed between two versions.

It keeps those workflows apart from loading data at run time, which is the type library’s: schema reads and generates from a declaration, and type is what the generated classes run on.

Obtaining it

The library is named by the coordinate schema:schema and imported as schema. It depends on type:type, which arrives with it, as does everything else it needs. The meta-schema, the schema every schema document is itself a document of, is the exported schema schema:metaschema.

Public API

The root exports the model types so consumers holding schema data can stay on root-only imports; it does not re-export the behavioural APIs, which are imported from their submodules directly.

Responsibilities

Runtime model loading, normalization, and validation remain in the type package.

The schema document

A schema document is a YAML file with three top-level keys:

metadata:
  schemaVersion: '0.0.0-dev'
  version: '1.2.0'
  export: true
  name: inventory
  namespace: acme
  documentRoots:
  - inventory
externalSchemas:
  meta:
    id: schema:metaschema
    vendor: true
definitions:
  inventory:
    name: inventory
    fields:
      items:
        type:
          base: list
          item: item
        help: Everything the inventory holds
        default: []
  item:
    name: item
    fields:
      label:
        type: string
        help: What the item is called
        x-normalize:
        - trim
        x-validate:
        - non-empty
      count:
        type: integer
        help: How many of it are held
        optional: true
        default: 0

A definition may state examples: values of it as a document writes them, which documentation generated from the schema shows beside the definition. They are worth stating where a value is written in a form of its own – a definition declaring shorthand is the usual case – and each should load as a value of the definition.

A field is a mapping stating its type and its help. There is no short form: label: string had nowhere to carry a help, and is refused. The mapping may add optional, default, alias (the key a document writes the field under), format, secret, and x-normalize / x-validate. secret: true marks a string field holding something like a credential: it is generated as pydantic’s SecretStr, masked in every repr, str and dump and read with get_secret_value(), and a secret field stating a default, or one of another type, is refused when types are generated. A type is a built-in token (the keys of BUILT_IN_TYPES), the key of a definition, an alias.definitionKey reference, or a structured type: {base: list, item: ...}, {base: set, item: ...}, {base: dict, key: ..., item: ...}, {base: union, anyOf: [...]} or {base: enum, enumValues: [...]}.

A dict’s key is a type whose every value is one scalar in a document: a built-in that BUILT_IN_TYPES declares with key=True, listed as DICT_KEY_TYPES, or an enum, declared in place or as a definition. Left out, keys are strings. Any other key is refused – a built-in or a structured type where the document is parsed, a definition that is no enum where types are generated – because the loader runs a type’s rules on a dict’s values and never on its keys, and a key cannot be written as a mapping.

A set, set[T], is a selection: values of one type, each held once, in no order. Its item is held to what keys a dict, for the same reason – a set of lists or mappings has no one obvious equality – and a set states its item. A document writes a set as a sequence, since YAML has no set, and a value written twice is refused where the document loads, naming the field and the value. The model is generated as SetOf[T], which is frozenset[T] to a type checker; a loaded set iterates, and is written back, in a stable order: an enum’s values in the order the enum declares them, anything else sorted. A default is written as a sequence, as a list’s is. A field whose order means nothing and whose values may not repeat is a set rather than a list stating x-validate: [unique]: the property belongs to the type, so no field can forget it.

The sections below state each part of the format in turn.

External schema references

A schema document references definitions from other schema documents through externalSchemas aliases. A field can then use alias.definitionKey as its type:

externalSchemas:
  state:
    id: pkg_state:state
    target: pkg_state
definitions:
  addedPackage:
    fields:
      pkg:
        type: state.pkgConfigItem
        help: The package as the store's index records it

An alias entry locates the external document one of three ways:

How the referenced definitions reach the generated code is controlled by two flags on the alias entry, target and vendor:

Target imports (target)

target names the Python package whose generated __types__ package holds the referenced classes. Generated code imports them from {target}.__types__.{module}, which makes the reference a runtime dependency on the target package. Use this when the consumer genuinely depends on the producer at runtime anyway — for example when producer functions return instances of the referenced types, so consumer and producer must share one class identity.

Type-only imports (typeOnly: true)

With typeOnly, the generated import is emitted under an if TYPE_CHECKING: block: the referenced class is available to annotations and static analysis but never imported at runtime. This makes the reference a dev dependency rather than a runtime one. It is only valid where annotations stay unevaluated — the dataclass-rendered output used for generic definitions — because pydantic models resolve field annotations at model-build time. typeOnly still requires a target, and it cannot be combined with vendor.

Vendoring (vendor: true)

With vendor, the referenced definition — and the closure of local definitions it references — is generated into the consumer’s own __types__ package as sibling modules with relative imports. The consumer gets real runtime classes without importing the producer package at all: the reference contributes a build dependency (schema generation ordering) instead of a runtime dependency.

externalSchemas:
  state:
    id: pkg_state:state
    vendor: true

Vendoring is safe because the rule DSL is schema-transported: all un-prefixed x-normalize / x-validate rules are built into the type runtime, so vendored copies normalize and validate identically without any producer Python code (see the type package README). Current constraints:

Choosing a reference form

Field help

Every field states help:, a sentence saying what it is for:

increment:
  type: integer
  help: The commit-carried number, read from `git log`

The generator emits it as pydantic’s own description, so it is readable off model_fields at run time rather than only in the source. That is what it is for: a schema describing a boundary an LLM writes into can render the prompt asking for that boundary from the same declaration that validates the answer, instead of keeping the request in prose and the check in code and hoping they stay in step.

The meta-schema declares it required, with x-validate: [non-empty, says-more-than-name], so a document is refused where it is loaded if a field states no help, a blank one, or one that only says the field’s name back – updateLog helped as “Update log” – compared with case and punctuation ignored. Nothing else about the sentence is judged. says-more-than-name is one of type’s rules, and the first that is handed the name its model is held under: here, the field’s key.

Paths render POSIX

A field that declares posix-path – which type: path does for itself, through BUILT_IN_TYPES – gets a generated field_serializer rendering its value with forward slashes, element-wise across a list or dict.

The normalize rule alone is not enough: it runs on the way in and returns type(value)(normalized), so a value that arrived POSIX-shaped is still stored as the platform’s own Path and written back out with the platform’s own separator. That is cosmetic until a document is read somewhere else: a\b is two path segments on Windows and a single filename everywhere else, and a document that names files is meant to be readable from another machine.

No field can want both this and a declared format: a format is refused unless the field’s type is a schema reference, and a reference is never a path.

Type shorthand

A definition can declare a canonical string form that expands into its fields at load time:

coordinate:
  name: coordinate
  shorthand:
    pattern: '^(?P<namespace>[^:@]+):(?P<name>[^:@]+)@(?P<version>[^@]+)$'
    format: '{namespace}:{name}@{version.major}.{version.minor}.{version.patch}'
  fields:
    namespace:
      type: string
      help: The namespace the package is published under
    name:
      type: string
      help: The package's own name within its namespace
    version:
      type: meta.version
      help: The release of the package the coordinate names

The example is for the shape of a shorthand. A package coordinate itself is a built-in type and no document declares one: a field holding a reference is typed coordinate, pinnedCoordinate where it must state @<version>, or floatingCoordinate where it may not.

The generator emits the declaration as a __shorthand__ attribute on the model and the type runtime expands string values before validation (see the type package README). The field-spec authoring shorthand (fieldName: someRef for fieldName: {type: someRef}) is itself declared this way on the meta-schema’s fieldSpec definition.

Field-level formats

A field whose type is a schema reference can declare how its value renders on serialization:

latest:
  type: meta.version
  format: '{major}.{minor}'

The generator emits a pydantic field_serializer for the field, rendering the template from the value’s fields (placeholders are validated against the referenced definition at generation time). This is the per-context complement to a definition-level shorthand.format: parsing stays on the type (context-free), while rendering shape lives on the declaring field — which is why one document can persist latest: '0.0' while the same version type renders 0.0.0 in another. Serialized strings load back through the type’s shorthand, so the round-trip holds.

Canonical version type

The meta-schema defines version (major[.minor[.patch]], a pattern-only type shorthand; minor/patch default to 0) as the shared version vocabulary. schemaMetadata.schemaVersion and schemaMetadata.version are version values, and any other schema field holding a version references it by id: meta.version, through an externalSchemas alias naming schema:metaschema. Rendering a version back to a string is context-dependent — one document writes all three segments, another major.minor — which is why the type declares no format: the field that holds a version states the shape it renders in.

The two version fields

A version is defined as version and referenced as schemaVersion, and both ends are inherent rather than declared per schema.

metadata.version is the defining end: the version a schema defines itself at. A version names a release, and only an exported schema is released, so an exported schema states one and an internal schema states none – no coordinate reaches it and nothing stores it under a version, so one would name nothing. The parser refuses an exported schema without a version and an internal schema with one; the meta-schema declares the field optional only because one field cannot say that it follows another. The generator renders the version as the VERSION constant in __types__/constants.py, and renders no VERSION for an internal schema: there is no VERSION = None for a check to fall through on.

schemaVersion is the referencing end: the version of its schema that a document was written under. It is not a field an author declares. A schema names the definitions that describe stored documents in metadata.documentRoots, and the parser grants each of them a schemaVersion field of the meta-schema’s own version type, rendered {major}.{minor}.{patch}{marker}, required. A stored document is read back by releases after the one that wrote it, and its schemaVersion names a version of its schema, so a schema naming document roots is exported, and the parser refuses one that is not. A definition left out of documentRoots describes a component of a document and carries no version – otherwise every nested object would have one. A document root that declares schemaVersion itself is refused: the type, the format and the optionality are the meta-schema’s in every case, and a definition that may restate them is a definition that may differ from them.

The one hand-written referencing end is schemaMetadata.schemaVersion, in the meta-schema itself. A schema document is a document of the meta-schema, but nothing can grant a field to the document that defines what a grant is, so that one is declared where the grant is defined. Every schema document states it as the meta-schema’s own version renders, marker included – 0.0.0-dev while the meta-schema is marked – and the parser holds it there as type’s loader holds any document: the majors agree, the document’s minor is not newer, and at the same major.minor the markers are the same, since a document written under another state of the meta-schema is what the marker is there to say. The meta-schema itself states none: the version it was written under is its own version, so the field would only restate it, and the parser refuses it there. That is why the field is optional, and the parser requires it of every other schema document.

type’s loader pairs the two: a loaded document’s schemaVersion is checked against the version of the schema its root came from, refusing a major mismatch and refusing a loader older than the file. The generator writes that version on each document root’s class as __schema_version__, and the loader reads it before its package’s VERSION: a root vendored into another schema’s __types__ package sits beside that schema’s VERSION, which is not its own. Beside it the generator writes __schema__, the owning schema’s namespace:name, where the schema has one: several schemas state the same version, so a release that moves one of them restates the roots naming it and no root that only shares its number. Because the field is granted rather than optional, a document root that states no version is refused by validation rather than skipped. Both ends have to be there: a type carrying schemaVersion describes a document, so a type that names no VERSION to check one against is refused too, rather than the check passing by having nothing to say.

An application’s configuration is granted its feature flags

An application declares its feature flags in its *.app.yml and a client switches one in that application’s configuration, under features. Every application’s configuration takes that mapping and none of them spells it: a schema names the definitions that are an application’s configuration – the one an *.app.yml names as its configType – in metadata.configRoots, and the parser grants each a features field, a dict[string, boolean], optional and empty by default. A config root need not also be a document root – some applications’ configurations state no version, and making one a root would refuse every file of it that states none – and one that declares features itself is refused for the reason a document root declaring schemaVersion is. The generator renders each config root on the runtime’s AppConfig, which declares features once, beside the definition the root extends where it extends one, so no generated model writes the field either. Which names a file may switch is no business of the schema’s: it is the application’s declaration, checked where the framework composes the configuration. Naming a config root adds an optional field, so it is a minor change to the schema.

The grant is the meta-schema’s rather than a loader’s because every reader of the document has to agree on it. The framework composes the configuration, but the same file is loaded and saved on its own, a schema store validates it, and the generated documentation describes it: a loader that took features out before the type saw it would leave each of the others refusing the field, and a save rendered from the model would drop every flag the file switched.

Why a document root obliges an externalSchemas entry

A schema that grants the field has to reach the meta-schema’s version type, so it names the meta-schema in its externalSchemas, and a schema that names a document root without that entry is refused with the coordinate prefix the grant looked for. The grant reads the entry rather than inventing one, because how a repository resolves and vendors the meta-schema is its own wiring:

externalSchemas:
  meta:
    id: schema:metaschema
    vendor: true

This is deliberate, and it costs a schema that was not already a meta-schema consumer a reference of its own: the store it resolves against has to hold schema:metaschema, all to produce an eleven-line version.py that imports nothing but TypeBase.

The alternative is a built-in version token in BUILT_IN_TYPES, which every schema could spell with no reference at all. It is not taken:

So a repository that names a document root takes the meta-schema as a vendored schema, the same way it takes any other schema whose definition it holds.

Type-level rules

A definition may declare x-normalize / x-validate of its own, beside its shorthand and toString, and every field declared as that type gets them. Built-in types carry rules the same way, from their own declaration. BUILT_IN_TYPES is the one table of schema type tokens: each entry states the annotation it renders as, whether that annotation names a type runtime model, and the rules every field of it gets. path renders as Path and normalizes with posix-path, so a schema that writes type: path needs no x-normalize to get path handling.

Membership in that map is also the reference test — a type: that is not a key in it names a definition — so the map answers what a built-in renders as and whether a token is one at all. It is deliberately the only such table.

A field’s x-normalize / x-validate hold the union of its type’s rules and its own, type first, so the generated field carries one list and the runtime has one place to look. The combination rule and the shapes it refuses are stated in the rule DSL section of the type package README.

Field rule vocabulary validation

parse_schema_document validates x-normalize / x-validate entries against the type runtime’s rule vocabulary (built-in rules, parameterized forms such as pattern:<regex> / sort-by:<key> / unique-tree:<key>/<children>, and explicitly package-local local:<rule> hooks). A definition’s own rules are checked by the same judgment as a field’s. Unknown un-prefixed rules are rejected at parse time, before generation. The vocabulary itself is documented in the type package README.

Named enums

A set of values a single field owns is declared on that field, as type: {base: enum, enumValues: [...]}, and generates a Literal named after the definition and the field together. A set that more than one field holds is declared once instead, as a definition of its own:

definitions:
  moduleKind:
    name: module-kind
    enum:
    - library
    - executable
  module:
    name: module
    fields:
      kind:
        type: moduleKind
        help: What the module is built as
      fallbackKind:
        type: moduleKind
        help: What it is built as where its own kind cannot be
        optional: true

The definition generates one module holding one alias — ModuleKind: TypeAlias = Literal['library', 'executable'] — and every field typed by it renders that name and imports it. Two fields holding the set are then one type rather than two that nothing keeps equal, and a value added to the set reaches both by reaching the declaration.

An enum definition is its values: it has no model behind it, so it may not also declare fields, anyOf, extends, generics, protocol, shorthand or toString, and nothing may extend it — a definition declaring one of those beside enum is two definitions sharing a name, and only one of them could be generated. The values themselves must be non-empty and distinct: a reference to an enum of nothing annotates nothing, and a value written twice makes the declaration disagree with itself about its own members.

A field reaches an enum declared elsewhere the same way it reaches any other definition — an externalSchemas alias, by coordinate or by path, with the alias-qualified name as the field’s type:

externalSchemas:
  kinds:
    id: acme:kinds
    target: acme_kinds
definitions:
  module:
    name: module
    fields:
      kind:
        type: kinds.moduleKind
        help: What the module is built as

The generated import is the producer’s own — from acme_kinds.__types__.module_kind import ModuleKind — so the consumer’s field is annotated by the declaration rather than by a copy of it. vendor and typeOnly mean here what they mean for any other reference.

Enum defaults

A field whose own type is an enum may only default to one of that enum’s values, whether the values are written on the field as enumValues or held by an enum definition the field references. An inline enum is refused at parse time by parse_schema_document, where the values sit beside the default; a referenced one is refused during generation, which is where the reference resolves and the set becomes visible. Either way the document is where the contradiction is legible: the renderer writes a default through verbatim, so default: invalid-kind becomes kind: ModuleKind = 'invalid-kind' — a value outside the Literal annotating it — and pydantic does not validate defaults, so a model built without that field carries the invalid value as though it had been declared. The check is deliberately limited to a field typed as the enum directly; a default on a list or dict of enums is a container-shaped question rather than a membership one, and is left alone.

Defaults on a field typed by an alias

A definition that declares anyOf is a bare type rather than a model: it renders as ReleaseStage: TypeAlias = Literal['dev', 'prod'], and a field referencing it as type: config.releaseStage gets that alias as its annotation. A default on such a field is the value itself — stage: ReleaseStage = 'dev' — where a default on a model-typed field is a shorthand string validated into an instance at class build time (Coordinate.model_validate('a:b')). The two cannot share a rendering: an alias has no model_validate to call, so the renderer tells the two references apart.

Two refusals come with it, both at generation, where the alias’s members are in hand and the parser’s document-local view is not:

An empty default (default: {}) is refused on an alias-typed field for the same reason: it builds the referenced model with its own defaults, and an alias is a bare type with nothing to build. A model-typed field keeps it, and still renders origin: Location = Location().

Classifying a change

schema.compatibility.classify_schema_change(released=..., current=...) says which version segment a schema document’s change earns – major, minor or patch – and why. It takes two parsed documents: the last released version and the one in the working tree. Where they come from is the caller’s: the released one is read from wherever every released version is kept, and never from a store a document still in development is registered into, where a marked document sits at its unmarked address and so stands in for the release it marks.

The answer is a SchemaChangeClassification: the segment, and every SchemaChange that contributed to it. Each change carries its own segment, a kind a consumer can match on, where it is (definition, field, or the externalSchemas alias as reference, rendered together as subject), and a reason a reader can argue with. The document’s segment is the highest any change earns. Where a segment is the rule’s choice rather than a consequence of the change, the change says so with judgement, and judgements lists every one, so a report can show a reader which parts of a bump are argued rather than derived.

The rules are also reachable piecemeal, for a document that is not a schema but declares its parts in the meta-schema’s terms. classify_field_changes(definition=..., released=..., current=...) holds a mapping of fieldSpecs to the field rows below, and classify_reference_changes(released=..., current=...) holds externalSchemas entries keyed by alias to the reference rows. A document whose inputs are fieldSpecs and whose schemas are externalSchemas entries is classified through these rather than by restating the rules.

What a segment promises is about documents and about the code generated for them. A patch changes neither what validates nor what a consumer’s code can name. A minor lets through a document the release refused, or gives a consumer something new to name, so a reader still on the release may refuse what a writer on the new version writes. A major refuses a document the release accepted, or takes away something a consumer’s code named.

Change Segment Why
Field removed major A document carrying it, and code reading it, both break.
Field renamed (one removed, one added with the same declaration) major A document writing the old name, and code reading it, both break. Reported as one rename rather than a removal and an addition.
Field retyped major A value of the old type need not be one of the new.
Optional field added minor Every document that validated still does; one using the field is refused by the release.
Required field added major Every existing document without it stops validating.
Field made required / made optional major / minor The first refuses a document leaving it out; the second lets one through.
Field default changed major, judgement Every document leaving the field out loads with a different value. Whether that breaks anything depends on what the value means, which the schema does not say, so the rule assumes it does.
Field alias changed major The key a document writes the field under changes.
Field format changed minor, judgement What loads is unchanged, but a document written in the new rendering is read by the release only if the type’s shorthand parses it.
Enum value added / removed minor / major Holds for a field’s inline enum and for a named enum definition alike.
Enum values reordered patch The set is unchanged.
Union variant added / removed minor / major As for enum values, for a field’s union and a union definition.
Union variants reordered patch, judgement Validation tries variants in order, so a value more than one accepts may load as a different one.
Definition added / removed minor / major A new type to name; a reference to the removed one stops resolving, here or through an alias elsewhere.
Definition renamed (key), or its name changed major A reference by the old key stops resolving; the generated module is named from name.
Definition changes kind (model, enum, union) major No value of the one is a value of the other. Nothing else about it is compared.
protocol, extends or generics changed major The generated class is a different kind of class, has a different base and inherited fields, or is subscripted differently.
Shorthand added / removed minor / major A string form the release refuses is accepted; one it accepted is refused.
Shorthand pattern changed major, judgement Whether the new pattern accepts every string the old one did is not decided here, so the rule assumes it does not.
Shorthand format changed minor, judgement As for a field’s format.
toString changed patch It renders a value for a reader; no document is read through it.
x-validate rule added / removed major / minor Tightened: a value that validated may be refused. Loosened: a value only the new version accepts is refused by the release.
x-validate rule re-parameterized (pattern:<regex> to pattern:<other>) major, judgement One rule changing, not one removed and another added. Whether two regular expressions accept the same values is not decided here, so the rule assumes the new one is tighter. local: hooks are distinct rules, never one re-parameterized.
x-normalize rule added, removed or re-parameterized minor, judgement It changes the value a document loads as rather than whether it loads. The rule counts that as new behaviour rather than a break.
help changed patch It describes and constrains nothing.
Nothing the parsed documents carry differs patch What changed is a comment, whitespace or the order entries are written in, none of which parsing keeps.
metadata.version not compared It is what the classification decides, and the working tree’s is marked.
metadata.schemaVersion changed patch The document was written under another meta-schema version; what it declares is compared on its own.
metadata.namespace or metadata.name changed major Every coordinate naming the released address stops resolving.
metadata.export turned off / on major / minor A coordinate reference stops resolving; the schema becomes resolvable by one.
Document root named / unnamed major Naming one grants a required schemaVersion that no stored document has; unnaming one removes it and its version check. A parsed document already carries the grant, so the field’s own addition or removal is reported beside it.
externalSchemas alias added / removed patch An alias on its own changes nothing. A field that starts or stops using it is retyped, and classified as a field.
Reference by coordinate moves forward the referenced schema’s own segment The referenced schema is versioned by this same rule, so the segment its version moved by is its classification, and every field typed through the alias carries it. The referenced documents are not needed: the version says it.
Reference by coordinate moves back major, judgement It undoes whatever the later version changed, which the references alone do not say.
Reference names a different schema major Every field typed through the alias is retyped.
Reference by coordinate floats on both sides (ns:name, ns:name@latest) none A development reference floats, and the two spellings are one reference: its text did not move, so there is no segment to inherit. How far what it resolves to moved is the referenced schema’s own classification; a caller that knows what each side resolved to pins both before classifying.
Reference moves between floating and pinned, or is by path or uri and moves major, judgement One side names no version, so how far what it names moved cannot be read off the reference, and the rule assumes the worst.
Reference’s target, vendor, typeOnly or name changed patch, judgement Documents validate as before, but the generated code takes the referenced classes from elsewhere, which a consumer comparing class identity would see.

The public import surface

The package root and the four submodules under “Public API” are the whole surface. schema/__init__.py declares it as __public_submodules__ = ("compatibility", "generator", "parser", "resolver"): the root may be imported, those four submodules may be imported, and everything else in the package is internal and may change without a version saying so.

A consumer that needs one more model type needs it exported from the root, which leaves the surface flat.