manifest
manifest.yml is the file at a repository’s root that says what the repository holds: what it is called, what version it is at, which modules it contains, where each is rooted, and what each depends on. dev reads it for everything it does by module – building, vendoring, selecting the tests a change reaches, and checking that what the code imports is what the manifest declares – so a manifest drawn at the right grain is where most of the tools’ value comes from. This package, manifest:manifest, owns the document’s schema, manifest:manifest, and reads a manifest into typed, normalized, validated models, so every program reads the same declaration the same way.
The library validates the document’s structure. What the declared dependency graph means – cycles, layering, which module may reach which – is decided by the program that reads the graph, and dev’s checks are the ones described below.
Writing one
This takes a repository with no manifest to one whose modules and dependencies dev accepts. It assumes the repository’s adapter is chosen and its adapter catalog wired, which repo-adapter-api’s “Setting one up” does; its step 5 writes a first manifest, and this walkthrough is that step done in full. Every command is run from the repository’s root. Where a command refuses, its message names a code and the field or edge at fault; repair what it names and run the step again.
-
Start the file. Create
manifest.ymlwith the repository’s own facts:schemaVersion: '0.0.0-dev' project: acme version: 0.1.0 repoAdapter: language:python catalogs: repo-adapter: path: repo-adapter pkg: repo-adapter modules: []schemaVersionis the version of the manifest’s schema these tools read,manifest.SCHEMA_VERSION, written whole and quoted. A file stating another is refused as it loads, asMANIFEST_VALIDATION_FAILED, naming the version to write.projectis the repository’s name. Write it in lowercase letters, digits and hyphens: it is also the namespace every module is published under, the first part of each module’s coordinate<namespace>:<name>. Statenamespacebeside it only where the packages are to be published under another name than the repository’s.versionismajor.minor.patch, the version the repository is at, and every module’s. Take it from the repository’s packaging metadata, respelled where that spells it otherwise (2.3.0.devis written2.3.0-dev). Nothing reads one from the other, so a release from here on changes the version in both files.repoAdapteris the adapter’s coordinate, andcatalogsthe catalog it resolves against, both as the adapter’s setup gave them.
Done when
dev module listanswersProject: acmeandTargets (0). A field the schema does not hold, or one missing or misspelt, is refused asMANIFEST_VALIDATION_FAILED, naming the field:(project): Field required,(projct): Extra inputs are not permitted,(version): Input should be a valid dictionary or instance of Versionfor a version that is not three numbers. -
List the candidates. Write down every package the repository holds that something imports or runs: each directory the language treats as an importable package at the top of the source tree, each application with an entry point, and each library of test helpers that tests import. Leave out tests, generated output and scripts nothing imports. An adapter document the repository keeps for itself is already a module, declared by the adapter’s setup with its
runtimeCoordinate; keep that entry as it is. Done when you have the list and, for each, the directory it is rooted at and the file a consumer enters it by –__init__.pyfor a Python package, the file that starts it for an application. -
Declare one module per candidate. Each is an entry under
modules:modules: - name: acme-cli rootDir: src/acme_cli entryPoint: __main__.py repoAdapter: language:python kind: executable - name: acme-core rootDir: src/acme_core entryPoint: __init__.py repoAdapter: language:python kind: librarynameis the module’s name, the second part of its coordinate: lowercase and hyphenated, whatever the directory is called. Prefix each with the project’s name (acme-core, notcore): a name has to be unique across every manifest whose modules are ever read together, and the prefix is what keeps it so.rootDiris the directory, relative to the repository’s root, spelled as it is on disk.entryPointis relative torootDir.repoAdapteris the top-level one again, or another where the module is in another language.kindislibraryfor a package that is imported,executablefor an application that is run, andtest-libraryfor helpers only tests import.- Keep
modulessorted byname; the file is read back sorted either way.
Done when
dev module listlists every module underTargets (<count>), one line each, as- acme-core: src/acme_core/__init__.py (language:python, library). A name declared twice is refused as it loads, asMANIFEST_VALIDATION_FAILEDwithduplicate 'name' value 'acme-core' in tree. -
Choose the granularity. A module is a unit something answers for: it has dependencies of its own, a place in the graph, and tests a change to it selects. Draw a module boundary wherever one of these holds, and nowhere else:
- The parts have different dependencies. A package whose half imports a database driver and half does not is two modules, so the half that does not is not rebuilt and retested when the driver moves.
- The parts are consumed or released separately. Something imports one part without the other: a client library beside the service that implements it, a data model several applications share.
- The parts have different audiences. One part is released to clients and the other kept in-house (step 7):
audienceis per module, so a boundary is needed for the two to differ. - One part is an application and another is what other code imports of it. An application is a leaf: nothing may depend on an
executablemodule. Code another module imports goes in alibraryof its own, and where that library is the application’s generated API, it statesapiOfnaming the application.
Do not split where none of these holds:
- No consumer would take the part alone. Two directories that always change together and are always imported together are one unit with two folders.
- The split only mirrors folders. A directory is not a reason. Declaring each one, or wrapping each loose file in a directory to declare it, produces a graph as large as the tree that says nothing the tree did not.
- The code cannot have the boundary. Two files that import each other are two halves of one thing; a declared graph refuses the cycle between them, so they stay one module.
Decide each candidate by these tests: keep it whole, merge it with another, or mark it to be split in step 6. A first manifest of one module per package, as step 3 wrote, is the usual answer, and a coarse manifest that passes the checks is a better start than a fine one that does not; a finer division can always be made later. Done when every candidate is decided; step 5 ends by checking the decision against the graph the code actually has.
-
Declare the dependencies, and let detection say what was missed. Each module lists what it depends on, in four lists:
dependencies– what it imports at run time.devDependencies– what only static analysis needs: an import made for type annotations alone, in Python one underif TYPE_CHECKING:.testDependencies– what its tests import that the module itself does not.buildDependencies– what has to be built or generated before it, such as a schema its types are generated from.
An entry is the
nameof a module this manifest declares, or the coordinate<namespace>:<name>of a package another repository declares. Third-party packages from the language’s own index are not listed: they are the packaging metadata’s.Write what you know, then run
dev module dependency detect --validate. It reads what each module’s source actually imports and compares it with what the manifest declares:[FAIL] Detected dependencies differ from declared dependencies Modules analyzed: 2/2 Detected dependencies: - acme-cli -> acme-core Undeclared detected dependencies: - acme-cli -> acme-core (runtime)Each line under
Undeclared detected dependencies:is one edge to add, in the list its bracket names –(runtime)isdependencies,(dev)isdevDependenciesand(test)istestDependencies, so hereacme-coregoes underacme-cli’sdependencies– and each underUnused declared dependencies:is one to remove. Imports in test files are read as(test)edges, and imports of third-party packages are not reported. The run fails asDETECTED_DEPENDENCY_VALIDATION_FAILEDuntil the two agree. The manifest is what is corrected, never the code: an edge you did not expect is the code’s real shape, and a boundary it crosses is a granularity question for step 4. Done when it prints[PASS] Detected dependencies match declared dependencies.Then run
dev module dependency list --validate, which judges the declared graph itself, refusing asDEPENDENCY_VALIDATION_FAILEDand listing each: a name nothing declares (Unknown dependencies:), a cycle (Dependency cycles found:), and a dependency on an application or on a test library from non-test code. Done when it prints[PASS] Declared dependencies valid.Last, look at the result.
dev module graphlists the edges between modules, one per line, asacme-cli -> acme-core [runtime] (declared+detected). Draw them, and ask whether the drawing is how the people who work on the repository would draw the system. A package that is really three subsystems becomes three modules (step 6); three packages nobody thinks of apart become one, by step 4’s tests. Go round steps 4 and 5 again until the drawing matches and both validators pass. -
Nest the modules a package is made of, where step 4 split one. A module inside another is declared under its parent’s
submodules, with arootDirinside the parent’s:- name: acme-core rootDir: src/acme_core entryPoint: __init__.py repoAdapter: language:python kind: library submodules: - name: acme-core-pricing rootDir: src/acme_core/pricing.py entryPoint: pricing.py repoAdapter: language:python kind: libraryA single file can be a module:
rootDirnames the file andentryPointis that file’s name. A nested module travels inside its parent’s package and is never obtained alone. The parent depends on what it holds without saying so, so its lists never name its own submodules; a nested module’s own lists name what it imports, siblings included. It reaches another top-level module of the same manifest only through its parent, which declares that dependency and hands it down – in Python, through the parent’ssubmodule_api.py. A package of another repository it declares and imports itself.dev module dependency list --validaterefuses a module rooted inside another but declared beside it (Nested modules must be declared as submodules instead of independent modules:), a parent listing its own submodule (Explicit parent-to-submodule dependency declarations are not allowed:), a nested module’srootDiroutside its parent’s, and a nested module depending directly on another top-level module (Submodules must not directly depend on modules outside their top-level module:). A nested name used twice anywhere in the manifest is refused as it loads, and asDUPLICATE_MODULE_NAMESacross manifests read together; prefixing each nested name with its parent’s (acme-core-pricing) keeps it unique.A nested module declared beside its parent instead fails both validators:
dev module dependency list --validateas above, anddev module dependency detect --validatewith the parent’s import of it undeclared (acme-core -> acme-core-pricing (runtime)), since the parent’s implicit edge holds only for its own submodules. Declared as a submodule, the edge is read as the parent’s own:dev module graphshows it asacme-core -> acme-core-pricing [runtime] (detected), detected and never declared, which is right.Then run
dev module export list --validate. It reads each parent’s export surface, the file the adapter names for one – in Pythonexternal_api.pyat the module’s root, its exports listed in__all__– and a parent keeping none is not asked anything. A name the surface imports from a nested module and re-exports straight through is a second door onto that module, and is refused asPARENT_EXPORT_VALIDATION_FAILED, each asacme-core/external_api.py:1 [price] imported from acme-core-pricing and re-exported. Either the nested module is the consumer’s to import, and the parent stops re-exporting it, or the parent’s surface is the consumer’s door and the boundary is in the wrong place: the part is no unit of its own and is left undeclared. Done when steps 5 and this one pass. -
Say who each module is released to.
audienceon a top-level module isinternalwhere it is not stated: reached only by the publisher’s own repositories, which is right for every module until someone outside is meant to obtain it.publicreleases it to clients and documents it for them;hiddenreleases it the same way and documents it nowhere, for a library a public one needs that nobody should be pointed at. What a released library depends on at run and build time is released with it, sodev module dependency list --validaterefuses a public or hidden library depending on an internal module (A library released to a client, public or hidden, must not depend on an internal module:), and an application declared hidden (An application is public or internal, never hidden:). An application may be public whatever its libraries are, since what it is built from is inside it. A public module’sREADME.md, at itsrootDir, is its client documentation, so a module declared public owes one: it travels in the module’s archive and is published as written. Done when every module that leaves the repository says so, and both validators pass. -
Check the whole file. Each of these reads the manifest whole and exits non-zero on what it refuses; the four that validate are checks of the gate, which a plain
dev gateleaves to CI anddev gate --allruns:dev module list dev module dependency list --validate dev module dependency detect --validate dev module export list --validate dev project list --validatedev project list --validatereads the project’s line,acme 0.1.0 (acme@0.1.0) untaggedbefore any release, and holds the version to the repository’s release tags,<project>@<version>: once a release is tagged, content that moves after it with the version unchanged is refused asPROJECT_UNMARKEDuntil the version is marked-dev, and before any release a dependency pinned at a version is refused asDEVELOPMENT_REFERENCE_PINNED, since a development reference floats – drop the@<version>and let a release write one. Done when all five pass. Commit the manifest; from here on a new package is declared in the change that adds it.
Reference
The manifest
schemaVersion– the version of the manifest schema the file is written under, written whole:manifest.SCHEMA_VERSION. Required.project– the repository’s name, and the namespace its modules publish under wherenamespaceis absent. Required.version– the version the repository and every module it declares are at,major.minor.patch, with a-devsuffix while the content has moved since that version was released. Required.namespace– the namespace the modules are published and identified under, where it is notproject. A namespace is a fact about a publisher, so it is stated once here rather than per module.repoAdapter– the coordinate of the adapter for the repository’s language and toolchain,<category>:<label>with an optional@<version>, aslanguage:python. A<catalog>|prefix names which ofcatalogsit resolves against, required where several are ofpkgrepo-adapter. Required.catalogs– the catalogs the repository reads or publishes through, by label: each apathinside the repository and thepkgits content is (repo-adapter,build,schema). How a catalog is wired to other stores is.pkgmgr/config.yml’s, not the manifest’s.description– optional prose about the repository.releaseStage–devorprod: whether something depends on the repository in production, which sets whether a breaking change is made outright (dev) or behind a flag or a contract-version bump (prod). No default; absent, a reader uses its own configuration.modules– the modules, each amanifestModule, kept sorted byname. A name is unique across every module and submodule.
A module
name– the module’s name, the second part of its coordinate, and what a sibling names it by in a dependency list. Required.rootDir– where it is rooted, relative to the repository root and inside it: a directory, or a single file that is then its ownentryPoint. Required.entryPoint– the file a consumer enters it by, relative torootDir, and for anexecutablethe file its build starts from. Required.repoAdapter– the module’s adapter, spelled as the manifest’s. Required.kind–library,test-libraryorexecutablefor a package this manifest declares;repositoryfor a checkout atrootDirwhose own manifest declares its packages, which become this module’s submodules and which this repository’s build builds. Arepositorymodule declares nosubmodulesof its own. Required.audience–public,hiddenorinternal(the default), amoduleAudience. Read on a top-level module only.hiddenis a library’s, never an application’s.dependencies,devDependencies,testDependencies,buildDependencies– each a set ofdependencyReferenceentries: a module of this manifest byname, or another repository’s package as<namespace>:<name>, with@<major>.<minor>.<patch>or@latestafter it once that project has been released. A version after a bare name, or anything else, is refused as the manifest loads. Each list is written in any order and read back sorted, holds no entry twice, and defaults to empty.submodules– the modules nested inside this one, each declared as a module is, kept sorted byname.apiOf– on alibrarythat is an application’s generated API, thenameof thatexecutablemodule. It is the one edge onto an application a manifest allows, and building it publishes the application’s source beside it.vendorPath– where a consumer that obtains the module unpacks it, relative to the consumer’s root. Absent, a package a language imports is unpacked underlib/at the basename of itsrootDir.runtimeCoordinate– says the module is resolved at run time by this coordinate rather than obtained as a build dependency; a repo adapter document is declared this way. The value may carry a<catalog>|prefix, asrepoAdaptermay.skipBuild– on arepositorymodule, leave that checkout out of this repository’s build. Defaultfalse.parentModule– filled in by the tools for a nested module, from where it sits; never written.
The fields’ own help in manifest:manifest is the full statement of each, and a tool reading the document shows that help. Strings are trimmed, paths normalized, repoAdapter and runtimeCoordinate lower-cased and held to the coordinate pattern, and module lists sorted, by the rules the schema declares in type’s rule vocabulary.
Reading a manifest from code
The library is named by the coordinate manifest:manifest and imported as manifest. It depends on type:type, which arrives with it. Everything is imported from the package root, which is the whole surface:
manifest.Manifest– the root model;manifest.ManifestModule, one module;manifest.CatalogDefinition, one entry ofcatalogs.manifest.SCHEMA_VERSION– the version amanifest.ymlstates as itsschemaVersion.manifest.MANIFEST_DEFAULT– the file name,manifest.yml.manifest.ModuleAudience– the type ofaudience; a schema elsewhere references it by coordinate asmanifest.moduleAudience.manifest.EXTERNAL_AUDIENCES,manifest.DOCUMENTED_AUDIENCEandmanifest.HIDDEN_AUDIENCE– the audiences released to a client (publicandhidden), the one documented for a client (public), and the one released undocumented (hidden), so a reader ofaudiencesays which question it asks and spells none of the values.manifest.manifest_namespace(manifest=...)– the namespace one manifest’s modules publish under.manifest.module_coordinate(manifest=..., module=..., version=...)– one module’s coordinate,<namespace>:<name>@<version>.
The models are loaded with type’s loaders, which run the normalization and validation the schema declares:
from pathlib import Path
from manifest import MANIFEST_DEFAULT, Manifest, module_coordinate
from type import load_data_file
loaded = load_data_file(schema_type=Manifest, path=Path(MANIFEST_DEFAULT))
for module in loaded.modules:
print(module_coordinate(manifest=loaded, module=module, version="1.0.0"))