Note · Enable
What determinism buys you in legacy documentation.
Generating documentation for an undocumented estate is an obvious use for a language model, and mostly the wrong one. The reason is not accuracy in the abstract. It is what happens to the reader who cannot tell which parts to check.
The ninety-five per cent problem
Suppose a tool documents a thousand programs and gets ninety-five per cent of the facts right. That sounds good. Now consider the maintainer holding the output. They do not know which fifty programs are wrong, and nothing in the document distinguishes a correct claim from a confident invention.
So they verify. And once they are verifying everything against the source, the documentation has saved them nothing — it has added a step. Documentation that is mostly right is not most of the way to useful. For this purpose it can be worse than none, because it invites a trust it has not earned.
What "deterministic" means here
That every fact in the output was read out of the source, and can be traced back to the line it was read from. Not summarized, not inferred from naming conventions, not reconstructed from what similar programs usually do. Read.
The practical test is provenance. If a document says a program writes to a dataset, it should be able to
say which line of which file says so. A fact with a file:line reference beside it can be
checked in seconds. A fact without one has to be re-derived from scratch.
What can be extracted, and what cannot
A surprising amount is mechanically derivable from COBOL and JCL:
- The program inventory, and which jobs and steps run each program
- Files each program declares, how it opens them, and the I/O verbs it actually issues
- What a DD name resolves to in each job — the same alias often points at different datasets
- The call graph, including targets that are called but missing from the corpus
- Record layouts to full depth, with picture clauses and usage
- Paragraphs nothing performs, which is dead code or a fall-through worth knowing about
What cannot be extracted is intent. No parser will tell you why a rule exists, which of two similar programs is authoritative, or that a step survives only because of an incident in 2003. That knowledge lives with people, and the case for doing the mechanical part quickly is that it leaves their limited time for the part only they can supply.
Gaps are output, not failure
The instinct when a reference cannot be resolved is to fill it in with the most likely answer. That is precisely the wrong move. A called program missing from the corpus, a DD name no job supplies, a copybook nobody has — these are the most valuable lines in the document. They are the map of what the organization has lost, and on an inherited estate that list is usually the first thing worth acting on.
Where a model does help
Not in extraction. But once the facts are established, a model is genuinely useful for turning them into
prose a maintainer will read, for proposing a plain-language name for WS-CD-SW-01, for drafting
the narrative that sits around a verified call graph. The distinction that matters is whether a statement is
derived or suggested — and whether the document makes that obvious to whoever is reading
it.
How to judge a tool that claims this
Three questions. Does every extracted fact carry a source reference you can follow? Are unresolved references reported as gaps rather than filled in? And is the extraction checked against fixtures, so that a change in the tool cannot silently change what it claims about your code?
Our own tool for this is public, so the answers are checkable rather than asserted — see estatemap, including the demo that runs the same extraction rules in your browser against source you paste in.
If this is your problem
Tell us what's stuck.
Describe what you are dealing with and we'll tell you honestly whether we're the right people. The first consultation is free.