Canyonland Technologies Inc.

Note · Extract

What a mainframe data extract actually involves.

Every modernization project reaches a moment where someone asks "can you just send us a CSV?" This is what stands between that question and a file your warehouse can trust.

The file does not describe itself

A relational table carries its own schema. A mainframe dataset does not. It is a stream of bytes whose meaning lives somewhere else entirely — in a COBOL copybook, a member of a source library, describing where each field starts, how long it is, and how it is encoded.

That separation is the first practical problem. The copybook may not be under the same change control as the data. It may exist in three versions, one of which matches production. Different programs may read the same file through different layouts. Until you have established which copybook actually describes the bytes you were given, nothing downstream is trustworthy — and the file will not tell you when you are wrong. It will just produce plausible numbers.

The bytes are not what your tools expect

Three encodings account for most of the surprises.

  • EBCDIC, not ASCII. Text arrives in an IBM code page — 037 in the US, others elsewhere. Which one matters: code pages disagree about square brackets, the exclamation mark and currency symbols, so a careless conversion corrupts exactly the punctuation that turns up in names and addresses.
  • Packed decimal (COMP-3). Two digits per byte, with the sign in the final half-byte. A PIC S9(7)V99 COMP-3 field occupies five bytes, holds nine digits, and has an implied decimal point that appears nowhere in the data. Read those bytes as text and you get nonsense; read them as a number without knowing the implied scale and you are out by a factor of a hundred.
  • Binary (COMP) and zoned decimal. Binary integers are big-endian. Zoned decimal keeps one digit per byte but folds the sign into the last one, so a positive value ending in 5 and a negative value ending in 5 differ by a single high-order nibble and render as two unrelated characters.

This is why the most common failure is not a crash but a silence. An FTP in text mode helpfully converts EBCDIC to ASCII across the whole record — including the packed and binary fields, which are not text. The transfer succeeds. The character fields look right. The money is quietly wrong. Binary transfer, then copybook-driven decoding, is the only safe order of operations.

One layout is often several

REDEFINES lets the same bytes mean different things depending on a value elsewhere in the record — a claim whose tail is one shape for medical and another for dental. OCCURS DEPENDING ON makes a record's length a function of its own contents. Neither is exotic; both are ordinary in estates built when storage was expensive.

An extract that ignores them produces rows that are individually well-formed and collectively meaningless. You cannot catch this by eyeballing the output, because the wrong interpretation still yields characters and numbers.

Records are not always the same length

Variable-length datasets prefix each record with a four-byte descriptor word giving its length; blocked datasets add another for the block. Strip them and the data shifts. Keep them without accounting for them and every field is four bytes out. Fixed-length files avoid this and introduce their own quirk: trailing spaces are real characters, not absent values.

Which raises the wider point — there is no NULL. Absence is conventional: spaces, low-values, zeros, or a sentinel like all nines. Deciding what each field's absence marker is, field by field, is part of the extract rather than something to sort out later in the warehouse.

Dates deserve their own paragraph

Expect two-digit years with a windowing rule that lives in code rather than data. Expect Julian dates — 2026031 for the thirty-first day of 2026. Expect 99999999 used as "no expiry", which becomes a spectacular outlier if loaded as a real date, and zeros used the same way at the other end. None of this is discoverable from the bytes alone. All of it is knowable from the programs that write them.

Getting a copy that means anything

A dataset read while the batch cycle is running is a photograph of a moving object. Extracts have to be taken at a defined point in the schedule — usually after a named job step — and that point has to be recorded, or two extracts taken an hour apart will disagree and nobody will be able to say which is right.

So what does "just send a CSV" cost?

The request is not unreasonable. It is that producing the CSV is the last step, and every decision above has to be made correctly first: the right copybook, binary transfer, per-field decoding, redefines resolved, absence markers agreed, a defined point in the cycle. Skip them and you still get a CSV. You get one that is subtly wrong in ways that surface months later, in a dashboard, in front of someone who now does not trust the platform.

The finished job has one more part: reconciliation. Record counts and control totals taken from the source and checked against the landed data, so that "does this match the mainframe?" has a number for an answer rather than an opinion. That is the difference between an extract you can build on and a file that happens to have parsed.

If this is your problem

Tell us what's stuck.

Describe what you are dealing with and we'll tell you honestly whether we're the right people. The first consultation is free.