Concordance DAT & EDRM XML Load File Production Standards & Defensibility

1. The Bedrock of eDiscovery Productions: Concordance .DAT, Opticon .OPT & EDRM XML

In electronic discovery, producing digital evidence to opposing counsel or government regulatory agencies (such as the SEC, FTC, or DOJ) involves far more than merely exporting a folder of PDF files. To be legally usable, searchable, and navigable within litigation databases (e.g., Relativity, Reveal, Everlaw, and DISCO), productions must be packaged with standardized load files that reconstruct document metadata, document family hierarchies, page-level image boundaries, and extracted text streams.

A production load file serves as the relational database blueprint for the entire collection. If a load file contains corrupted delimiters, missing image pointers, or misaligned Bates ranges, the receiving party cannot ingest the production, leading directly to formal deficiency letters, costly re-processing disputes, and emergency motions to compel under Federal Rule of Civil Procedure 37.

Anatomy of Standard eDiscovery Production Deliverables

• Concordance .DAT File: The primary document-level metadata file. It uses specialized ASCII control characters to separate fields and records, containing crucial attributes such as PROD_BEG, PROD_END, CUSTODIAN, DATE_SENT, SUBJECT, FILE_NAME, and SHA256_HASH.
• Opticon / Relativity .OPT File: The page-level image cross-reference file. It maps every individual Bates-stamped image page (single-page TIFF or JPEG) to its corresponding document boundary, volume identifier, and relative file path.
• Extracted Full-Text Files (.TXT): Individual plain text files containing unformatted OCR or extracted text corresponding 1:1 with each produced document.
• Native Files Folder: Original binary files (e.g., complex Excel spreadsheets, audio recordings, CAD drawings) produced in native form when image conversion would compromise usability.
• Volume Directory Staging: Organizing production packages into standardized volume subfolders (VOL001/IMAGES, VOL001/NATIVES, VOL001/TEXT) with distinct size caps (typically 25GB–50GB per ISO volume image) for smooth physical or secure SFTP transfer.
• EDRM XML Schema: An XML-based hierarchical production standard designed by the Electronic Discovery Reference Model (EDRM) to capture deeply nested document families, custodian lineages, and rich metadata within a single self-validating XML manifest.

+---------------------------------------------------------------------------------------------------+
|                        CONCORDANCE .DAT & OPTICON .OPT PRODUCTION ARCHITECTURE                    |
+---------------------------------------------------------------------------------------------------+
| 1. CONCORDANCE .DAT (DOCUMENT-LEVEL METADATA):                                                    |
|   þPROD_BEGþ þPROD_ENDþ þPROD_BEG_ATTACHþ þPROD_END_ATTACHþ þCUSTODIANþ þSUBJECTþ þSHA256þ      |
|   þABC-00100þ þABC-00100þ þABC-00100þ þABC-00102þ þSmith, Johnþ þQ4 Forecastþ þe3b0c44298fc1c...þ|
|   þABC-00101þ þABC-00102þ þABC-00101þ þABC-00102þ þSmith, Johnþ þFinancial_Model.xlsxþ þ8f43a...þ|
+---------------------------------------------------------------------------------------------------+
                                            +
+---------------------------------------------------------------------------------------------------+
| 2. OPTICON .OPT (PAGE-LEVEL IMAGE CROSS-REFERENCE):                                               |
|   ABC-00100,VOL01,IMAGES\001\ABC-00100.TIF,Y,,,1                                                 |
|   ABC-00101,VOL01,IMAGES\001\ABC-00101.TIF,Y,,,2                                                 |
|   ABC-00102,VOL01,IMAGES\001\ABC-00102.TIF,,,,                                                   |
|   (Where 'Y' indicates the Document Break Flag defining the start of a new document)              |
+---------------------------------------------------------------------------------------------------+

Comparison Matrix: Concordance DAT/OPT vs. EDRM XML vs. Direct API Transfer

For litigation support managers and discovery counsel, adhering to verified load file standards prevents production rejection and protects clients from procedural sanctions. When delivering production volumes across international jurisdictions or complex regulatory enforcement actions, even a single mismatched delimiter can invalidate a multi-terabyte production, forcing emergency meet-and-confer negotiations and damaging litigation momentum.

For related quality engineering protocols and load file production standards, explore our companion guides on Trial Exhibit Bates Stamping & Production Quality Audits and Container File Unrolling & Embedded Attachment Handling.

2. Technical Architecture: Delimiter Integrity, Family Relationships & Hash Validation

Generating a court-compliant Concordance `.DAT` file requires absolute precision across character encodings, delimiter standards, and relational data fields:

1. Standard ASCII Delimiter Tokens

To avoid collision with text content (e.g., commas or tabs within email bodies), Concordance load files utilize four non-printable ASCII control characters:

• Field Separator: ASCII 020 (DC4 / ) — standardly depicted as .
• Text Qualifier: ASCII 254 (Thorn / þ) — standardly depicted as þ. Used to quote text strings.
• Newline Delimiter: ASCII 174 (Registered Sign / ®) or standard Windows CRLF (\r\n) with unescaped internal returns converted to space or line-break tokens (ASCII 029).
• Multi-Value Separator: ASCII 059 (Semicolon / ;) or ASCII 028 (FS) for multi-custodian fields.
• Escape Sequence Handling: Handling raw quotation marks or Thorn characters embedded within custodian text streams through double-escape syntax to prevent premature column truncation.

2. Mandatory Relational Metadata Header Schema

Standard federal court ESI specifications (including DOJ and SEC production protocols) mandate a standardized set of metadata fields:

1. PROD_BEG / BEGBATES: Starting Bates number of the document.
2. PROD_END / ENDBATES: Ending Bates number of the document.
3. PROD_BEG_ATTACH / ATTACH_RANGE: Starting Bates number of the entire document family.
4. PROD_END_ATTACH: Ending Bates number of the entire document family.
5. PARENT_BATES / PARENT_ID: Bates number of the parent email or document.
6. CUSTODIAN / ALL_CUSTODIANS: Name of the primary custodian and deduplicated cross-custodians.
7. NATIVE_PATH: Relative directory path to the produced native file.
8. TEXT_PATH: Relative directory path to the corresponding extracted text file.
9. MD5_HASH & SHA256_HASH: Cryptographic hash signatures computed on the original native file stream.
10. FILE_SIZE_BYTES & PAGE_COUNT: Exact byte counts and rendered page totals to audit document completeness.

3. Quality Assurance Linting Checklist

Before any production archive is delivered, automated linting scripts must verify ten zero-defect rules:

• Zero Missing Bates Pages: Every image row in the `.OPT` file must correspond to an actual image file on disk.
• Strict Bates Continuity: No overlapping Bates numbers or numbering gaps across consecutive documents.
• Family Boundary Integrity: Ensure PROD_BEG_ATTACH matches the parent PROD_BEG, and PROD_END_ATTACH matches the final child’s PROD_END.
• Encoding Consistency: Enforce UTF-8 with BOM or UTF-8 standard encoding to preserve foreign characters and accented names.
• Zero Unreferenced Extracted Text Files: Ensuring every document has an associated `.TXT` file on disk without empty or orphan text files.

3. Judicial Defensibility, ESI Production Orders & Courtroom Standards

Under Federal Rule of Civil Procedure 34(b)(2)(E)(ii), a party must produce documents as they are kept in the usual course of business or organize and label them to correspond to the categories in the request. Producing corrupted load files violates this standard and exposes the producing party to severe evidentiary sanctions:

1. Defending Against Motions to Compel

Courts have repeatedly held that delivering improperly delimited load files or stripped metadata constitutes non-production. In cases such as National Day Laborer Organizing Network v. ICE and Bray & Gillespie v. Lexington Ins. Co., judges issued severe evidentiary sanctions against parties who stripped load file metadata or delivered non-standard formats.

2. Federal Rule of Evidence 902(13) and 902(14) Certifications

Modern litigation practices routinely leverage FRE 902(13) (records generated by an electronic process) and FRE 902(14) (data copied from an electronic device) to authenticate electronic productions without calling live IT custodians to the witness stand. Including verified SHA-256 hashes directly within the Concordance `.DAT` load file provides the cryptographic self-authentication required by federal judges.

3. Rule 26(f) ESI Protocol Negotiation

During the Rule 26(f) meet-and-confer conference, parties should execute a formal ESI Production Protocol detailing exact load file specifications:

1. Explicit Delimiter Standards: Agreeing on standard Thorn (þ) and DC4 ( ) characters.
2. Image Format Standards: Specifying 300 DPI Group 4 single-page TIFF or text-searchable single-document PDF.
3. Date and Time Formatting: Requiring strict ISO 8601 formatting (YYYY-MM-DDTHH:MM:SSZ) with coordinated UTC timestamps.
4. Redaction Transparency: Providing redaction reason codes (e.g., ATTORNEY-CLIENT PRIVILEGE, PII REDACTION) in dedicated load file metadata fields.

For more insights on litigation operations and evidence management, review our comprehensive guides on Enterprise Litigation Support Services & Quality Engineering and Courtroom Admissibility Standards for AI-Generated Fact Chronologies.

4. Production Python Concordance .DAT & Opticon .OPT Validator & Linter

Below is a production-grade Python script that validates Concordance `.DAT` and Opticon `.OPT` load files, checking for delimiter corruptions, Bates continuity, document family integrity, and missing file references:

import re
import json
from dataclasses import dataclass, asdict
from typing import List, Dict, Tuple, Optional

@dataclass
class LoadFileValidationError:
    line_number: int
    bates_number: str
    error_type: str
    error_message: str

@dataclass
class LoadFileAuditReport:
    total_records: int
    error_count: int
    is_production_ready: bool
    errors: List[Dict[str, str]]
    bates_range_start: str
    bates_range_end: str

class ConcordanceDatLinter:
    def __init__(self, field_delimiter: str = "\u0014", text_qualifier: str = "\u00FE"):
        self.field_delim = field_delimiter
        self.text_quote = text_qualifier

    def validate_dat_content(self, dat_raw_text: str) -> LoadFileAuditReport:
        lines = dat_raw_text.strip().split("\n")
        if not lines:
            return LoadFileAuditReport(0, 1, False, [{"error": "Empty DAT file"}], "", "")

        # Parse Header
        header_raw = lines[0]
        headers = [h.strip(self.text_quote) for h in header_raw.split(self.field_delim)]
        expected_cols = len(headers)

        errors: List[LoadFileValidationError] = []
        bates_list = []

        for idx, line in enumerate(lines[1:], start=2):
            raw_fields = line.split(self.field_delim)
            if len(raw_fields) != expected_cols:
                errors.append(LoadFileValidationError(
                    line_number=idx,
                    bates_number="UNKNOWN",
                    error_type="COLUMN_COUNT_MISMATCH",
                    error_message=f"Expected {expected_cols} columns, found {len(raw_fields)}"
                ))
                continue

            fields = [f.strip(self.text_quote) for f in raw_fields]
            row_dict = dict(zip(headers, fields))

            beg_bates = row_dict.get("PROD_BEG") or row_dict.get("BEGBATES", "")
            end_bates = row_dict.get("PROD_END") or row_dict.get("ENDBATES", "")
            beg_attach = row_dict.get("PROD_BEG_ATTACH", "")
            end_attach = row_dict.get("PROD_END_ATTACH", "")

            if not beg_bates:
                errors.append(LoadFileValidationError(idx, "", "MISSING_BEGBATES", "PROD_BEG is blank"))
            else:
                bates_list.append(beg_bates)

            # Verify Attachment Range logic
            if beg_attach and end_attach:
                if beg_attach > end_attach:
                    errors.append(LoadFileValidationError(
                        idx, beg_bates, "INVALID_ATTACH_RANGE",
                        f"PROD_BEG_ATTACH ({beg_attach}) > PROD_END_ATTACH ({end_attach})"
                    ))

        bates_start = bates_list[0] if bates_list else ""
        bates_end = bates_list[-1] if bates_list else ""

        return LoadFileAuditReport(
            total_records=len(lines) - 1,
            error_count=len(errors),
            is_production_ready=(len(errors) == 0),
            errors=[asdict(e) for e in errors],
            bates_range_start=bates_start,
            bates_range_end=bates_end
        )

# Production Verification Demonstration
if __name__ == "__main__":
    sample_dat = (
        "þPROD_BEGþ þPROD_ENDþ þPROD_BEG_ATTACHþ þPROD_END_ATTACHþ þCUSTODIANþ þSUBJECTþ\n"
        "þSEC-00100þ þSEC-00100þ þSEC-00100þ þSEC-00102þ þSmith, Johnþ þProject Apex Deal Memoþ\n"
        "þSEC-00101þ þSEC-00101þ þSEC-00100þ þSEC-00102þ þSmith, Johnþ þValuation_Model.xlsxþ\n"
        "þSEC-00102þ þSEC-00102þ þSEC-00100þ þSEC-00102þ þSmith, Johnþ þBoard_Deck.pptxþ"
    )

    linter = ConcordanceDatLinter()
    report = linter.validate_dat_content(sample_dat)
    print(f"Load File Audit Report:\n{json.dumps(asdict(report), indent=2)}")

Automated load file verification guarantees that every production delivery is structurally perfect, legally defensible, and immediately usable across all litigation platforms.

5. Conclusion & Key Takeaways

Concordance `.DAT` and Opticon `.OPT` load files, alongside standardized EDRM XML manifests, are the foundational bridge of electronic discovery productions. Ensuring absolute delimiter integrity, contiguous Bates sequencing, and flawless parent-child family linkage prevents production rejections, eliminates costly motions to compel, and guarantees courtroom defensibility under the Federal Rules of Civil Procedure.

Key strategic takeaways for litigation support managers, eDiscovery directors, and trial counsel include:

1. Standardize Delimiter Standards Early: Formalize ASCII 020 ( ) and ASCII 254 (þ) delimiters during Rule 26(f) meet-and-confer negotiations.
2. Enforce UTF-8 with BOM Encoding: Guarantee seamless rendering of foreign language characters, accents, and special symbols across review databases.
3. Maintain Strict Document Family Linkage: Populate PROD_BEG_ATTACH and PROD_END_ATTACH fields across all parent emails and child attachments.
4. Validate Page-Level Image Mappings: Cross-check Opticon `.OPT` files to ensure zero missing page rows and exact document break flag alignments.
5. Audit Cryptographic SHA-256 Hashes: Compute and include cryptographic signatures in load files to prove authenticity under FRE 902(13)/(14).
6. Deploy Automated Pre-Delivery Linting: Execute automated Python validation scripts to detect delimiter collisions, Bates gaps, and column mismatches before shipping.
7. Standardize Date and Time Formats: Adhere strictly to ISO 8601 UTC timestamp standards to prevent chronological sorting errors in multi-jurisdictional litigations.
8. Produce Native Files Responsibly: Include relative native file paths for dynamic spreadsheets, audio recordings, and databases that cannot be properly reviewed as static images.
9. Generate Comprehensive Delivery Transmittal Reports: Include detailed volume manifests, Bates range summaries, and exception logs with every production volume.
10. Establish Formal Receipt Verification Checklists: Require receiving counsel to execute automated linting upon receipt and confirm production readability within 5 business days.

6. Frequently Asked Questions (FAQ)