In records management, dealing with compound records—documents that are related and often must be retained together—can be complex. These documents pose a challenge when trying to apply appropriate retention policies. In this post, we’ll explore this dilemma, the limitations of auto-labeling and AI, and effective solutions, including using the Primary document method and Document Sets.
Before we dive into the dilemma of managing compound records, let’s understand what they are.
First, we have Institutional Records. These are standard records maintained for general administrative purposes and compliance with regulations. For example, meeting minutes, policy documents, and annual reports.
A good example would be Employee Records: personal information, job performance, and payroll records retained for a specified period according to regulations. You’d normally retain such records according to an established File Plan for each type of record.
Compound Records are groups of related documents that must be retained together due to their interdependencies. While each individual document might have different retention schedules by regulations, this may not be the case with compound records. They often support each other and are critical in legal disputes.
Examples:
Adjudication Files: Documents related to a court case, such as evidence, witness statements, and legal arguments.
Mortgage Documentation: Loan agreements, payment records, property deeds, and correspondence related to a mortgage.
Institutional records are your "normal" records, regularly managed to comply with regulatory requirements, while compound records require special attention due to their interconnected nature and potential legal implications.
The Dilemma of Compound Records
Consider the following scenario:
Document A: Needs to be retained for 3 years.
Document B: Needs to be retained for 7 years.
However, Document A supports Document B. If Document A is crucial for understanding Document B, then Document A cannot be deleted based on its content alone and should be retained for the same period as Document B - 7 years.
Note: The need for compound records does not justify non-compliance with regulatory requirements. But in special cases, such as a lawsuit, specific laws may supersede standard regulations, allowing for adjustments to retention schedules. These situations are evaluated case-by-case by corporate lawyers. I am here to explain the technical aspects of managing these records within Microsoft 365, not to provide legal advice.
Microsoft 365 provides several ways to automatically apply retention labels:
Content Types: Retention labels based on predefined content categories.
Sensitive Information Types: Retention labels for documents containing sensitive data.
Keywords: Retention labels triggered by specific keywords in documents.
Trainable Classifiers: AI models trained to recognize specific content within documents.
Metadata: Retention labels applied using document metadata fields.
Microsoft Syntex: Advanced AI for classifying and processing content.
Event-Based Retention: Retention labels applied based on specific events.
But all these methods struggle with the relational context and dependencies between documents, making them less effective for managing compound records. AI lacks the ability to fully understand and manage complex dependencies without explicit metadata or predefined rules.
Some Solutions I'd consider
1. Primary Document Method
In the ideal world with properly designed information architecture, related documents should be stored in the same container, library, or folder. However, there are scenarios where this might not be the case due to various reasons such as company structure, security needs, or oversight. Additionally, when documents are in the same container, they may be mixed with unrelated documents.
In such situations, we can create a custom metadata field which will be used to track an ID of the Primary Document and ID of related documents. The end user can then enter the Document IDs for compound documents to establish relationships between the documents. We can then configure the Labels to apply to the supporting documents with the same retention schedule as the Primary Document.
We could also leverage the Power Automate Flow to aid the users in populating Document IDs using automation.
Steps to Use the Primary Document Method:
1. Create Metadata Fields:
Primary Document: "Related Documents" field
Secondary Documents: "Primary Document ID" field
2. Configure Content Types:
Add crated metadata fields to the respective content types
3. Manual Entry or Automation:
Enter IDs of Primary Document for Related Documents
Enter Related Documents for Primary Document
Once the metadata fields are set, you need to apply retention labels based on the Primary Document.
You can leverage Power Automate Flow to ensure that the retention label applied to the Primary Document is also applied to all related documents.
Remember, that the users can manually override auto-applied labels when necessary. Note that some restrictions apply, such as the inability to override Regulatory Records, and some actions may require SharePoint Admin permissions.
The Primary document method works well for simple cases where one main document (the Primary) is supported by other related documents or when there is an absolute need not to have the documents in the same containers, for whatever reason.
Pros:
Simple and easy to implement for single primary-secondary relationships.
Ensures that all supporting documents are linked to the primary document.
Solves the dilemma of linking dispersed documents.
Cons:
May not work well for complex cases with multiple dependencies and interdependencies.
Requires manual effort or automation to link documents.
Prone to errors. Could be a “crutch” for bad Information Architecture.
2. Document Sets
The Primary document method, quite frankly, is not the best solution if you can group related files together. In such cases, using Document Sets is the best solution.
Document Sets allow you to group related documents and manage them as a single entity with unified metadata and retention policies.
At the same time, files within Document Sets can have their own unique metadata. Document Sets simplify retention and record management by applying retention to the entire container.
If supporting documents are needed on an ad hoc basis, move them to the Legal Case Document Set from their original location.
Ensure you preserve document metadata during this process, as it is required by court procedures.
Use the "Move" function instead of "Copy" to transfer documents to the Document Set. This helps maintain metadata integrity.
This method is ideal for managing a set of documents instead of managing complex relationships between them.
Document Sets simplify compliance and records management.
In fact, Microsoft introduced Document Sets to address Legal Cases, among other things. To me, Document Sets are the best thing that happened to SharePoint since baked bread.
I am working on a comprehensive Document Sets tutorial, which I will publish shortly.
For now, I am just going to leave you with advice to investigate Document Sets if you haven’t already done so.
This post had an objective to bring to your attention compound files and some concerns when dealing with them.
And one more thing, I would like to make sure that when dealing with compound files you are aware of the following:
Potential Non-Compliance
Retaining files beyond their designated retention period without a valid legal reason can breach data protection laws (e.g., Freedom of Information Act, Personal Health Information Protection Act).
Storing personal information longer than necessary without justification can lead to legal liabilities.
Role of Lawyers and Records Managers
Deciding when to transition a file to adjudication status requires careful consideration by legal counsel and records managers. It should be based on actual legal needs rather than speculative scenarios.
Comments