Twist in the Tale: When M365 Classifiers Misfire on Compound Records

In records management, dealing with compound records—documents that are related and often must be retained together—can be complex. These documents pose a challenge when trying to apply appropriate retention policies. In this post, we’ll explore this dilemma, the limitations of auto-labeling and AI, and effective solutions, including using the Primary document method and Document Sets.

Before we dive into the dilemma of managing compound records, let’s understand what they are.

First, we have Institutional Records. These are standard records maintained for general administrative purposes and compliance with regulations. For example, meeting minutes, policy documents, and annual reports.

A good example would be Employee Records: personal information, job performance, and payroll records retained for a specified period according to regulations. You’d normally retain such records according to an established File Plan for each type of record.

Compound Records are groups of related documents that must be retained together due to their interdependencies. While each individual document might have different retention schedules by regulations, this may not be the case with compound records. They often support each other and are critical in legal disputes.

Examples:

• Adjudication Files: Documents related to a court case, such as evidence, witness statements, and legal arguments.

• Mortgage Documentation: Loan agreements, payment records, property deeds, and correspondence related to a mortgage.
  

Institutional records are your "normal" records, regularly managed to comply with regulatory requirements, while compound records require special attention due to their interconnected nature and potential legal implications.

The Dilemma of Compound Records

Consider the following scenario:

• Document A: Needs to be retained for 3 years.

• Document B: Needs to be retained for 7 years.
  

However, Document A supports Document B. If Document A is crucial for understanding Document B, then Document A cannot be deleted based on its content alone and should be retained for the same period as Document B - 7 years.

Note: The need for compound records does not justify non-compliance with regulatory requirements. But in special cases, such as a lawsuit, specific laws may supersede standard regulations, allowing for adjustments to retention schedules. These situations are evaluated case-by-case by corporate lawyers. I am here to explain the technical aspects of managing these records within Microsoft 365, not to provide legal advice.

Microsoft 365 provides several ways to automatically apply retention labels:

• Content Types: Retention labels based on predefined content categories.

• Sensitive Information Types: Retention labels for documents containing sensitive data.

• Keywords: Retention labels triggered by specific keywords in documents.

• Trainable Classifiers: AI models trained to recognize specific content within documents.

• Metadata: Retention labels applied using document metadata fields.

• Microsoft Syntex: Advanced AI for classifying and processing content.

• Event-Based Retention: Retention labels applied based on specific events.

But all these methods struggle with the relational context and dependencies between documents, making them less effective for managing compound records. AI lacks the ability to fully understand and manage complex dependencies without explicit metadata or predefined rules.

Some Solutions I'd consider

1. Primary Document Method

In the ideal world with properly designed information architecture, related documents should be stored in the same container, library, or folder. However, there are scenarios where this might not be the case due to various reasons such as company structure, security needs, or oversight. Additionally, when documents are in the same container, they may be mixed with unrelated documents.

In such situations, we can create a custom metadata field which will be used to track an ID of the Primary Document and ID of related documents. The end user can then enter the Document IDs for compound documents to establish relationships between the documents. We can then configure the Labels to apply to the supporting documents with the same retention schedule as the Primary Document.

We could also leverage the Power Automate Flow to aid the users in populating Document IDs using automation.

Steps to Use the Primary Document Method:

1.     Create Metadata Fields:

   Primary Document: "Related Documents" field

Secondary Documents: "Primary Document ID" field

2.     Configure Content Types:

Add crated metadata fields to the respective content types

3.     Manual Entry or Automation:  

Enter IDs of Primary Document for Related Documents

Enter Related Documents for Primary Document

Once the metadata fields are set, you need to apply retention labels based on the Primary Document.

You can leverage Power Automate Flow to ensure that the retention label applied to the Primary Document is also applied to all related documents.

Remember, that the users can manually override auto-applied labels when necessary. Note that some restrictions apply, such as the inability to override Regulatory Records, and some actions may require SharePoint Admin permissions.

The Primary document method works well for simple cases where one main document (the Primary) is supported by other related documents or when there is an absolute need not to have the documents in the same containers, for whatever reason.

Pros:

• Simple and easy to implement for single primary-secondary relationships.

• Ensures that all supporting documents are linked to the primary document.

• Solves the dilemma of linking dispersed documents.

Cons:

• May not work well for complex cases with multiple dependencies and interdependencies.

• Requires manual effort or automation to link documents.

• Prone to errors. Could be a “crutch” for bad Information Architecture.

2. Document Sets

The Primary document method, quite frankly, is not the best solution if you can group related files together. In such cases, using Document Sets is the best solution.

Document Sets allow you to group related documents and manage them as a single entity with unified metadata and retention policies.

At the same time, files within Document Sets can have their own unique metadata. Document Sets simplify retention and record management by applying retention to the entire container.

• If supporting documents are needed on an ad hoc basis, move them to the Legal Case Document Set from their original location.

• Ensure you preserve document metadata during this process, as it is required by court procedures.

• Use the "Move" function instead of "Copy" to transfer documents to the Document Set. This helps maintain metadata integrity.

This method is ideal for managing a set of documents instead of managing complex relationships between them.

Document Sets simplify compliance and records management.

In fact, Microsoft introduced Document Sets to address Legal Cases, among other things. To me, Document Sets are the best thing that happened to SharePoint since baked bread.

I am working on a comprehensive Document Sets tutorial, which I will publish shortly.

For now, I am just going to leave you with advice to investigate Document Sets if you haven’t already done so.

This post had an objective to bring to your attention compound files and some concerns when dealing with them.

And one more thing, I would like to make sure that when dealing with compound files you are aware of the following:

Potential Non-Compliance

• Retaining files beyond their designated retention period without a valid legal reason can breach data protection laws (e.g., Freedom of Information Act, Personal Health Information Protection Act).

• Storing personal information longer than necessary without justification can lead to legal liabilities.
  

Role of Lawyers and Records Managers

• Deciding when to transition a file to adjudication status requires careful consideration by legal counsel and records managers. It should be based on actual legal needs rather than speculative scenarios.

Minding the Gap: Are We Missing Label Varieties in M365?

Microsoft Purview offers various labeling options to help organizations manage their content effectively. These labels—Retention, Records, and Regulatory Records—each serve distinct purposes and come with their own sets of features and limitations.

I believe that some gaps in their design could lead to compliance risks and operational challenges. This post aims to highlight these gaps, focusing on the shortcomings in Records Labels regarding metadata immutability and the restrictive nature of Regulatory Record Labels.

I previously posted a blog entry regarding the issues with Regulatory Records that could lead to potential non-compliance and how treacherous they could become in organizations that do not invest in governance or Information Architecture design. This is precisely the reason why Regulatory Record Labels, while providing the desired immutability for both content and metadata, are not being considered as the perfect solution.

Gaps in Microsoft Purview Label Options

	Existing	Existing	Existing	Proposed
Feature	Retention Label	Record Label	Regulatory Record Label	Record Label with Immutable Metadata and Content
Purpose	Manage lifecycle of working documents, internal use, business value assets, and those not subject to regulatory requirements	Declare content as records; manage some business value assets with Locked/Unlocked functionality	Ensure compliance with strict regulations	Ensure metadata and content immutability while allowing label or duration changes
Content Immutability	Not enforced	Not enforced when Unlocked; Enforced when Locked	Enforced	Enforced
MetadataImmutability	Not enforced	Not enforced	Enforced	Enforced
Compliance Assurance	Basic lifecycle management, not for records	Moderate compliance	High compliance	High compliance
Use Case	Document stage of lifecycle, Business Value assets	Records management with some flexibility; suitable for some business value assets with Locked/Unlocked functionality	Strict records management	Strict records management with flexibility in label and duration management
Gap 1	Not suitable for regulatory record keeping	Allows changes to metadata	Fully compliant but rigid	None - Suitable for regulatory record keeping with flexible administration
Gap 2	Basic compliance features	Metadata changes undermine record integrity	Lack of flexibility for evolving needs	None - No metadata changes, maintaining record integrity
Gap 3	Limited to basic retention functionality for working documents and business value assets (and in some cases, DLP-like functionality for working documents)	Limited audit capabilities for metadata changes	May be overly restrictive for some scenarios	None - Comprehensive audit capabilities for both content and metadata

The key gap in Microsoft Purview’s Record Labels is their allowance for metadata changes, which contradicts the fundamental concept of records management. Ensuring metadata consistency and integrity is crucial for maintaining the authenticity of records, which is a regulatory requirement for many organizations.

On the other hand, Retention Labels serve well for managing the lifecycle of working documents, internal use, and Business Value assets but fall short for serious record-keeping.

While Microsoft does offer Regulatory Record Labels, their highly restrictive nature makes them less suitable for setups that are not strictly governed or well-designed. The irreversibility of actions taken under Regulatory Record Labels can be a significant risk if not managed correctly.

Therefore, recommending Record Labels can be a more practical approach. With strict governance in the Purview Center, Record Labels can become effectively immutable in practice when locked. However, since metadata can still be changed, they do not fully meet the requirements for ensuring the integrity and authenticity of records in compliance-heavy environments.

Proposed Solution

Introducing a new type of label: Record Labels with Immutable Metadata and Content.

This label would combine the flexibility of Record Labels with the assurance that both content and metadata cannot be altered, while still allowing administrators to change the label applied to the document or the duration. There should be no locking/unlocking of the records with this label type. Both Record Content and Metadata should always be locked.

Why It's Worth Considering

Based on experience, here are some key points:

1. Regulatory Compliance: Many organizations must adhere to strict regulatory requirements. This proposal addresses these needs more comprehensively, helping organizations avoid compliance risks.
   

2. Practical Flexibility: While Regulatory Record Labels are very rigid, this suggested label offers a balance by providing immutability for content and metadata while allowing necessary administrative flexibility.
   

3. Improving the Product: Constructive feedback is vital for product improvement. Highlighting these gaps can help Microsoft enhance their product, ultimately benefiting all users.
   

4. User Advocacy: Advocating for the needs of users and organizations is always valuable. If these gaps are causing challenges, it’s important for Microsoft to be aware so they can address them.

"M365 Regulatory Records: 'Oops' Sounds Even Worse!"

Introduction

In the previous post, I showed you how to enable Regulatory Records, which are disabled by default.

But what are Regulatory Records, and why wouldn’t they be enabled by default?

There are three types of Retention Labels in M365:

• Retention Labels (let’s call it a regular retention)

• Record Labels

• Regulatory Record Labels

A Regulatory Record Label, once applied to an item, cannot be changed, overridden, or removed. This immutability ensures compliance with regulatory requirements. When the retention period of a regulatory record ends, it follows the actions defined by the original label (e.g., deletion). You cannot apply a new label to extend or change this action. Regulatory Record Labels are tamper-proof.

The Issue

So, if the desired outcome is compliance, what’s wrong with having the Regulatory Records option at your disposal right away? Why wouldn’t they be enabled by default?

Because there are quite a few problems with the usage of Regulatory Record labels. In some cases, their inflexible nature becomes an issue, which ironically, could lead to potential non-compliance.

The immutable nature of Regulatory Records in Microsoft 365 is both a feature and a challenge.

One reason Regulatory Records option is not enabled by default is that they are designed for organizations with strict compliance requirements.

Mislabeling

Here is the first common concern: mislabeling the item. Imagine that you made a mistake when applying the label: you cannot reverse that. There is no button to press to undo that action. Incompetent Information Management Workers can create a non-compliance quagmire very quickly.

Having Regulatory Record Labels will require additional training programs and governance. Lack of a bullet-proof file plan and exhaustive documentation can also contribute to potential issues. And even if you have top-notch knowledgeable staff, to err is human.

Poor communication between departments can also lead to inconsistent application of regulatory labels.

Mergers, acquisitions, restructuring, or any other changes in organizational structure can disrupt information management processes and lead to inconsistent application of regulatory labels.

Need for Governance

Would you dare to use Regulatory Record Labels, especially in an organization that doesn’t have a solid file plan or solid personnel training?

Managing immutable records requires a high level of governance, as any mistakes in labeling can pose serious issues. Misinterpreting or misunderstanding regulatory requirements can result in the incorrect application of retention labels.

Additionally, if the end user is presented with a bunch of labels that were not properly named, if there are no naming conventions and Regulatory Record Labels are not reflected in the name, the user might just use them due to confusion. The user would see the Alert when applying a Regulatory Record Label, but we all know that most users don't bother reading them.

Without regular audits and reviews, mislabeling and non-compliance issues can go unnoticed for long periods.

Changes in Regulatory Requirements

Changes in regulatory requirements, although rare, do happen. Imagine if you have a few thousand documents that you were keeping for 7 years, but new regulation requires them to be kept for 10 years. You can increase your retention period.

If the new regulation changes the retention requirement to a shorter period (e.g., from 7 years to 5 years), there's no way to shorten the retention period of an already labeled regulatory record.

The biggest concern is human errors and issues with Information Architecture and Security.

Security Breaches and Unauthorized Access

Security breaches or unauthorized access to records can result in tampering or accidental mislabeling. An insider with malicious intent can cause significant damage by labeling items incorrectly. This malicious activity may go unnoticed, and if discovered, the insider could claim it was an error.

Lack of Security Permissions

M365 has a specific set of security permissions in MS Purview (formerly Compliance Center). Only personnel with specific permissions are allowed to create and publish Labels to the workloads, such as SharePoint.

The issue is that all published labels can be applied by a SharePoint Site user.

It does seem like a significant oversight that Microsoft did not provide more granular control over the application of Regulatory Record Labels in SharePoint Online. Given the importance and immutability of these labels, having more precise permission settings could help prevent accidental or unauthorized application, thereby reducing the risk of compliance issues and operational headaches.

Conclusion

Make sure that you enable the Regulatory Record Label option only if your governance, training, and structure are in place.

Conduct auditing exercises routinely and adjust accordingly.

Build a custom solution that provides Role-Based Access Control (RBAC) to restrict who can apply Regulatory Record Labels.

While I do hope that Microsoft addresses this issue, I am not holding my breath – I am currently building my own framework that will provide better control over Regulatory Record Labels in M365.