M365 Quagmire alert!
Intro
Hello Record Wranglers!
What if I told you that In-Place Record Management might not be the best solution for your M365 implementation? If you are a sizeable organization with high compliance requirements, In-Place Record Management might be the worst solution for you!
How come? After all, Microsoft recommends In Place Records Management!
In this article, we’re going to look at all the issues that In-Place Record Management presents. Some issues are solvable, but you’ll need to tinker with your tenant. However, some issues cannot be solved at all.
We’ll explore each issue in detail, and hopefully, by the end of this post, you’ll agree with me that for many client solutions, In-Place Record Management could be a very bad idea.
So, today, I'm going to build the case against In-Place Record Management in favour of the good old Records Center.
Let’s dive right in!
What is a Record Center?
A Record Center is a repository that is set up just for records. Think of it as a super organized vault - only specially trained Record Managers have management access to a Record Center.
It's not meant for keeping documents or for collaboration.
This is a classic records management setup.
In-Place Records Management
In-Place Records Management in SharePoint allows documents to be declared as records without moving them to a separate Records Center. This means the records remain in their original location for the entire duration of the retention period until the record is disposed of.
What it also means is that both regular users and records managers use the same library for working with documents and records. The responsibility for managing records typically falls on designated records managers rather than regular users. These records managers are trained professionals responsible for ensuring that records are properly managed, retained, and disposed of in accordance with organizational policies and regulatory requirements.
Microsoft introduced both In-Place Records Management and the Records Center in SharePoint 2010. In SharePoint Online, In-Place Records Management is emphasized as a primary method of Records Management. Microsoft calls it Modern Records Management.
The Classic Records Center template is no longer available in SharePoint Online, but the idea of having a dedicated space for managing records is still important for many organizations.
PROS AND CONS
Before looking into some of the obvious and readily advertised pros and cons of these different approaches, it's important to understand that these general advantages and disadvantages can vary significantly depending on the context and requirements of your M365 implementation. As we start diving into the real issues, you may find that some of these pros and cons shift dramatically based on specific needs.
In-Place Record Management Pros
Eliminates the need for a separate Records Center.
Eliminates need to move documents to a separate Records Center.
Users can declare records within their familiar sites and libraries.
In-Place Record Management Cons
Depending on your settings, records may be subject to unintentional or unauthorized changes.
Storing large volumes of records alongside active documents can make libraries bloated and result in performance degradation.
Usability issues: Managing records within active libraries can lead to bloated libraries, making it harder to work with documents due to the clutter of records dispersed throughout.
Record Center Pros:
Centralized management: Provides a dedicated repository for managing records according to regulatory and organizational requirements.
Enhanced security: Ensures that only Records Managers have access to records, providing a higher level of protection and control over sensitive information.
No bloated Libraries.
Record Center Cons:
Complexity: Setting up and managing a separate Records Center can be more complex and time-consuming. However, while it is often being cited as a disadvantage, I’d say that if you managed to create Information Architecture structure for collaboration, creating structure for Record Center is easier.
Management complexity: Managing a separate Records Management Center will introduce additional administrative overhead.
Library Size Challenges
Remember what I said about records sitting in the original library for the entire duration of retention? Yes, imagine if the records have a retention period of 25 or 30 years.
Now, how many files would accumulate in that library? In 3 years, in 6 years? In a decade? Even if your retention period is within 7 years, that’s a lot of documents and they just keep adding!
Microsoft did increase the size of the libraries, but UI throttling is still around. It is still 5,000 items. Sure, there are always Views and Managed Metadata Library Partitioning, but those have limitations. Normally, it is not enough of available partitioning methods to cover the basic needs of collaboration, let alone needing additional partitioning for the records.
There are also talks about creating Views for records on the Internet. Yes, while you can add the “Item is a Record” column to the View, which will allow you to quickly identify records, you can’t filter the view by the "Item is a Record" column. Effectively, creating a View for Records Managers only is not possible.
Even a regular volume of working documents can put a strain on the end users and on the system. Sometimes one of the considerations when designing your SharePoint repositories is to have library partitioning strategies. Many clients just want to start a new library each fiscal year, of course, if their business processes allow that. Always think ahead when designing your libraries and your overall solutions. While In-Place Records Management solution sounds like a quick win, it might bite your clients in a few years.
Performance issues and degraded user experience are critical considerations when implementing any document management system, especially when dealing with large volumes of data. Mixing records with active documents within the same libraries can exacerbate these issues due to the increased complexity and size of the libraries.
Library Lifecycle
How long will you use the collaboration library? Are there forever libraries? What about the site? What if they are no longer required and need to be deleted? The records are always kept longer than the projects. Why would you want to impose the overhead of maintaining sites and libraries after they are not in use?
Collaboration vs Record Management
While collaboration is essential for many documents and projects within organizations, records should be treated differently due to their legal, regulatory, and historical significance.
While it seems easy to declare a record in a collaborative space, In-Place Records Management introduces complexities and potential risks to the management and governance of records.
Without the ability to declare records as immutable, there's a risk that records or metadata may be modified or tampered with.
A word on Preservation Hold Library
Preservation Hold Library in SharePoint Online provides a mechanism to preserve documents when they are modified or deleted.
However, relying too heavily on the PHL presents several challenges:
Only SharePoint Site Admins can access the Preservation Hold Library in, making it difficult for regular users or record managers to retrieve documents. This creates a bottleneck and potential dependency on administrative staff for restoring preserved items.
Restoring documents from the Preservation Hold Library is not straightforward. It is not a Recycle Bin.
When documents are under retention for long periods (e.g., 10 years), the PHL can become cluttered with numerous preserved versions of various documents.
While the Preservation Hold Library serves an essential function in preserving document, its practical limitations make it less ideal. And if you’ve been paying attention, the In-Place Record Management solution is more prone to accidental edits and deletions.
When Life and Limb Matter
Consider this scenario: within SharePoint Online, a multitude of records, including outdated regulations, coexist alongside current ones. While these records may not be actively used, their presence amidst other documents poses a risk.
New employees, unaware of their outdated status, might inadvertently rely on these records, potentially leading to errors or safety hazards, particularly in industries like oil and gas, engineering, or construction where accuracy is paramount.
This underscores the importance of proper records management practices, especially within In-Place systems, to segregate outdated records and prevent them from being mistaken for current standards. In critical industries, ensuring that obsolete records are out of sight could be a matter of life and death.
Physical and Digital Records Management
Organizations managing both physical and digital records require specialized facilities and processes to ensure the proper handling, storage, and retrieval of records in various formats.
A dedicated Records Center offers integrated management capabilities for both physical and digital records, streamlining records management workflows and ensuring compliance with regulatory requirements for all record types.
Managing records in place will not work well with physical records management systems.
Complex Policies and Disposition Review
Organizations with complex records management policies, including multiple retention schedules, disposition rules, and access controls, require the flexibility and granularity offered by a dedicated Records Center. This allows for tailored management of records based on their lifecycle stages and business context.
Have you ever encountered a situation where records pile up, disposition reviews get backed up, and compliance becomes a distant dream? I've seen it firsthand in organizations where the sheer volume of records makes planning disposition reviews difficult to nearly impossible. As a result, records linger past their retention periods, putting the organization at risk of non-compliance.
Now, consider the dilemma of managing disposition reviews within In-Place records management systems. While it's possible to organize records within regular collaborative libraries, it often lacks the specialized mechanisms needed for efficient planning and reporting. Without dedicated tools for disposition planning, tracking overdue records, and generating compliance reports, records managers face significant challenges in ensuring timely and compliant disposition of records. While MS Purview does provide disposition schedule capabilities, many organizations have their own way of conducting Disposition Reviews. In some organizations, I’ve seen it done on the annual basis, some do it quarterly, and are required to submit the Records for Disposition Review in a certain way.
Having dedicated mechanisms for records management ensures that the disposition review process can be conducted with greater precision and control, enabling records managers to effectively prioritize their efforts, track the status of records throughout their lifecycle, and generate reports to support compliance and decision-making.
In-Place Records Management and Search Implications
With In-Place Records Management, the search can return a huge number of items, making it difficult to narrow down results. On the other hand, in a dedicated Records Center, the search can be done on records only, allowing users to find records more efficiently and effectively.
Of course, it is entirely possible to create a custom search experience, but it will require additional effort and might not solve the issue entirely.
Security and Insider Risk Management
Normally, organizations require different security for active documents versus records.
Dedicated Record Center Security access is limited to Record Managers.
In contrast, while In-Place Records Management also allows for setting permissions and access controls on individual items or libraries, managing security for records dispersed throughout various locations within the organization's SharePoint environment may be more challenging.
Additionally, content prioritization would be impossible when setting up Insider Risk Management policies, since we couldn’t differentiate between locations of Collaborative spaces and Records repositories.
Backup and Recovery
Yet another neglected area is the challenge of balancing backup frequency with resource efficiency.
Overstuffed libraries will be driving up a size of SharePoint Sites, which might complicate backup and recovery operations. The sheer time required to restore in the event of a disaster might be disruptive to business operations.
Additionally, some backup options, particularly if you don’t want to keep all your eggs in one basket (in Azure), might impose maximum size limits.
Active documents is a more volatile environment and would have different requirements for point of restoration than static records. Therefore, creating a backup and recovery schedule would be more challenging for In-Place Records Management systems. Continuously backing up a large volume of records, especially those that remain unchanged over time, can strain resources and lead to unnecessary costs.
Normally, backup strategies are aligned with the type of data, so that organizations can optimize resource utilization, ensure data protection when it matters most, and avoid unnecessary expenses. This approach not only reduces operational burdens but also optimizes data protection efforts.
Auditing
One critical aspect to consider is the audit of records. When records need to be audited, having them in a centralized location like the Records Center simplifies the process. However, with In-Place Records Management, records remain in their original collaboration sites. If both records and collaboration documents start to accumulate, how do we manage auditing then?
In SharePoint Online, audit logs are powered by the Unified Audit pipeline, which means specific events can no longer be chosen for editing, and site-level audit log trimming is no longer supported.
Records and M365 Restructuring
Imagine that the solution you created today would need to be changed in the next several years if the company goes through restructuring, acquisition, etc. Or, plain and simple, the Information Architecture structure is no longer working, or it was not quite well done to begin with (happens so often). What if the new stakeholders demand changes?
On top of reshuffling documents, you would need to deal with records. What if you also have to deal with regulatory records? I am not saying that In-Place Records Management solves the issue. But in my experience, the need for restructuring is almost always driven by issues within collaborative containers. At the very least, it’s something that you should consider.
Segregation of Duties
If your organization has a separate team of Records Managers who are only responsible for managing records, a separate Records Center would work better. There will be no advantage to making the users who are producing information work on declaring the records manually.
User Training Requirements
Some organizations are subject to strict laws and regulations, and strict regulations would often require specialized training for anybody who has access to records. If your regular information workers have access to records within In-Place Records Management, it might be an issue.
When Record Center is a better Solution
Strict Regulatory Compliance
Organizations subject to stringent regulatory requirements, such as those in the healthcare, finance, or legal sectors, need a dedicated Records Center to ensure compliance with industry regulations.
Physical Records
Organizations keeping physical records alongside with digital records should not be using In-Place Record Management.
Long-Term Archiving and Preservation
Records with long-term archival requirements, such as historical documents or institutional records, benefit from dedicated storage and preservation facilities offered by a Records Center.
What’s involved in setting up Record Management Center?
The purpose of this blog post was to question certain aspects of the Modern Records Management. Setting up a Records Center is a huge topic on its own and is out of scope for this article. But here are the main activities.
First activity would be setting up a Records Management Center. In SharePoint Online, it could be a Hub with SPO Communication Sites. Your Information Architecture menu will vary.
Secondly, you would need to restrict permissions to Records Managers. Depending on the organization and their regulatory requirements, users might get read-only access. But some organizations will restrict even read access and create approval processes to access particularly sensitive or secret information.
Then, you would need to design the mechanisms for moving your documents to the Records Center where they become records. You must do it without changing files' metadata. Moving items instead of copying will achieve that result.
Automated processes like Power Automate Flows could handle the moving of documents to a Records Center. You can also create multi-stage Retention Policies.
It is doable.
Putting It Into Practice
As you consider your own records management strategy, take time to evaluate the specific needs and requirements of your organization. Think about the points raised in this article, such as the importance of compliance, the potential for bloated libraries, and the need for effective auditing and backup processes.
Conclusion
While In-Place Records Management is a viable solution for small or unregulated businesses, organizations seeking stringent segregation of duties and strict compliance may find centralized management solutions more suitable for their needs. Each approach should be evaluated based on organizational requirements, compliance obligations, and the desired level of control over records management processes.
Ultimately, the choice between In-Place Records Management and traditional Records Centers depends on various factors. Both approaches have their merits, and the choice is yours!
Comments