ADP: Auto Re-Migration

We have also previously showcased the APIs that control the Profiles, Settings, and Rules that comprise your Migration Policy, as well as how to create draft policies and view the impact of the changes you are planning.

Building on both of those, in this article we will look at Auto Re-Migration, the counterpart functionality to Auto Re-Characterization in Preservica’s Automated Digital Preservation technology.

What is Auto Re-Migration

We’ve talked about this feature a lot of the last few years, at User Group meetings, Special Interest Group Webinars, and in white papers, so hopefully you’re familiar with what we mean by this. But as a quick recap, decisions about what content should be migrated, to what other formats and using which tools are subject to change and evolution. These might be decisions that are driven by external factors, such as diminishing support for a particular file format, or availability of new and better tooling, or they might be based on internal factors, such as which user communities need access to the content and which formats they will be able to support.

This means that migration policies are subject to change over time, and when they change, content should be re-assessed against the policy to ensure that it still aligns. Manually checking which content may be affected by changes, and executing migrations that are required is time-consuming and error-prone. Preservica has two mechanisms for automating this, depending on what is changing within a policy.  We will now look at each of these in turn.

User Policy Change

Since version 6.0, Migration Settings in Preservica have controlled which migrations should be performed as new content is ingested into the system. With the arrival of Auto Re-Migration, and the ability to control these settings in our Next Generation User Interface, there is now an additional functionality, which enables these settings to form an active policy. That means that when a user makes changes, for example to include more formats, the change will be applied to content that already exists in the system, and not just to new content as it arrives. 

When creating or editing a Preservation Policy with Auto Re-Migration in our Next Generation UI, you have the ability to set “Preservation mode”:

This still supports only applying on ingest (i.e. existing behavior), and also supports having the policy not automatically apply at all.

You can create format Profiles, which determine what format of content should be migrated, and which can be composed from pre-defined “Rulesets”, which specify migrations for a range of file formats, and format specific overrides. If the same format is specified both manually and as part of a ruleset, the manual rule from the manual specification will apply “overriding” the behavior of the ruleset.

Once you have defined your profiles , you can choose where they should apply. This allows you to apply a profile to content under a particular branch of your folder hierarchy, or to content with a particular security tag, to content in a particular representation type, or a combination of these. You can also choose to apply your profile “Everywhere”.

Because these settings now form a policy, and because automatically applying changes could trigger large numbers of migrations, Preservica will now require an Impact statement to be run. Doing this will save your changes as a draft policy and evaluate the impact of applying those changes (i.e. working out how many migrations would be triggered).

Once that has run, you can view the results and decide whether or not to apply your changes.

Once your changes have been applied, background processes will be triggered that evaluate your changes and determine what migrations need to be run. Preservica will attempt to minimize the number of processes it runs, and it does this by checking the difference between the policy before and after the change to figure out what set of assets could be affected. For example, if the change is simply to add one format to a profile that is mapped to content in a specific folder, then we only need to check content in that folder with that specific format. 

When Preservica checks a specific asset, it will look to see whether it already aligns with the policy. To do this, we check whether the content has already been migrated with the same business rule as the policy dictates, and if not, the migration will be performed. For normalization rules, this check is always made against the original content, with the goal being that pre-existing content would have the same migrations applied as if it were being ingested today.

This means that if you have some Microsoft Word Documents (fmt/40) that were originally migrated to DOCX  upon ingest, and you modify your policy to say that Word documents should actually be migrated to Open Document Text files, we will migrate the original document to ODT, and remove the DOCX as redundant content that no longer matches policy.

At this point, it’s worth considering what happens if the change you want to make is to not migrate content that you were previously migrating. In this case, you have two options, you can specify that you want to stop migrating new content, but leave the existing migrations in place; or you can choose to stop migrating new content and remove existing migrations. These are represented in the API using the special string business rules “doNotConsider” and “none”, and are also listed in the UI for every format:

Using “Do not migrate and leave existing migrations” also means that multi-generation and multi-representation content that is ingested using PAX will also get ignored by the migration on ingest.

For very old data, we may not have recorded exactly which business rule was used to perform a migration, in this case, we will look at the expected output format of the migration to determine whether the outcome of an existing migration matches what we would expect to happen if we performed the migration again. If it does, then we will skip migrating that content again.

Although the above is talking about normalizations made for preservation purposes, the same is true for migrations to create Access copies. There is a separate section of a migration profile that controls these migrations, but it works in the same way.

Preservica Originated Changes

Preservica will not actually change your migration policies, but from time to time, changes that we make will affect the outcome of your policies if you use the pre-defined rulesets.

Preservica’s rulesets are mappings of formats to the business rules that should be used to perform migrations on that type of content. They are created to describe a high level requirement and are usually Content-Type specific, so for example we have “Minimal Intervention Raster Image Normalization” and “NARA Document Normalization” rulesets.

What that means is that you’re selecting to perform whatever migrations satisfy the high level conditions. In these cases, that means “only perform migrations on images that are most at risk of obsolescence” and “perform migrations on documents that are in formats that are not on NARA’s preferred or acceptable format lists”. Preservica can obviously only migrate content in those formats where the current tooling allows, but new tools can become available, or existing tools can be upgraded to increase the list of formats they support. Additionally, over time, the list of image formats at risk and the list of document formats that NARA deems acceptable may both change.

Preservica’s Digital Preservation experts perform a Technology Watch to ensure we are aware of changes such as these. As those format lists change, or new tooling becomes available, the contents of the rulesets will be updated accordingly. When that happens, the outcome of applying your migration policy may change, and so Preservica needs to be able to trigger a re-evaluation of the policy even where the actual policy has not changed.

This is similar to the process for re-characterizing content, our Digital Preservation team can make recommendations for policy evaluations that they think should happen automatically on Preservica systems. We can use the same mechanism as for re-characterization, i.e. publishing Recommended Processes to the Technical Registry.

These are the same JSON objects as we described in the re-characterization article. We can set a list of formats to consider, we can restrict content to specific date or event ranges, but we can also make use of additional options to determine what should happen if migrations have already occurred.

The above looks very much like the Recommended Process used to trigger a re-characterization, except that the “processType” here is set to “migrate”. In this case we have included the “rerun-same-outcome” additionalOption.

By default, as with the evaluation triggered by a user changing policy, Preservica would check each item to see if a corresponding migration has already taken place, and if a migration with the same outcome had already occurred, would skip that content. This means that if the change in the ruleset was, for example, to change the migration tool, but not the output format, Preservica would not just replicate a migration.

By setting “rerun-same-outcome”, we are telling Preservica, even if the output format is the same, re-migrate the content. We might do this where changing the tool used improves the quality of the migration so much that it is worth re-migrating everything already processed by the old tool.

We can go a step further and specify “rerun-same-rule”, by which we are telling Preservica, even if the business rule itself hasn’t changed, re-migrate the content. We might do this if an upgrade the same tool significantly improves the quality of the migration.

How do we make these recommendations

Our resident File Format Expert, David Clipsham, performs a comprehensive Technology Watch, analysing changes to our tooling and the external environment in which formats are supported (or not) with each release and piecing together what impact those changes may have on what rulesets we should offer and what those rulesets should specify.

As existing rulesets get changed, any impact that might have on existing policies that use them will be considered and may result in recommendations to perform re-evaluations of policy for affected content.

Most data that ends up in the Preservica Technical Registry is bundled into the application so that it gets added during upgrades, however, the Recommended Processes are added in an essentially ad-hoc manner while the application is running. This means that they can published as and when required.

When will we publish these recommendations

As Auto Re-Migration rolls out you can look forward to us starting to make our way through the backlog of existing recommendations.