Incremental mummificiation.

Description

Add a way to detect that a source file has not changed since it was last mummified, in which case skip the mummification process for that file.

Things we can probably skip doing are the following:

  • Reading the resource for metadata (if a metadata files is already present—but what about detecting metadata file changes in the future)?

  • Reading the resource for processing if the resource has not been modified.

  • Writing the resource if it already exists and the source was not modified.

We can also add a --full flag that will create a full mummification (analogous to a full vs incremental backup) if present.

Existing behavior will change slightly:

  • Descriptions will be written after site generation, not before. In fact each description will be written individually, as part of the mummification process.

  • Every target file will result in a target description, as every description will now contain timestamp and other information necessary for incremental mummification.

Environment

None

Activity

Show:
Garret Wilson
July 4, 2020, 3:27 PM

Dependency change detection issue is a much larger issue and cannot be quickly changed in time for the v0.4.0 release, so that functionality has been split out into GUISE-153.

Garret Wilson
February 22, 2020, 11:51 PM

The first implementation is working great, but there is an issue of a page needing to be updated if some of its dependencies/references change. This includes the template of a page, or its link (such as a regenerated navigation list or an embedded post list widget from ). At some point we'll probably have to create some mechanism so that an artifact (or the mummification context) can accumulate references during mummification and somehow save them for later—perhaps in the description (which might provide a chicken-and-egg difficulty).

Garret Wilson
February 14, 2020, 4:15 PM

This is working correctly for virtually all resources, but the technique will need to be tweaked for generated resources. For example a DefaultXhtmlPhantomArtifact is generated, and does not have a source file. Perhaps we could add a method similar to openSource() to return the source timestamp from a different location. The normal approach would be implemented in BaseCorporealSourceFileArtifact, using getCorporealSourceFile() in case the actual source file is in a different location (e.g. blog entries). The complication is that the determination of the source file timestamp is performed before creation of the artifact, so a little more thought needs to go into this.

It is not clear what we would do for resources that are completely generated from scratch, but we may not have to worry about this for some time. Even blog entries will be generated from some corporeal file, just not in the source tree.

Garret Wilson
February 13, 2020, 4:47 PM
Edited

The incremental mummification logic turns out a little simpler than anticipated. It involves three checks based upon description properties:

  • mummy/sourceModifiedAt: Indicates whether the source file has changed, and therefore the description is recalculated.

  • mummy/targetModifiedAt: Indicates whether the target file has changed, and therefore needs to be regenerated.

  • mummy/targetDescriptionDirty: Indicates whether the description is dirty and needs to be recreated. If the description is recalculated (or didn't exist), it will have no mummy/targetModifiedAt and implicitly case the target file to be generated.

Garret Wilson
February 13, 2020, 4:24 PM
Edited

Currently directory mummification doesn't even use description files, relying on the description files of the directory "content files" (e.g. index.html). Consider improving the directory mummifier to at least check if the directory exists. Or maybe that is inherent in the call to create the directories as needed. Perhaps the check would at least make the debug log entry for directory creation more accurate. In fact the general debug log message for creation of target files should be made prettier.

Fixed

Assignee

Garret Wilson

Reporter

Garret Wilson

Labels

None

Components

Fix versions

Affects versions

Priority

Critical