Questions & Answers

TimoKalker · ‎10-17-2023

(2023.7 Server, SmartLiving Test Project)

To be GDPR compliant a client asked us to clean all new pictures from EXIF-Data which can contain information deserving protection like Geolocation, time of creation, Author Name and a description.
To do that, we decided to implement an upload hook.

What confused us was that the upload hook is initially triggered 2 times for uploads to the Smart Living project. This was the first sample code:

public class ExifCleanUploadHook implements ExifUploadHook {
    @Override
    public void preProcess(@NotNull BaseContext baseContext, @NotNull Media media, @NotNull Picture picture, Resolution resolution, @NotNull InputStream inputStream, long length) {
        baseContext.logInfo("\n\nSTART PRE UPLOAD HOOK - PICTURE");
        baseContext.logInfo("PRE - Media: " + media);
        baseContext.logInfo("PRE - length:  " + length);
        baseContext.logInfo("END PRE UPLOAD HOOK\n\n" + inputStream);
    }

    public void postProcess(@NotNull BaseContext baseContext, @NotNull Media media, @NotNull Picture picture, long length) {
        baseContext.logInfo("\n\nSTART POST UPLOAD HOOK - PICTURE");
        baseContext.logInfo("POST - Media:   " + media);
        baseContext.logInfo("POST - length:  " + length);
        baseContext.logInfo("END POST UPLOAD HOOK\n\n");
    }

which resulted in the following log when uploading an image: (I stripped the additional resolutions as they have no relevance)

START PRE UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:10:01.446 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1747" revision="1647" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:10:01.451 (de.espirit.firstspirit.access.BaseContextImpl): PRE - length: 180508

INFO 17.10.2023 16:10:01.452 (de.espirit.firstspirit.access.BaseContextImpl): END PRE UPLOAD HOOK

java.io.BufferedInputStream@3c76dca6

INFO 17.10.2023 16:10:01.453 (de.espirit.firstspirit.access.BaseContextImpl):

START PRE UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:10:01.455 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1747" revision="1647" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:10:01.460 (de.espirit.firstspirit.access.BaseContextImpl): PRE - length: 180508

INFO 17.10.2023 16:10:01.461 (de.espirit.firstspirit.access.BaseContextImpl): END PRE UPLOAD HOOK

java.io.BufferedInputStream@22c3e4b3

INFO 17.10.2023 16:10:01.814 (de.espirit.firstspirit.access.BaseContextImpl):

START POST UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:10:01.815 (de.espirit.firstspirit.access.BaseContextImpl): POST - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1747" revision="1647" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:10:01.818 (de.espirit.firstspirit.access.BaseContextImpl): POST - length: 180508

INFO 17.10.2023 16:10:01.818 (de.espirit.firstspirit.access.BaseContextImpl): END POST UPLOAD HOOK

We then tried implementing the actual logic - Use the preProcess Hook, create another InputStream of the picture without exif data - based on the given picture InputStream - and then replacing the picture with the exifless InputStream.
As setting the picture counts as an additional media upload and thus triggers the UploadHook again, we would run into an infinite loop. To fix that, before we replace the picture, we control if the new file size is smaller than the file without exif so when the file arrives at the exif-removal the 2nd time, nothing happens and the UploadHook comes to a stop. This results into this expected program flow:

Upload an image with exif data
PreProcess Upload Hook 1 triggers
1. The exifless image is smaller than the exif image
2. Replace the image with the exifless image
3. This triggers an additional upload event
4. PreProcess Upload Hook 2 triggers
  1. The exifless image has the same size as the exif image
  2. Do nothing
5. End PreProcess Upload hook 2
  1. PostProcess Upload hook triggers off of PreProcess 2 ending
    1. Do some logging
  2. End PostProcess Upload hook
End Pre Process Upload Hook 1
1. PostProcess Upload hook triggers off of PreProcess 2 ending
  1. Do some logging
2. End PostProcess Upload hook
Modified (exifless) image is now stored in the CMS.

In total, we expect 2 PreProcess and 2 PostProcess events.

This is the code trying to achieve that:

    @Override
    public void preProcess(@NotNull BaseContext baseContext, @NotNull Media media, @NotNull Picture pictureWithExif, Resolution resolution, @NotNull InputStream inputStreamPictureWithExif, long fileSizeBefore) throws UploadRejectedException, IOException {
        baseContext.logInfo("START PRE UPLOAD HOOK - PICTURE");
        baseContext.logInfo("PRE - Media: " + media);

        ByteArrayOutputStream outputStreamPictureNoExif = new ByteArrayOutputStream();
        try {
            new ExifRewriter().removeExifMetadata(inputStreamPictureWithExif, outputStreamPictureNoExif); // https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/formats/jpeg/exif/ExifRewriter.html
        } catch (ImageReadException | ImageWriteException e) {
            throw new RuntimeException(e);
        }
        int fileSizeAfter = outputStreamPictureNoExif.toByteArray().length;
        baseContext.logInfo("PRE - size-before: " + fileSizeBefore);
        baseContext.logInfo("PRE - size-after:  " + fileSizeAfter);

        if (fileSizeAfter < fileSizeBefore) {
            baseContext.logInfo("PRE - Picture has exif, replace picture with version of itself without exif");

            ByteArrayInputStream inputStreamPictureNoExif = new ByteArrayInputStream(outputStreamPictureNoExif.toByteArray());
            pictureWithExif.setPicture(resolution, fileSizeAfter, inputStreamPictureNoExif, null);

            baseContext.logInfo("PRE - EXIF got removed");
        } else {
            baseContext.logInfo("PRE - Picture has no exif, no changes required");

        }
        baseContext.logInfo("END PRE UPLOAD HOOK\n\n" + inputStreamPictureWithExif);
    }

which results in the following log:

INFO 17.10.2023 16:48:00.471 (de.espirit.firstspirit.access.BaseContextImpl): START PRE UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:48:00.472 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1748" revision="1652" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:48:00.497 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-before: 180508

INFO 17.10.2023 16:48:00.499 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-after: 114972

INFO 17.10.2023 16:48:00.499 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Picture has exif, replace picture with version of itself without exif

INFO 17.10.2023 16:48:00.515 (de.espirit.firstspirit.access.BaseContextImpl): START PRE UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:48:00.517 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1748" revision="1652" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:48:00.519 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-before: 114972

INFO 17.10.2023 16:48:00.520 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-after: 114972

INFO 17.10.2023 16:48:00.522 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Picture has no exif, no changes required

INFO 17.10.2023 16:48:00.523 (de.espirit.firstspirit.access.BaseContextImpl): END PRE UPLOAD HOOK

java.io.BufferedInputStream@609a35a1

INFO 17.10.2023 16:48:00.524 (de.espirit.firstspirit.access.BaseContextImpl): START PRE UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:48:00.525 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1748" revision="1652" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:48:00.527 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-before: 114972

INFO 17.10.2023 16:48:00.527 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-after: 114972

INFO 17.10.2023 16:48:00.528 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Picture has no exif, no changes required

INFO 17.10.2023 16:48:00.529 (de.espirit.firstspirit.access.BaseContextImpl): END PRE UPLOAD HOOK

java.io.BufferedInputStream@e8c04ed

INFO 17.10.2023 16:48:00.862 (de.espirit.firstspirit.access.BaseContextImpl):

START POST UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:48:00.863 (de.espirit.firstspirit.access.BaseContextImpl): POST - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1748" revision="1652" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:48:00.865 (de.espirit.firstspirit.access.BaseContextImpl): POST - length: 114972

INFO 17.10.2023 16:48:00.866 (de.espirit.firstspirit.access.BaseContextImpl): END POST UPLOAD HOOK

INFO 17.10.2023 16:48:00.867 (de.espirit.firstspirit.access.BaseContextImpl): PRE - EXIF got removed

INFO 17.10.2023 16:48:00.868 (de.espirit.firstspirit.access.BaseContextImpl): END PRE UPLOAD HOOK

java.io.BufferedInputStream@996cd50

INFO 17.10.2023 16:48:00.870 (de.espirit.firstspirit.access.BaseContextImpl): START PRE UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:48:00.872 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1748" revision="1652" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:48:00.874 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-before: 180508

INFO 17.10.2023 16:48:00.875 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-after: 114972

INFO 17.10.2023 16:48:00.876 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Picture has exif, replace picture with version of itself without exif

INFO 17.10.2023 16:48:00.908 (de.espirit.firstspirit.access.BaseContextImpl): START PRE UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:48:00.909 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1748" revision="1652" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:48:00.911 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-before: 114972

INFO 17.10.2023 16:48:00.912 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-after: 114972

INFO 17.10.2023 16:48:00.912 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Picture has no exif, no changes required

INFO 17.10.2023 16:48:00.914 (de.espirit.firstspirit.access.BaseContextImpl): END PRE UPLOAD HOOK

java.io.BufferedInputStream@623d3224

INFO 17.10.2023 16:48:00.915 (de.espirit.firstspirit.access.BaseContextImpl): START PRE UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:48:00.916 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1748" revision="1652" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:48:00.918 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-before: 114972

INFO 17.10.2023 16:48:00.919 (de.espirit.firstspirit.access.BaseContextImpl): PRE - size-after: 114972

INFO 17.10.2023 16:48:00.920 (de.espirit.firstspirit.access.BaseContextImpl): PRE - Picture has no exif, no changes required

INFO 17.10.2023 16:48:00.921 (de.espirit.firstspirit.access.BaseContextImpl): END PRE UPLOAD HOOK

java.io.BufferedInputStream@5685cd14

INFO 17.10.2023 16:48:01.207 (de.espirit.firstspirit.access.BaseContextImpl):

START POST UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:48:01.208 (de.espirit.firstspirit.access.BaseContextImpl): POST - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1748" revision="1652" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:48:01.210 (de.espirit.firstspirit.access.BaseContextImpl): POST - length: 114972

INFO 17.10.2023 16:48:01.210 (de.espirit.firstspirit.access.BaseContextImpl): END POST UPLOAD HOOK

INFO 17.10.2023 16:48:01.211 (de.espirit.firstspirit.access.BaseContextImpl): PRE - EXIF got removed

INFO 17.10.2023 16:48:01.212 (de.espirit.firstspirit.access.BaseContextImpl): END PRE UPLOAD HOOK

java.io.BufferedInputStream@6d5cc5b4

INFO 17.10.2023 16:48:01.622 (de.espirit.firstspirit.access.BaseContextImpl):

START POST UPLOAD HOOK - PICTURE

INFO 17.10.2023 16:48:01.622 (de.espirit.firstspirit.access.BaseContextImpl): POST - Media: <MEDIUM editor="496" filename="Grumpy-cat-980x651" id="1748" revision="1652" type="pic" uniquedescription="grumpy_cat_980x651">

</PICTURE>

</MEDIUM>

INFO 17.10.2023 16:48:01.625 (de.espirit.firstspirit.access.BaseContextImpl): POST - length: 180508

INFO 17.10.2023 16:48:01.626 (de.espirit.firstspirit.access.BaseContextImpl): END POST UPLOAD HOOK

Or in short:

START PRE
1. size before: 180kb, size after: 110kb
2. START PRE
  1. size before: 110kb, size after: 110kb
3. END PRE
4. START PRE (?)
  1. size before: 110kb, size after: 110kb
5. END PRE
6. START POST
  1. length: 110 kb
7. END POST
END PRE
START PRE (?)
1. size before: 180kb (???), size after: 110kb
2. START PRE
  1. size before: 110kb, size after: 110kb
3. END PRE
4. START PRE (?)
  1. size before: 110kb, size after: 110kb
5. END PRE
6. START POST
  1. length: 110kb
7. END POST
END PRE
START POST
1. length: 180kb (???)
180kb image is uploaded (???)

So this raises a couple of questions:

Why is the Pre-Hook triggered twice? This also happened when he only logged
- This happened in Steps 1.4, 3, 3.4
Why does the Pre-Hook in 3.1 recognize 180kb?
Why there is no Post-Hook triggered in between steps 1.3/1.4, 2/3, 3.3/3.4 - I thought the post hook would trigger activate as the next step of the upload, after the preprocess hooks activate
Why is the image recognized in step 5.1 the file with exif-data?

Is there any way to resolve this?

Windmüller · ‎10-18-2023

Hi Timo! Thanks for reaching out and providing that much debug information.

The first issue you discovered is the multiple execution of the preProcess method. This looks like a bug and I am going to create an internal ticket for that.

That said, your current approach is not going to work because the InputStream passed to the preProcess method cannot be modified. In fact, each UploadHook receives a new instance of it. This is why you see the 180kb image over and over again.

In order to accomplish your goal, you can implement postProcess and call Picture#setPicture with the modified InputStream. For this you just need to implement the UploadHook interface, since postProcess is not called for ExifUploadHooks. Also beware that setting the InputStream will trigger all UploadHooks again, so it should only be called if your hook actually modified the stream.

TimoKalker · ‎10-19-2023

Hi, thanks for the reply!
Just before sending out the ticket I also got the idea that the (internal) "upload this image" function I expect to be called between the Pre and PostProcess hook takes the reference/instance of how the image file was before I changed it in the preProcess hook - so thanks for confirming my suspicion.

I also thought about using the postProcess hook, but I struggled finding the resolution (which is only a param in the preProcessHook). I hacked it using

Picture.MediaElement.StoreElement#getProject#getOriginalResolution

but I am not sure if there is ever a case where an Upload is not of Resolution "Original" - and there seems to be no way to get the resolution from the postProcess hook params. Is there a better solution or did I perhaps miss a method that suits my usecase?

My code works now, but I dont feel good bringing it to prod only having the assumption an upload always has the "original" resolution:

    public void postProcess(@NotNull BaseContext baseContext, @NotNull Media media, @NotNull Picture pictureWithExif, long fileSizeBefore) {
        baseContext.logDebug("START POST UPLOAD HOOK - PICTURE");
        baseContext.logDebug("POST - Media:   " + media);
        baseContext.logDebug("POST - length:  " + fileSizeBefore);

        Resolution originalResolution = pictureWithExif.getProject().getOriginalResolution(); //WARNING: Image might not have been uploaded in the original resolution, this can cause bugs
        InputStream inputStreamPictureWithExif = null;
        try {
            inputStreamPictureWithExif = pictureWithExif.getInputStream(originalResolution);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

        ByteArrayOutputStream outputStreamPictureNoExif = new ByteArrayOutputStream();
        try {
            new ExifRewriter().removeExifMetadata(inputStreamPictureWithExif, outputStreamPictureNoExif); // https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/formats/jpeg/exif/ExifRewriter.html
        } catch (ImageReadException | ImageWriteException | IOException e) {
            throw new RuntimeException(e);
        }
        int fileSizeAfter = outputStreamPictureNoExif.toByteArray().length;

        baseContext.logDebug("POST - size-before: " + fileSizeBefore);
        baseContext.logDebug("POST - size-after:  " + fileSizeAfter);

        if (fileSizeAfter < fileSizeBefore) {
            baseContext.logInfo("POST UPLOAD HOOK - Picture has exif, replace picture with version of itself without exif");

            ByteArrayInputStream inputStreamPictureNoExif = new ByteArrayInputStream(outputStreamPictureNoExif.toByteArray());
            try {
                pictureWithExif.setPicture(originalResolution, fileSizeAfter, inputStreamPictureNoExif, null);
            } catch (IOException e) {
                throw new RuntimeException(e);
            }

            baseContext.logInfo("POST UPLOAD HOOK  - EXIF data got removed from file");
        } else {
            baseContext.logInfo("POST UPLOAD HOOK - Picture has no exif (same file size after removal), no changes required");
        }
        baseContext.logDebug("END POST UPLOAD HOOK");
    }

About the multiple preProcess execution - it would be great to update me once the team finds out what the issue was - I am very curious how this happens.

TimoKalker · ‎10-20-2023

and btw, I got it working with "implements ExifUploadHook" - ExifUploadHook is just an Interface without any methods which itself implements UploadHook, it has no functionality, just a more specific name than UploadHook which I preferred over the generic name to show my intent

Windmüller · ‎10-20-2023

Is there a better solution or did I perhaps miss a method that suits my usecase?

I am sorry but at the moment I do not know of any better way to handle this.

About the multiple preProcess execution - it would be great to update me once the team finds out what the issue was - I am very curious how this happens.

That is quite easy. ExitUploadHooks also implement UploadHook and were not excluded from the general "execute all upload hooks" logic.

ExifUploadHook is just an Interface without any methods which itself implements UploadHook

That is true but ExifUploadHooks are treated differently. I would still recommend to use a generic UploadHook, since once we fix the issue mentioned above, it is likely that postProcess methods will no longer be called.

TimoKalker · ‎10-20-2023

Well, I guess I will log whenever an image is uploaded as not-Original and hope that it never happens

ah, I see, so I guess you had a "UploadHookImpl" and an "ExifUploadHookImpl" that both fired and will discontinue "ExifUploadHookImpl" to fire in future releases.

I should also combine the postProcess(Picture) and postProcess(File) into one and add a check for the file ending in case some pictures are defined as files (recently with webp as documented in the 2023.10 changelogs), just to be sure to really catch every exif-able file type

Questions & Answers

UploadHook - remove EXIF information of an image on upload