Getting Started #

Core concepts #

Workflow #

Using Comparizen usually follows these steps:

From your test script:
1. Create a Test Run
2. Send screenshots to that Test Run
3. Finalize the Test Run
4. Get Test Run results
When there are any differences found, review results in the Comparizen Dashboard and accept or reject the differences

Comparison and Baseline #

When sending a screenshot to Comparizen, you’re actually creating a Comparison between two screenshots: the one you’re uploading and one you have previously uploaded and marked as accepted. This set of previously accepted screenshots we call a Baseline.

With every screenshot you upload to Comparizen, you attach a name. After upload, Comparizen will compare your screenshot to one in the Baseline with the same name.

If you’re just started using Comparizen, your Baseline will be empty. This means that the first screenshots you’re uploading will be considered new. After uploading, you can review these uploaded screenshots in the Comparizen Dashboard, mark them as accepted if they are OK, and then they will become part of the Baseline.

Test Runs #

A Test Run is a grouped set of screenshot comparisons, usually linked to a specific ci/cd pipeline build. When uploading a screenshot to Comparizen, you first have to create a new Test Run, get its Test Run ID and pass that to each screenshot-upload.

While your test script is running and uploading screenshots, the Test Run is marked as being in progress. To let Comparizen know that the Test Run is finished, you need to finalize the Test Run. This also prevents any new screenshots to be erroneously added to this Test Run.

Projects #

In Comparizen, a Project is considered a grouping of Test Runs. This allows you to organize your test runs per app or website.

Workspace #

Within Comparizen, a Workspace is group of people working together on one or more projects. This can be analogous to a company, a department or a group of friend working on a fun side project. Every Comparizen account can be part of multiple workspaces.

Comparison Thresholds #

Each screenshot is compared on a pixel-by-pixel basis. While this has the benefit of being very precise in finding differences, it can sometimes be a littie too precise. For instance, artifacts related to font smoothing and anti-aliasing can generate a difference that is not relevant to your test.

To prevent this, Comparizen allows you to configure thresholds which influence how precise the pixel comparison is performed. These come in two forms: per-pixel and total-diff thresholds.

Although the defaults for these thresholds are good enough for most projects, you can tweak them to become more or less sensitive. You can do this on two levels:

On a per-project level from the Comparizen Dashboard’s Project settings. This will change the sensitivity for each subsequent screenshot comparison in the entire project
On a per-screenshot level, given as a parameter when you send a screenshot. This allows you to overrides the project’s settings for that screenshot upload.

Per-pixel threshold #

The per-pixel threshold determines when two pixels are considered different. Comparizen uses the CIEDE2000 algorithm to determine the ‘visual distance’ between pixels based on how human eyes work. This allows Comparizen to have a sensible default, where only differences are reported that are actually visible to people. You can tweak this configuration to make it more of less sensitive.

The value of a per-pixel threshold is between 0 and 100, meaning:

0 - 1 is not visible by the human eye
1- 2 is perceptible through close observation
2 - 10 perceptible at a glance
11 - 49 colors are more similar than opposite
100 is the exact color opposite (ie. black and white).

Total diff threshold #

The total-diff threshold determines the minimum percent of the image’s total amount of pixels should be marked as different before marking the image itself as ’not matching’. For example:

A value of 0 means: one single pixel-difference will mark the images as ’not matching’.
A value of 50 will cause images to be ’not matching’ only if more than 50% of the image’s pixels are marked as different.

Using Comparizen #

Creating/finalizing Test Runs and uploading screenshots is done using a REST API. You can find the API reference here.

We have a couple of client libraries to help you to get started quicker. This is still a work-in-progress, so expect to see newer and more complete versions of these libraries soon. See the helper libraries chapter.

Gathering results #

Depending on your project, you can choose to get comparison results in two ways:

per-comparison: get the results of one screenshot upload
per-testrun: get the results of a test run after all screenshots have been uploaded and processed

Per-Comparison results #

After an upload of a screenshot, the API will return a Comparison ID. You can use comparison-status call to poll for the result.

This API returns one of these results:

UNPROCESSED: the screenshot is still being processed
NEW_TO_BASELINE: the screenshot is considered new to the Baseline
USER_OK: the screenshot is considered equal (enough) to the Baseline image, so it’s automatically marked as ok
USER_NOT_OK: someone has marked this screenshot as not ok, ie it was rejected by someone reviewing it
DIFF_FOUND, DIFF_SIZEMISMATCH_FOUND: the screenshot has differences to the Baseline and needs to be reviewed
ERROR: something went wrong while processing this screenshot. If this happens, please contact Comparison support

Along with this status, you will receive a URL to the comparison result page in the Comparizen Dashboard.

A helper library might give you convenience methods for this too.

Per-Test-run results #

After finalizing a Test Run, you can use the testrun-status API to see what the results were.