Project Stage 1 - Compiling and Benchmark Butteraugli

Introduction
This post is about the project I am working on the SPO600 class. I will benchmark one open-source, profile it, and optimize the project. The requirements for this project are the following:

• The running time has to be more than 4 minutes
• The open-source software package has to include a CPU-intensive function or method
• The open source software package has to implement in a language that compiles to machine code (such as C, C++, or Assembler)

The detailed project instruction can be found on the Winter 2020 SPO600 Project, I had the struggle to find a proper open-source with the requirement. I narrowed down the type of software I want to work on and found the open-source that is proper for this project. I will describe what is a project I selected, the ways I build and compile the open-source, how to benchmark the project.

The Project I Selected: Butteraugli
The Butteraugli is the project Google created. It estimates the psychovisual similarity of two images. The detailed information about this project is written on the Google Opensource website about the Butteraugli. The source code for this project is located here: GitHub - Butteraugli.

Build and Compile the Open Source Package
1. Clone the repository into my machine. 
The command is the following:
      git clone https://github.com/google/butteraugli.git

2. Compile the software
Move to the repository which contains the 'Makefile' and enter 'make' to compile the project. I found the 'Makefile' into the 'butteraugli' repository so I moved to the repository and enter the command.
      cd butteraugli
      make

3. Store the images for this project
The purpose of this project is comparing two images. Before I compared two images, I copied two similar-size images from my Seneca matrix account. The website I got the sample images is this. To get the time for more than 4 minutes, I used the 5mb and 10mb images.
      scp mysAccount@matrix.senecacollege.ca:~/the/path/image/stored/test10mb1.png .
      scp mysAccount@matrix.senecacollege.ca:~/the/path/image/stored/test10mb2.png .

4. Run the software
After the images are prepared, run the tool to compare two images. As I need to get how much time it takes to run the tool, I put 'time' in front of the running command.
      time ./butteraugli test5mb.png sample5mb.png
It shows the real, user, and sys time to run the tool.

Apply Various Optimization Level
There 7 different optimization levels.

• -O0: disables most (but not all) optimizations
• -O1: enables basic optimizations that can be performed quickly
• -O2: enables almost all safe optimizations
• -O3: enables aggressive optimization including optimizations which may not always be safe for all code
• -Os: optimizes for small binary and runtime size, possibly at the expense of speed
• -Ofast: optimizes for fast execution, possibly at the expense of size
• -Og: optimizes for debugging. It applies optimizations that can be performed quickly and avoids optimizations which convoluted the code

The information above is from Winter 2020 SPO600 Weekly Week4. You can get more detailed information on this website.

To apply different optimization levels, I need to modify the 'Makefile'.
1. Open the 'Makefile' to modify the file.
    vi Makefile

2. Modify the file to apply optimization level O2

The image below is the original content when you clone the repository.

You should put the optimization level in the 'CSSFLAGS' part. For example, if you want to apply -O2, you should enter
    CXXFLAGS += -std=c++11 -l.. -O2

3. Recompile the software
If you have not compiled yet, enter 'make' let you compile the software. However, if you already have compiled, it does not let you clear (command: make clean)or recompiled (command: 'make' or 'make all') the software even though you enter 'make clean'. 

In this case, there is one way you can recompile this project. After you remove the generated files you got when you compiled the software, it allows you to recompile the software. There are two generated files named 'butteraugli_main.o' and 'butteraugli.o'. The command to remove those files and compile the software are the following:
   rm butteraugli_main.o butteraugli.o
   make

Now the optimization level is successfully applied. You can repeat these steps when you want to change the optimization level.

Benchmark the Performance of the Current Implementation
For the benchmark the performance, I used 'Aarchie' and 'Betty' servers for the 'Aarch64' system benchmark. I used 'xerxes' and 'matrix' servers for the 'x86_64' system benchmark. I applied -O0, -O1, -O2, -O3, -Os, -Ofast, -Og optimization levels for the benchmark.

1. -O0 Benchmark

2. -O1 Benchmark

3. -O2 Benchmark

4. -O3 Benchmark

5. -Os Benchmark

6. -Ofast Benchmark

7. -Og Benchmark

Reflection

The most interest part I have learned during stage 1 is the performance depending on the optimization level. The performance under -O0 and is 9 times slower than the performance under -O1 optimization level. It means if I apply -O1 manually on the source code, I can reduce a lot of time using -O. In the next posts, I will profile the software to find what functions take a long time to perform and optimize them.