Edit page History

ImgLib2 Benchmarks

    ImgLib2
    Benchmarks
    Topics
    Overview
    FAQ
    Tutorials
    Getting Started
    Accessors
    Accessibles
    ImgLib2 Examples
    Introductory Workshop
    Advanced Programming Workshop
    Into ImgLib - Generic Image Processing in Java
    Creating Imglib2 images in MATLAB
    Development
    Developing ImgLib2
    ImgLib2 Discussion

    This page was last updated 2016 May 3.

    This page compares the time performance of image processing operations using raw byte arrays, ImageJ 1.x and ImgLib2. The benchmark tests these various methods for a “cheap” per-pixel operation (inverting an 8-bit image) as well as an “expensive” operation (some calls to java.util.Math) at several image resolutions.

    Some of the charts plot results at several iterations, meaning the test was performed repeatedly in a loop. This is important because the just-in-time compiler (JIT) is able to optimize performance increasingly well as the same code is executed more than once. Hence, we show results after both a single iteration, as well as ten iterations.

    Scenarios

    The data below cover the following scenarios:

    • Cheap operation on 1Mpx image (1000 x 1000), by iteration
    • Expensive operation on 1Mpx image (1000 x 1000), by iteration
    • Cheap operation on 25Mpx image (5000 x 5000), by iteration
    • Expensive operation on 25Mpx image (5000 x 5000), by iteration
    • Cheap operation on various image resolutions, 1st iteration (fresh JVM)
    • Expensive operation on various image resolutions, 1st iteration (fresh JVM)
    • Cheap operation on various image resolutions, 10th iteration
    • Expensive operation on various image resolutions, 10th iteration

    Hardware and software specifications

    • ImgLib2 version 2.9.0
    • ImageJ 1.x version 1.50i
    • Mid 2015 MacBook Pro
    • Mac OS X 10.11.4
    • 2.5 GHz Intel Core i7 processor
    • 16 GB 1600 MHz DDR3 RAM
    • Oracle Java(TM) SE Runtime Environment (build 1.8.0_77-b03) with Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

    Analysis of time performance

    For cheap operations, time performance is dominated by the overhead of looping itself, meaning several methods are significantly slower. However, this loop overhead is generally very small–and for several methods, such as ImgLib Array, the JIT quickly optimizes it down to raw performance. Hence, in the expensive case, performance converges across all methods.

    Looking at trends as image resolution increases (the “various image resolutions” charts), most methods have less than 1/8th second overhead even for relatively large 25Mpx (5000 x 5000) images. And again, for non-trivial image processing operations, performance is extremely comparable. One oddity is that the JIT appears to optimize performance unevenly across image resolutions. However, the exact details of such discrepancies are not consistent across multiple executions of the benchmark code.

    In conclusion, we believe there is little reason for concern regarding time performance of any of these libraries. And the advantages of ImgLib2’s type- and container-agnostic algorithm development certainly outweigh any minor differences in time performance—especially since the flexible containers provide a mechanism for optimizing space performance based on the data type.

    Source code

    The main benchmark code can be found at:

    The script that runs the benchmark at various image resolutions is:

    The shell script also uses a Python script to transform the CSV output into the pChart data on this page:

    Cheap operation results

    ImageJ 1.x, ImgLib2 Array, ImgLib2 Cell, ImgLib2 ImagePlus, ImgLib2 Planar, Raw 9, 15, 15, 12, 12, 4 5, 5, 9, 8, 5, 0 0, 0, 2, 1, 0, 0 0, 0, 3, 0, 0, 0 0, 0, 3, 0, 0, 0 0, 0, 3, 0, 0, 0 0, 0, 3, 0, 0, 0 0, 0, 2, 0, 1, 0 0, 0, 3, 0, 0, 0 0, 0, 2, 0, 0, 1
    ImageJ 1.x, ImgLib2 Array, ImgLib2 Cell, ImgLib2 ImagePlus, ImgLib2 Planar, Raw 10, 42, 92, 39, 40, 7 5, 35, 78, 33, 30, 5 2, 2, 79, 2, 3, 3 2, 2, 78, 2, 3, 3 2, 2, 78, 2, 2, 3 1, 2, 79, 3, 3, 3 2, 1, 75, 2, 2, 4 1, 2, 79, 2, 3, 3 2, 2, 78, 4, 2, 3 1, 3, 83, 3, 3, 4
    , ImageJ 1.x, ImgLib2 Array, ImgLib2 Cell, ImgLib2 ImagePlus, ImgLib2 Planar, Raw 1 Mpx, 9, 15, 15, 12, 12, 4 4 Mpx, 8, 16, 25, 14, 15, 5 7 Mpx, 9, 18, 36, 17, 17, 5 10 Mpx, 9, 25, 38, 23, 21, 5 13 Mpx, 9, 28, 49, 24, 23, 6 16 Mpx, 9, 31, 60, 28, 27, 6 19 Mpx, 9, 36, 66, 31, 31, 6 22 Mpx, 10, 37, 74, 35, 33, 7 25 Mpx, 10, 42, 92, 39, 40, 7
    , ImageJ 1.x, ImgLib2 Array, ImgLib2 Cell, ImgLib2 ImagePlus, ImgLib2 Planar, Raw 1 Mpx, 0, 0, 2, 0, 0, 1 4 Mpx, 0, 0, 15, 1, 0, 1 7 Mpx, 1, 0, 21, 2, 0, 0 10 Mpx, 1, 0, 30, 1, 1, 1 13 Mpx, 1, 1, 39, 1, 2, 1 16 Mpx, 1, 1, 52, 1, 2, 2 19 Mpx, 1, 1, 65, 1, 2, 2 22 Mpx, 2, 1, 70, 2, 2, 2 25 Mpx, 1, 3, 83, 3, 3, 4

    Expensive operation results

    ImageJ 1.x, ImgLib2 Array, ImgLib2 Cell, ImgLib2 ImagePlus, ImgLib2 Planar, Raw 61, 61, 68, 58, 56, 58 43, 44, 55, 52, 49, 50 39, 41, 50, 43, 43, 41 42, 44, 47, 42, 39, 40 40, 43, 41, 39, 41, 43 38, 43, 44, 43, 44, 40 40, 44, 41, 42, 41, 41 39, 38, 40, 42, 40, 41 45, 39, 43, 43, 47, 43 38, 43, 47, 43, 46, 40
    ImageJ 1.x, ImgLib2 Array, ImgLib2 Cell, ImgLib2 ImagePlus, ImgLib2 Planar, Raw 1161, 1171, 1332, 1126, 1162, 1277 1052, 1062, 1210, 1066, 1069, 1024 1037, 1044, 1174, 1030, 1033, 1000 1008, 1038, 1137, 1033, 1026, 978 1027, 1041, 1137, 1037, 1037, 997 1017, 1034, 1096, 1031, 1058, 984 978, 1049, 1079, 1060, 1042, 983 999, 1040, 1072, 1046, 1030, 989 993, 1071, 1100, 1047, 1042, 983 989, 1037, 1061, 1030, 1041, 973
    , ImageJ 1.x, ImgLib2 Array, ImgLib2 Cell, ImgLib2 ImagePlus, ImgLib2 Planar, Raw 1 Mpx, 61, 61, 68, 58, 56, 58 4 Mpx, 185, 196, 232, 202, 190, 222 7 Mpx, 323, 338, 379, 336, 324, 365 10 Mpx, 477, 485, 595, 457, 458, 508 13 Mpx, 608, 615, 692, 601, 614, 667 16 Mpx, 747, 766, 944, 723, 750, 831 19 Mpx, 871, 884, 1019, 849, 878, 1075 22 Mpx, 1013, 1045, 1170, 973, 1013, 1090 25 Mpx, 1161, 1171, 1332, 1126, 1162, 1277
    , ImageJ 1.x, ImgLib2 Array, ImgLib2 Cell, ImgLib2 ImagePlus, ImgLib2 Planar, Raw 1 Mpx, 38, 43, 47, 43, 46, 40 4 Mpx, 165, 169, 173, 157, 180, 167 7 Mpx, 286, 292, 298, 284, 299, 279 10 Mpx, 390, 435, 423, 412, 420, 397 13 Mpx, 518, 589, 529, 535, 547, 512 16 Mpx, 646, 655, 668, 652, 661, 636 19 Mpx, 764, 779, 799, 775, 774, 745 22 Mpx, 866, 913, 922, 911, 909, 859 25 Mpx, 989, 1037, 1061, 1030, 1041, 973