HANK@BARILVM.BITNET (Hank Nussbacher) (11/11/90)
The client was a Sun Sparcstation 1. The network was an isolated
Ethernet (via a DELNI) with an HP Lanalyzer checking the network.
40 separate benchmarks were performed on all six X-terminal stations.
The results are presented below. I did not perform the actual
benchmarks but they have been provided to me by the person who
did them and who I have worked with. No further benchmarks are planned.
I will let you each draw your own conclusions from these benchmarks,
but one fact is quite obvious: not all X-terminals are equal.
X-terminal benchmarks
November 1990
---------------------
Vaxstat. Visual Visual NCD16 NCD17c Tektronix
3100 x19turbo x19PLUS XP27
+---------+---------+---------+---------+---------+---------+
Server |DEC- |Visual |Visual |NCD |NCD |Tektronix|
vendor | WINDOWS | | | | | |
Xserver ver.|11.11 |11.3 |11.3 |11.2001 |11.2001 |11.0 |
Revision # |0 |0 |0 |0 |0 |0 |
Defaultdepth| | | | | | |
rootwindow |1 |1 |1 |1 |8 |8 |
Server | | | | | | |
Byteorder |LSBfirst |MSBfirst |MSBfirst |MSBfirst |MSBfirst |MSBfirst |
Server | | | | | | |
BitmapBit- | | | | | | |
order |LSBfirst |MSBfirst |MSBfirst |MSBfirst |MSBfirst |MSBfirst |
Disp. Width |1024 |1280 |1152 |1024 |1024 |1152 |
Disp. Height| 864 |1024 | 900 |1024 | 768 | 900 |
------------+---------+---------+---------+---------+---------+---------+
line 10 | 7761.6 | 4749.8 | 2037 | 2080.2 | 6003.60 | 6367.8 |
line 100 | 4631.4 | 3495 | 1351.20 | 1247.45 | 4545.60 | 6411 |
line 400 | 1813.09 | 1735.09 | 625.50 | 643.64 | 2594.40 | 6261 |
1) Draw a line of length 10". Results in vectors/sec.
2) Draw a line of length 100". Results in vectors/sec.
3) Draw a line of length 400". Results in vectors/sec.
------------+---------+---------+---------+---------+---------+---------+
dline 10 | 858 | 4159.8 | 1598.73 | 1351.2 | 3795 | 6046.20 |
dline 100 | 330 | 2319.27 | 893.50 | 780 | 2066.18 | 6111 |
dline 400 | 97.64 | 1158 | 363.23 | 330 | 896.73 | 3709.20 |
1) Draw a dashed line of length 10". Results in vectors/sec.
2) Draw a dashed line of length 100". Results in vectors/sec.
3) Draw a dashed line of length 400". Results in vectors/sec.
------------+---------+---------+---------+---------+---------+---------+
wline 10 | 740.73 | 625 | 321.86 | 139.06 | 375.5 | 877.64 |
wline 100 | 393.50 | 254.62 | 111.60 | 47.06 | 113.05 | 604.36 |
wline 400 | 143 | 87.56 | 35.75 | 16.03 | 35.83 | 304 |
1) Draw a wide line (linewidth=5) of length 10". Results in vectors/sec.
2) Draw a wide line (linewidth=5) of length 100". Results in vectors/sec.
3) Draw a wide line (linewidth=5) of length 400". Results in vectors/sec.
------------+---------+---------+---------+---------+---------+---------+
rect 10 | 3489.6 | 2230.18 | 993.20 | 836.36 | 2672 | 2117.20 |
rect 100 | 1725.45 | 1725.82 | 637.45 | 544.36 | 2044 | 2117.20 |
rect 400 | 703.64 | 1009.09 | 323.43 | 255.67 | 1168.36 | 2102.80 |
1) Draw a rectangle with 10 pixel sides. Results in rectangles/sec.
2) Draw a rectangle with 100 pixel sides. Results in rectangles/sec.
3) Draw a rectangle with 400 pixel sides. Results in rectangles/sec.
------------+---------+---------+---------+---------+---------+---------+
fillrect 10 | 2207.09 | 2448.20 | 992.36 | 642.80 | 1456 | 1132.40 |
fillrect 100| 1061 | 1438.40 | 500.91 | 426.73 | 948.8 | 1020.18 |
fillrect 400| 229.67 | 365.67 | 121.25 | 125.69 | 187.17 | 473.09 |
1) Draw a filled rectangle with 10 pixel sides. Results in rect./sec.
2) Draw a filled rectangle with 100 pixel sides. Results in rect./sec.
3) Draw a filled rectangle with 400 pixel sides. Results in rect./sec.
------------+---------+---------+---------+---------+---------+---------+
tilerect 10 | 1928.91 | 1958.60 | 816.18 | 520.40 | 1048 | 1111.8 |
tilerect 100| 760.55 | 918.18 | 348.67 | 269.09 | 510.18 | 334 |
tilerect 400| 172.77 | 258.92 | 85.89 | 80.29 | 133.54 | 31.66 |
1) Draw a tile-filled rec. with 10 pixel sides. Results in rect./sec.
2) Draw a tile-filled rec. with 100 pixel sides. Results in rect./sec.
3) Draw a tile-filled rec. with 400 pixel sides. Results in rect./sec.
------------+---------+---------+---------+---------+---------+---------+
stiprect 10 | 2121.60 | 1907.6 | 857 | 463.82 | 1214 | 1101.8 |
stiprect 100| 741.82 | 797.64 | 280.67 | 221.17 | 538 | 632.6 |
stiprect 400| 149.73 | 196.86 | 62.09 | 51.11 | 141.38 | 63.88 |
1) Draw a stipple-filled rec. with 10 pixel sides. Results in rect./sec.
2) Draw a stipple-filled rec. with 100 pixel sides. Results in rect./sec.
3) Draw a stipple-filled rec. with 400 pixel sides. Results in rect./sec.
------------+---------+---------+---------+---------+---------+---------+
invtrect 10 | 7202.60 | 5590.09 | 2384.64 | 1426.91 | 3526.1 | 5746.27 |
invtrect 100| 1333.10 | 1837.36 | 663 | 664.64 | 1075.09 | 918.73 |
invtrect 400| 118.30 | 255.38 | 93.20 | 95.61 | 119.50 | 64.47 |
1) Draw an inverted rec. with 10 pixel sides. Results in rect./sec.
2) Draw an inverted rec. with 100 pixel sides. Results in rect./sec.
3) Draw an inverted rec. with 400 pixel sides. Results in rect./sec.
------------+---------+---------+---------+---------+---------+---------+
arcs 10 | 556.55 | 72.30 | 176.62 | 363.27 | 1120.55 | 409.64 |
arcs 100 | 371.09 | 63.80 | 101 | 241 | 841.6 | 262.91 |
arcs 400 | 219.67 | 50.27 | 40.96 | 124.31 | 347.82 | 115.43 |
1) Draw an arc with 10 pixel diameter. Results in arcs/sec.
2) Draw an arc with 100 pixel diameter. Results in arcs/sec.
3) Draw an arc with 400 pixel diameter. Results in arcs/sec.
------------+---------+---------+---------+---------+---------+---------+
fillarcs 10 | 496 | 68.86 | 36.87 | 34.82 | 103.29 | 246.6 |
fillarcs 100| 102.13 | 51.04 | 26 | 28.42 | 85.14 | 146.91 |
fillarcs 400| 26.23 | 30.19 | 12.87 | 16.52 | 40.32 | 61.44 |
1) Draw a filled arc with 10 pixel diameter. Results in arcs/sec.
2) Draw a filled arc with 100 pixel diameter. Results in arcs/sec.
3) Draw a filled arc with 400 pixel diameter. Results in arcs/sec.
------------+---------+---------+---------+---------+---------+---------+
fillpoly 100| 137.83 | 122.14 | 46.94 | 35.62 | 35.62 | 233.09 |
1) Draw a filled polygon with 5 points (size=100). Results in poly/sec.
------------+---------+---------+---------+---------+---------+---------+
screencp 10 | 345.27 | 577.64 | 298.5 | 206.18 | 570 | 763.64 |
screencp 100| 219.50 | 298.50 | 120 | 158.50 | 409.20 | 770.18 |
screencp 400| 47.70 | 73.26 | 18.27 | 68.40 | 83.57 | 64.12 |
1) Copy a screen to screen with 10 pixel sides. Results in copies/sec.
2) Copy a screen to screen with 100 pixel sides. Results in copies/sec.
3) Copy a screen to screen with 400 pixel sides. Results in copies/sec.
------------+---------+---------+---------+---------+---------+---------+
scroll | 82.80 | 140.86 | 43.18 | 50.25 | 58.47 | 42.32 |
1) Scroll an area 640x400. Results in scrolls/sec.
------------+---------+---------+---------+---------+---------+---------+
bitmapcp 10 | 314.75 | 1059 | 517.91 | 337.55 | 755.30 | 928 |
bitmapcp 100| 218.67 | 410.75 | 162.54 | 244.45 | 302.64 | 928.1 |
bitmapcp 400| 60.06 | 89.67 | 20.74 | 64.07 | 41.47 | 77.79 |
1) Copyplane (Bitmap->Screen) with 10 pixel sides. Results in copies/sec.
2) Copyplane (Bitmap->Screen) with 100 pixel sides. Results in copies/sec.
3) Copyplane (Bitmap->Screen) with 400 pixel sides. Results in copies/sec.
------------+---------+---------+---------+---------+---------+---------+
drawimagestr|22213.80 |27880 | 9815.4 | 7749 | N/A |20350.91 |
1) Draw an image string (font=fixed, height=13). Results in chars/sec.
Note: for NCD17c, font fixed is not available.
------------+---------+---------+---------+---------+---------+---------+
window c/d/d| 34.80 | 35.70 | 16.80 | 12.40 | 28.70 | 38.20 |
1) Window create, draw, destory. Results in runs/sec.
------------+---------+---------+---------+---------+---------+---------+rnovak@mips.com (Robert E. Novak) (11/13/90)
Are the sources for the benchmarks available?
Do the benchmarks take any steps to verify that the requested operations
are actually performed by the server?
SPEC has been looking at the various X benchmarks but they have all been
unsuitable since there is no verification of operation. My
understanding is that with X11R4 it is possible to read back from the
server a bitmap of the drawing that was created on the server. Once
this capability is in place, SPEC will probably be more aggressive in
exploring graphics performance.
One of th base requirements for all SPEC benchmarks (as you may have
guessed) is that the results of a program can be mechanically verified
against known good results.
---
Robert E. Novak Mail Stop 5-10, MIPS Computer Systems, Inc.
{ames,decwrl,pyramid}!mips!rnovak 950 DeGuigne Drive, Sunnyvale, CA 94086
rnovak@mips.COM (rnovak%mips.COM@ames.arc.nasa.gov) +1 408 524-7183graeme@labtam.labtam.oz (Graeme Gill) (11/14/90)
In article <43091@mips.mips.COM>, rnovak@mips.com (Robert E. Novak) writes: > Are the sources for the benchmarks available? > From the look of the tests xbench was probably used to produce these figures. The other widely known test suit is x11perf. xbench as it was distributed over the network has some problems. One major one is that it does not disable or accept the no-expose events generated by its copy area tests. This can cause strange results as the host starts running out of memory trying to store all the events. There are also other peculiarities, ie the invert rectangles test only does one rectangle at a time, hence it is partially a measure of network latency rather than pure invert area performance. This is inconsistent with the other tests. x11perf is most useful for developmental work on servers, although it is possible to use its results to draw conclusions about the relative performance of an X server. I make use of x11perf extensively in verifying the results of my server optimisations. This often involves modifying and/or adding tests to the x11perf suite. > One of the base requirements for all SPEC benchmarks (as you may have > guessed) is that the results of a program can be mechanically verified > against known good results. This could be a difficult issue. Although the X specifications specify exact pixelisation rules for most graphical operations, some are deliberately relaxed - ie zero width lines - so that machine dependent hardware can be used. Since zero width lines are widely used by X applications, one cannot simply leave out these tests. The MIT X11R3 example server (which a number of vendors products are still based on) does not even meet the pixelisation rules for some operations. Verification would be almost impossible to do at the same time as speed testing, since efficient use of the X protocol calls for doing as many graphics operations as possible per packet, and reading back an image is a relatively slow operation. If you really wanted to 'cook' the results of a server could save all the commands and only render them on receiving a getimage request. The performance of all operations except getimage would then seem exceptionally fast. Verification of the graphics rendering is only part of the problem, as other commands would also have to be verified - ie window creation, cursor operation, exposures etc etc. It would certainly be very useful to have an X pixelisation verification suite, but this seems to be a difficult project, as the closest thing available for the MIT consortium is a partially completed X protocol verification suite. If such a tool was available then one could use it to verify the correct functioning of the device under test, and then run the performance benchmarks, but whether this is what you are looking for, I don't know. The other slight possibility would be to come up with a series of operations that leave you with a (hopefully) unique pattern that can then be verified. The current benchmarking tools only cover a small fraction of the spectrum of drawing operations that may differ markedly in speed on a particular X server. For instance, X allows the 16 boolean logical operations, but generally only fill and invert are tested by benchmarks. Many servers will special case these two ops as they are the ones used by applications the vast majority of the time. The speed of textured fills will vary markedly with the size of the texture pattern used, since servers may have 2 or 3 different algorithms depending on whether a line of the pattern will fit in a register or whether it is even, and therefore doesn't need bit shifting. Performance will vary widely depending on the number of operations that can be grouped together (ie the size of the poly fill rect request etc.), the frequency of sync commands etc. Benchmarking of X terminals is especially difficult since many of the results will depend on the speed and exact implementation of the host machines communication interface - ie Ethernet, TCP/IP etc., and how that interacts with the server communications. X servers can have notoriously uneven performance, so that two applications that make different demands on the X server may vary markedly in relative speed when running on different servers. In summary, trying to benchmark X servers in a fair way may make CPU benchmarking look very simple. Considerable investigation of the issues that may affect performance is needed. If perfect verification of operation is needed, then benchmarking may not be possible at all. Graeme Gill Labtam I.S. Pty. Ltd. graeme@labtam.oz.au