Reviews   ::        

Articles   ::        

Home   ::        

Links   ::        

Archives   ::        

Search   ::        

About Us   ::        
 
HDTV Costs     

HDTV Guide     

Diskeeper 9     

Stor n Go PRO     

Blade SSD Server     

 
     
 
 

 
  PCMark04 Benchmark Preview
 
 

If you have been following the benchmarking scene lately, then you know it's a very tumultuous time. With all the cheating and finger-pointing, it seems like a giant soap opera. Consumers just want a reliable way to evaluate the merits of different hardware...but it's not that easy. There are many who extol the virtues of real-world gaming benchmarks. On the otherhand, there are many synthetic benchmarks that aim to accomplish the same thing. Some benchmarks are specific to a particular subsystem or component (i.e. HD Tach, CDSpeed, etc.), while others offer the full benchmarking suite for systems (i.e. SiSoft Sandra, SysMark, etc.). The problem with synthetic benchmarks is that hardware manufacturers may write specific code (at least for for the most popular ones) to artificially inflate their results. If a hardware manufacturer does it for a game, it's called "tweaking", but if they do it for a benchmark, it's called cheating. At the heart of the synthetic benchmarking firestorm is a company called Futuremark, creators of the 3DMark programs.

Over the years, Futuremark's (formerly Mad Onion) 3DMark programs were a staple in many reviews. While there was some grumblings against 3DMark99, 3DMark2000 and 3DMark2001, many websites continued to use the programs in their hardware reviews. It wasn't until 3DMark2003 was released that whole cheating scandal broke wide open. While 3DMark2003 faced the onslaught of the debate over "non-game" benchmarks, another program developed by Futuremark, PCMark2002, flew largely under the radar. But with PCMark04, Furemark hopes to make some noise.

What is PCMark04?

PCMark04 is an application-based benchmark. It uses portions of real applications instead of including very large applications or using specifically created code. This allows PCMark04 to be a smaller installation as well as to report very accurate results. As far as possible, PCMark04 uses public domain applications whose source code can be freely examined by any user.

PCMark04 measures home PC usage. The workload is designed to stress the PC in the same manner as typical home usage does. The workload includes a test suite that gives an overall PCMark score as well as several component test suites that give individual scores measuring the CPU, memory, graphics, and hard disk drive. Some of the tests run concurrently in separate threads. Multithreading is commonly used in software design to maximize performance and resource utilization. Lastly, PCMark04 includes the ability to define custom test suites that are tailored to your own specific needs. The user can choose various tests and compose them in multithreaded sets. This allows the user to model real-life usage where several applications running in a multithreaded fashion compete for PC resources.

System and Component Benchmarking

System benchmarking allows users to accurately evaluate the overall performance of their PC hardware for a defined usage type. Component benchmarking, by contrast, measures the performance of individual PC components, such as CPU, memory, graphics card, etc. Benchmarking PCs can be a complex process as hardware comes in a tremendous variety. Each PC component ‚?? motherboard, CPU, system memory, graphics card, etc. ‚?? has multiple possible manufacturers and variations. Furthermore, the rapid pace with which manufacturers introduce improved versions of these components makes system benchmarking a very challenging task.

PCMark04 supports both system and component benchmarking. System benchmarking produces a measure of the PC‚??s overall performance for a specific type of usage, and in the case of PCMark04, home usage. It produces a single number that is easily comparable to that of other PCs. However, advanced users and professional testers may want to further explore the performance of individual components such as the CPU, the memory, the graphics subsystem, the hard disk, etc. Component benchmarking helps them to isolate the performance of individual components, exposing their strengths and weaknesses. PCMark04 even includes the ability to create custom benchmarks by putting together tests tailored to specific needs.

Benchmarking Approaches

PCMark04 stresses the PC in a similar manner as PCs are stressed in real life-home usage and there are two common approaches for building the workload: Application and Synthetic benchmarking.

Application benchmarks include complete applications the user may use. Such benchmarks would include the complete word processor, email client, graphics authoring software, etc. The primary benefit of this approach is that the performance results are likely to correlate closely with the actual performance of using the included applications. Application benchmarks have some drawbacks in their usability and accessibility. They typically have a very large installation footprint ‚?? they require a large amount of disk space making distribution and usage more challenging. Moreover, they may take a long time to run due to the inherent size and complexity of the benchmark.

Synthetic benchmarks tend to have the opposite strengths and weaknesses as application benchmarks. They include code fragments written specifically to mimic the work performed by real-world applications. This allows them to be small in file size and quick to run. As mentioned earlier, however, synthetic benchmarks often face the criticism of not being the ‚??real thing‚?Ě as actual applications are not being used. Despite this, synthetic benchmarks can be powerful tools for isolating performance of certain parts of the hardware. PCMark04 draws the best parts of both of these approaches by building the workload using an Applicationbased approach. The workload consists of actual applications with freely available public code as much as possible‚?°. However to keep the benchmark small and easy to run, we use only relevant parts of these applications instead of complete applications.

PCMark04 Tests Used

Task Category
Task
Test Used
Applications Used
Productivity Text Editing Grammer Check Link Parser
Picture Viewing Image Processing JPEG image decompression
Graphics Memory DirectX
Internet Internet Browsing Web page rendering Internet Explorer 6
Entertainment Music Playing and Recording Audio Conversion OGG Vorbis
Video Playing and Recording Video en/decoding WMV and DivX video encoding and decoding
Gameplay Physics Calculation Havok
3D Graphics DirectX
General Desktop Usage Transparency, moving windows Windows 2D graphics API
Compression File compression and decompression Zlib
Encryption File encryption and decryption Blowfish
Virus Checking Virus scanning F-Secure

PCMark04 was developed in cooperation with the following members of Futuremark‚??s Benchmark Development Program (BDP): AMD, ATI, Creative Labs, Dell, Gateway, Imagination Technologies, Innovision, Intel, Microsoft, NVIDIA, S3 Graphics, Transmeta, and XGI.

Security Addressed

PC component manufacturers often provide tools to optimize code for their hardware. Typical examples are compiler optimization flags. Software developers use these to achieve the best possible performance for their software. As these are generic optimizations, they are also acceptable for use in PCMark04. We allowed CPU manufacturers to propose a compiler and a set of optimizations flags for any code Futuremark compiles. Guidelines for these optimizations were:

  • Any compiler and flags proposed must produce correct code, should be documented, must be supported by the manufacturer and should be encouraged for public use.
  • Futuremark reserves the final right to accept or reject any compiler and flags proposals.

If no compiler and flags are proposed, generic Microsoft Visual Studio .net compilation is used Optimized compilation is used by default and can be disabled from the user interface. We also allowed vendors to examine any code that was written by us, for example the memory tests. The vendors suggested optimized code paths, which we reviewed for accuracy and implemented at our discretion. Note that vendors are not allowed to detect a running PCMark04 instance in any of their drivers or software and use that to reduce the workload or alter the behavior. Our policy for generic optimizations was published earlier this year.

Observations

Before I go any further, it is important to note one significant caveat that could cause some people to avoid this program...it requires Windows Media Player 9 and the Windows Media Encoder to provide final PCMarks. There are many who do not like what Microsoft has done to Media Player and Digital Rights (DRM). You can still run the benchmark and get results for individual tests, but you won't be able to obtain a final PC Mark.

As has been my experience with all Futuremark/Mad Onion products, PCMark04 was rock stable. I don't think I've ever had one of their applications crash on me.

I ran PCMark04 on two machines without incident. However, one machine did not have Excel on it to save the results. I saved them to the default ".pcr" format, but found no way to open the files up. It's probably somewhere in the documentation, but I just didn't have time to dig for it. It would have been convenient to just double-click on the file and have it open up the results browser.

Examples of PCMark and CPU Scores Scaling
System
PCMark Score
CPU Score
Athlon XP 1700+ (1469MHz)
2663
2470
512MB Memory
ATI Radeon 9700 PRO
Athlon XP 2700+ (2170MHz)
3549
3434
512MB Memory
ATI Radeon 9700 PRO
Athlon XP 2700+ (2170MHz)
3309
3456
512MB Memory
ATI Radeon 9000
Athlon 2500+ (1837MHz)
3182
3158
1GB Dual Channel 333MHz DDR
GeForce Ti 4200

(test system in bold)

Looking at the scores above, my Athlon 2500+ scored right in between the 1700+ and the 2700+ processors. The results for the CPU test show little impact from the videocard used. In fact, the lowly 9000 scored higher than the mightly 9700 PRO.

Examples of PCMark and Memory Scores Scaling
System
PCMark Score
Memory Score
Celeron 2.6GHz (2593MHz)
2894
2317
256MB 266MHz DDR
GeForce Ti 4600
Pentium 4 3.2GHz (3192MHz)
4486
2993
256MB 266MHz DDR
GeForce Ti 4600
Pentium 4 3.2GHz (3192MHz)
4753
4679
256MB Dual Channel 400MHz DDR
GeForce Ti 4600
Athlon 2500+ (1837MHz)
3182
2331
1GB Dual Channel 333MHz DDR
GeForce Ti 4200

(test system in bold)

Here the results left me baffled. The memory performance of the nForce2 in dual channel mode with 1GB of 333MHz DDR should have scored much higher. Unfortunately, there wasn't enough time to investigate this further.

Over the years, the the Futuremark/Mad Onion products have delivered some serious eye-candy. From XL-R8R to 3D Mark, you were almost guarenteed some visual delights. However, that is not the case with PCMark04. Instead, the program seems to shy away from the graphics end of things. The only significant graphics display is the Escher-esque scene shown below.

Graphics test

The ability to run individual tests is always appreciated when a benchmark is released. It allows the user to isolate the results of hardware or driver changes as well as test out system tweaks.

Final Thoughts

There are many benchmarks out there that stress how well a PC handles gaming challenges. However (believe it our not), PC's are used for other things besides games. The ability of our rigs to handle tasks like virus scanning and audio conversions can be just as important to some people. PCMark04 hopes to fill the need to evaluate those areas of performance.

In the brief time I had to play with PCMark04, it appears to be a significant improvement over PCMark2000. It's a solid benchmarking tool that focuses on measuring the computer's ability to handle the more mundane tasks it's required to perform...when not actively engaged in gaming ;-) And because of that, we will be using it for some of our upcoming reviews.

For more information on PCMark04, click here.

 
      Posted by: , November 25, 2003, 2:59 pm  

 
    Cool banner #1
 
 
       ::  USB News

       ::  Bjorn 3D

       ::  [H]ardOCP

       ::  BurnOutPC

       ::  I am Not a Geek

 
 
Top Products


Processors

AMD

Intel

More...


Cases

Antec

Enlight

More...


Motherboards

Abit

Asus

Tyan

More...


Sound Cards

Creative Labs

Hercules

More...


Graphic Cards

ATI

nVidia

More...


Hard Drives

IBM

Maxtor

Quantum

More...


 

 

 
2001 - 2004 Digital Silence
Digital Silence is not responsible for the information or the accuracy of the information above.
All trademarks and copyrights owned by their respective companies.

Graphical Design by Mohsin Ali
Website Layout by Universal Interactive

PHP Programming by Network Innovations
Additional HTML Programming by Moddin.Net