GPU2 Old
From GPU2 Linux
Contents |
Folding@Home GPU v2.0 Windows Client on Linux Wiki
This Wiki assumes you have a general understanding of Linux and are comfortable at the command line. Most of the commands given can be used as is with minor obvious edits, but this still may be to advanced for Linux new comers. You should also be familiar with installing and using a console version of Folding@Home.
Warning!
The writer of this Wiki and all associated parties assume no responsibility for any damages as a result of this Wiki, code, commands, or files used. While all of the following has been tested, it has only been tested on a handful of systems by a handful of users. Please back up anything of value before starting. Do not use this on a production system where crashes, data loss, or general system slowness and instability is unacceptable. Only use this guide and associated files on a system you own or have permission to use for this purpose.
Why?
The GPU version 2 client for Folding@Home is currently only available on Windows XP and Windows Vista. The Folding@Home project has stated that a GPU client for Linux and Mac OS X will be released at some point after the Windows client has stabilized. Some dedicated Folding@Home users that have compatible high end video cards that either can not or refuse to use Microsoft Windows still wanted to donate their GPU's. See the about GPU2 page for more information.
Should We Do This?
We do not have access to the Folding@Home source code and any reverse engineering of that code would be against their terms of use. We do not want to get on anyones bad side and we are very dedicated to this project. We do not want to submit bad results either. Many people close to the Folding@Home project know of what we have done and they have monitored a few results sent back using the wrapper from this Wiki. The only response we received so far is that they would prefer to see us not use this as the client is still not stable. They never responded saying the results that were sent back are bad.
Given that CUDA works the same on any platform and that Nvidia confirmed they use the same source code on all platforms, we feel there is a minimal issue with the data being processed incorrectly and sending back garbage results. It also looks like the Folding client itself catches many issues pretty well on its own.
How?
The CUDA drivers for Windows and Linux accept the same commands for processing data on a GPU. A CUDA program written for Windows should be able to be recompiled for Linux with no changes to the CUDA code itself. Nvidia has made CUDA very cross-platform compatible. The only exception to this is on the Windows platform CUDA can call Directx functions and on Linux CUDA can call OpenGL functions. This does not effect how Folding@Home operates as the GPU client does not use this set of functions.
So the complexity of getting a Windows only CUDA program to run in Linux was significantly reduced due to this cross-platform compatibility. When we first tried to run F@H GPU2 in Wine we saw that the client was trying to communicate with he CUDA driver through cudart.dll (later version of the client renamed this file to nvcuda.dll). The system calls for Windows are structured differently from Linux. So the solution was to create a wrapper which would convert these system calls from the nvcuda.dll library to the cudart.so library found in the Linux CUDA toolkit. This wrapper is the nvcuda.dll.so file which is a Wine specific file.
Prerequisites
Nvidia Video Card
This only works for Nvidia users who have a CUDA compatible video card. The CUDA technology first was introduced on the Nvidia G80 based cards, GeForce 8x00, and has continued support on all later Nvidia GPU generations to date.
Nvidia Binary Driver with CUDA Support
You will need to install a driver with CUDA support and the CUDA Toolkit which can be found here:
http://www.nvidia.com/object/cuda_get.html
If you have a 64-bit Kernel you can install the 64-bit driver. Regardless of which kernel you are using you MUST install the 32-bit Toolkit. This is a compatibility issue with Wine and the CUDA wrapper.
Remove All Prior Installations of Wine
See Video Card Spoofing before continuing. Compiling Wine is no longer needed. These instructions will stay up for special cases.
For Debian based systems like Ubuntu
sudo apt-get remove wine
Download and Install Kernel Source Code
Install the kernel source and headers for your distro. The packages will vary based on distro and version. make sure to extract the packages after they are downloaded. Generally they can be found in the /usr/src/ directory and will need to be extracted there as well.
- More specific information is needed here, please help
Configuring Wine
Download Source
First you will need to download the Wine source. This has been tested with the stable 1.0 release and current unstable releases.
Latest Stable Release 1.0
http://prdownloads.sourceforge.net/wine/wine-1.0.tar.bz2
Latest Development Release 1.1.4 - as of 9-16-2008
http://prdownloads.sourceforge.net/wine/wine-1.1.4.tar.bz2
Extract the source to a location of your choice. A common place would be
/home/(your user name)/wine-1.1.4/
Video Card Spoofing
NOTE: Video Card Spoofing, and thus recompiling Wine has been pointed out to not be required. The same effect can be achieved by starting the GPU2 client with the option "-forcegpu nvidia_g80"
The following will be left for special cases.
Wine only reports a generic video card by default. Most programs are not effected by this. GPU2 however, uses this information to verify you are using a compatible video card.
Open the directx.c file
wine/dlls/wined3d/directx.c
From where you extracted the wine source. Change the adapters driver and description section from this:
Adapters[0].driver = "Display"; Adapters[0].description = "Direct3D HAL";
To this:
Adapters[0].driver = "nv4_disp.dll"; Adapters[0].description = "NVIDIA GeForce 8800";
This section can be found at
Line 3915 in Wine 1.0 Stable
Line 4201 in Wine 1.1.4 Unstable and Wine 1.1.5 Unstable
You should be able to change the "8800" part to which ever card you currently use, omitting any card revisions such as GS, GT, GTS, or GTX. At this time you should not see any increase in speed by setting a better model card in this line. This setting is only to inform the Folding@Home client that you have a compatible video card and is independent from the driver itself. For example, if you have a GTX 280, it will run at full speed regardless of this setting.
Wine Dependencies
If you are running a 64-bit distro then make sure you have package ia32-libs installed.
32-bit and 64-bit kernels run this command so Wine has all dependencies
Assumes Debian based distro
sudo apt-get build-dep wine
For 64-bit distributions please follow this guide to help prepare Wine to install on your system.
If you are using 64-bit Ubuntu Hardy Heron then manually run these commands from the folder you extracted wine to. You will only need to do this once. If you need to recompile the source you will NOT need to re-run these commands. If you have replaced the source with a new version you might need to re-run these.
mkdir -p `pwd`/lib32 ln -s /usr/lib32/libX11.so.6 `pwd`/lib32/libX11.so ln -s /usr/lib32/libXext.so.6 `pwd`/lib32/libXext.so ln -s /usr/lib32/libfreetype.so.6 `pwd`/lib32/libfreetype.so ln -s /usr/lib32/libfontconfig.so.1 `pwd`/lib32/libfontconfig.so ln -s /usr/lib32/libGL.so.1 `pwd`/lib32/libGL.so ln -s /usr/lib32/libGLU.so.1 `pwd`/lib32/libGLU.so ln -s /usr/lib32/libXrender.so.1 `pwd`/lib32/libXrender.so ln -s /usr/lib32/libXinerama.so.1 `pwd`/lib32/libXinerama.so ln -s /usr/lib32/libXxf86vm.so.1 `pwd`/lib32/libXxf86vm.so ln -s /usr/lib32/libXi.so.6 `pwd`/lib32/libXi.so ln -s /usr/lib32/libXrandr.so.2 `pwd`/lib32/libXrandr.so ln -s /usr/lib32/liblcms.so.1 `pwd`/lib32/liblcms.so ln -s /usr/lib32/libpng12.so.0 `pwd`/lib32/libpng.so ln -s /usr/lib32/libcrypto.so.0.9.8 `pwd`/lib32/libcrypto.so ln -s /usr/lib32/libssl.so.0.9.8 `pwd`/lib32/libssl.so ln -s /usr/lib32/libxml2.so.2 `pwd`/lib32/libxml2.so ln -s /usr/lib32/libjpeg.so.62 `pwd`/lib32/libjpeg.so ln -s /usr/lib32/libXcomposite.so.1 `pwd`/lib32/libXcomposite.so ln -s /usr/lib32/libcups.so.2 `pwd`/lib32/libcups.so ln -s /usr/lib32/libXcursor.so.1 `pwd`/lib32/libXcursor.so ln -s /usr/lib32/libdbus-1.so.3 `pwd`/lib32/libdbus-1.so ln -s /usr/lib32/libhal.so.1 `pwd`/lib32/libhal.so ln -s /usr/lib32/libsane.so.1 `pwd`/lib32/libsane.so ln -s /usr/lib32/libgphoto2.so.2 `pwd`/lib32/libgphoto2.so ln -s /usr/lib32/libgphoto2_port.so.0 `pwd`/lib32/libgphoto2_port.so ln -s /usr/lib32/libldap-2.4.so.2 `pwd`/lib32/libldap.so ln -s /usr/lib32/libldap_r-2.4.so.2 `pwd`/lib32/libldap_r.so ln -s /usr/lib32/liblber-2.4.so.2 `pwd`/lib32/liblber.so ln -s /usr/lib32/libxslt.so.1 `pwd`/lib32/libxslt.so ln -s /usr/lib32/libcapi20.so.3 `pwd`/lib32/libcapi20.so ln -s /usr/lib32/libjack.so.0 `pwd`/lib32/libjack.so ln -s /usr/lib32/libodbc.so.1 `pwd`/lib32/libodbc.so
Compiling The Source
Now compile the source from your Wine directory.
If 64-bit then run
CC="gcc-4.2 -m32" LDFLAGS="-L/lib32 -L/usr/lib32 -L`pwd`/lib32 -Wl,-rpath,/lib32 -Wl,-rpath,/usr/lib32" ./configure -v
make depend && make
sudo make install
If 32-bit then run
./configure
make depend && make
sudo make install
If you are using a muli-core processor
Compiling Wine takes a while even on rather fast processors. While compiling, only one core will be utilized. Multi-core processors are very common now and it would be a shame to not take advantage of the additional processing power. If you have a multi-core processor then run this
make -j# depend && make -j#
Where # is the number of cores +1, so on a dual core it would be
make -j3 depend && make -j3
If at any point you need to recompile wine then run
make clean
from the wine source folder and start over.
CUDA Wrapper
Wrapper
Second Generation Wrapper
This wrapper has a built in delay. It was found that CUDA is using some sort of busy loop which was causing 100% CPU usage and major user interface slow downs. Very little real work is being done during this loop so a delay was added that brings the CPU usage down much closer to what the client actually needs for feeding the GPU. This delay is for 300 microseconds and may not work correctly on all systems. If you notice instability, client running slower than it should, or any other anomalies, then please use the first generation wrapper.
Note: you will need a high resolution timer in your Kernel for this delay to work correctly. Run
cat /proc/timer_list
and look for "hrtimer_interrupt" and ".hres_active: 1", if those are present then you should be all set.
The current kernels (up to 2.6.26) will fail to enable the system's high resolution timers when support for the "CPU C1E state" is enabled. Disabling the support of the C1E state in the BIOS can solve this.
Download the wrapper nvcuda.dll.so
First Generation Wrapper
Wrapper Save Location
Place the nvcuda.dll.so file in this directory
/usr/local/lib/wine
- Need to add source code and compile instructions to make more OSS friendly
Linking The Toolkit
You will need to tell the system that the CUDA toolkit is installed and make sure the nvcuda.dll.so file can find its required dependencies. Add /usr/local/cuda/lib to /etc/ld.so.conf and run as root
ldconfig
Then run
ldd /usr/local/lib/wine/nvcuda.dll.so
If you see any "not founds" then something has been setup wrong. Commonly we have seen this
user@box:~$ ldd /usr/local/lib/wine/nvcuda.dll.so linux-gate.so.1 => (0xffffe000) libcudart.so.2 => not found libwine.so.1 => /usr/local/lib/libwine.so.1 (0xf7e4b000) libm.so.6 => /lib32/libm.so.6 (0xf7e25000) libc.so.6 => /lib32/libc.so.6 (0xf7cd6000) libdl.so.2 => /lib32/libdl.so.2 (0xf7cd2000) /lib/ld-linux.so.2 (0xf7fac000)
If this happens then your CUDA toolkit has not been installed correctly or you installed the 64-bit version. You must install the 32-bit version of the toolkit regardless of what your system is. Remember this is independent of the Nvidia driver which can be 64-bit.
Also make sure the ldconfig method used above is the same for your distro. This method is currently used for Ubuntu 8.04.
Note: The LD_LIBRARY_PATH environment variable can be used as an alternative. Set it in your login script or before starting Wine.
user@box:~$ export LD_LIBRARY_PATH=/usr/local/cuda/lib:$LD_LIBRARY_PATH
Running The Client
Download Client
Go to the link and download the Windows Folding@Home client for the console. It should be the second Windows client option.
http://folding.stanford.edu/English/Download
Extract the zip file into a new folder. Open a console, go to the folder you extracted the client to, and then run the client.
nice -n 19 wine Folding@home-Win32-GPU.exe
or if you did not recompile Wine from source, you can simply force the proper GPU by instead running the client with:
nice -n 19 wine Folding@home-Win32-GPU.exe -forcegpu nvidia_g80
The first time you run this you should be prompted for your user information like with the other console clients. This version will accept the same console flags such as -config or -configonly. The "nice -n 19" part is to make it easier to stop the client later as your system will respond faster to user input.
To stop the client, have the console selected and press
crtl + C
Important
Please read all of this before continuing to use the client.
Client Instability
I have been noticing the Fahcore_11.exe process still running after I stop the client. After you have stopped the client check your system monitor, or top, for running processes and kill the fahcore process if it is still running. The WU seems to start back up without an issue after doing this, but if you leave this process running and try to start GPU2 again then you will make your current WU EUE as well as all other WU's the client tries to start. The client will reach a limit eventually and freeze all further activity for 24 hours, or until the client has been restarted. Please be responsible and check for this issue so we can limit the amount of EUE's sent back.
3D desktop effects (compositing) may need to be disabled while running the client. This includes Compiz-Fusion and KDE 4 special effects or anything else like it. With a 3D desktop enabled I have seen random EUE's and Xorg reset after about ten minutes of use. This may no longer be the case with Nvidia driver 177.67, after installing this driver I have had no problems keeping KDE compositing on while running the GPU client, YMMV so keep an eye out for issues.
A switch to a login console (for example with ALT+F1), a X server shutdown or a X session logout can cause the core to hang.
System Lock Up
Only applies to the first generation wrapper.
For some reason while the client is running there are two processes taking up a large portion of processing power, Xorg and Fahcore_11.exe. The fahcore is a given but we can't seem to figure out why Xorg is using so much power. The assumption is that Xorg may be waiting on the Nvidia driver to finish what it is doing. On my dual core processor each process gets 50% CPU time or one core each. This causes the system to be unusable while the client is running. Ctrl + C to stop the client makes the system usable again. There have been reports that on Quad core processors this is not so much the case.
Another option is to edit the file
client.cfg
located in your F@H client folder. Then edit the "core" line
[core] cpuusage=100
and change it to
[core] cpuusage=50
(If you do not have this file in your F@H client folder or it doesn't contain a [core] section, run the client with the "-configonly" flag, answer the questions, and say "yes" to the question "Change advanced options?," and continue to answer all questions - this will generate a config.cfg with a [core] section). This will only let the client use up to 50% CPU power. This will increase your frame time, so in my case each 1% of a WU takes ~2 minutes to process, with the processor usage limit set to 50% it doubles to 4 minutes per 1%. You can place any percentage in that option depending on the load balancing you wish to use. On a quad core you could keep the client running on one core by setting 25%. Or you could set it to 50% and run the SMP client at the same time. I currently switch it back and forth. 100% when I am away from my computer and 50% plus one uni processor client while I am using the system. This method provides a minimal hit on system usability.
KDE 4.x notes with 1st Gen Wrapper
In the KDE 4.1.1 release with Special effects enabled, I have noticed ctrl + C is not as reponsive as it was in 4.0 and 4.1, sometimes taking up to five minutes to stop the client. I have been running the client at 95% CPU usage witn only a few seconds hit in performance but can now instantly stop the client with ctrl + C. I have also found my background apps function better (KOrganizer Calendar, System updates). The second generation wrapper eliminates the need for this.
Overclocking
To enable GPU overclocking, you need to add the "CoolBits" option to your X11 configuration file (for example /etc/X11/xorg.conf):
Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
Option "CoolBits" "1"
EndSection
Once enabled will you get access to the frequency settings in the NVIDIA X Server Settings:
user@box:~$ nvidia-settings
