Linux
From GPU2 Linux
Contents |
Prerequisites
Nvidia Video Card
This only works for Nvidia users who have a CUDA compatible video card. The CUDA technology first was introduced on the Nvidia G80 based cards, GeForce 8x00, and has continued support on all later Nvidia GPU generations to date. Embedded GPU's such as the 8100, 8200, and Ion will most likely not make the deadlines. An 8600/9600 card is a good entry level card and will make the deadlines.
Nvidia Binary Driver with CUDA Support
You will need to install a driver with CUDA support. For Ubuntu 9.04 and earlier please use a 180.x Nvidia driver. Later drivers have a CUDA bug which stops the wrapper from working.
For Ubuntu 9.10 and later please use a 195.x Nvidia driver. Earlier drivers both break support for the wrapper and will not work with the included kernel version.
Click here to search for a driver - you may need to select Beta to get the correct version.
And you will need the CUDA Toolkit which can be found here:
http://www.nvidia.com/object/cuda_get.html
If you have a 64-bit Kernel you can install the 64-bit driver. Regardless of which kernel you are using you MUST install the 32-bit Toolkit. This is a compatibility issue with Wine and the CUDA wrapper. Any CUDA 2.x toolkit will work. That page tends to lag behind a version, so if you are running Ubuntu 9.10 it is okay to select 9.04.
Installation of Wine
If you do not have Wine installed, you will need to install it. See the Winehq for distro specific methods of installation.
Only version of wine 0.9.45 or greater have been test for compatibility.
CUDA Wrapper
Wrapper
Second Generation Wrapper
This wrapper has a built in delay. It was found that F@H is using some sort of busy loop which was causing 100% CPU usage and major user interface slow downs. Very little real work is being done during this loop so a delay was added that brings the CPU usage down much closer to what the client actually needs for feeding the GPU. This delay is for 300 microseconds and may not work correctly on all systems. If you notice instability, client running slower than it should, or any other anomalies, then please use the first generation wrapper.
Note: you will need a high resolution timer in your Kernel for this delay to work correctly. Run
cat /proc/timer_list
and look for "hrtimer_interrupt" and ".hres_active: 1", if those are present then you should be all set.
The current kernels will fail to enable the system's high resolution timers when support for the "CPU C1E state" is enabled. Disabling the support of the C1E state in the BIOS can solve this.
Wrapper 2.1 is needed for F@H core 14. This will now replace all the linked wrappers below. Please use this one for all current GPU2 cores.
Download the 2.1 wrapper cudart.dll.so
First Generation Wrapper
Wrapper Save Location
Place the cudart.dll.so file in this directory and rename it to cudart.dll
/home/your_user/.wine/drive_c/windows/system32
Then add a link to that file named nvcuda.dll and place it in the same directory using you favorite file manager or with the following command while in the system32 directory
ln -s cudart.dll nvcuda.dll
- Need to add source code and compile instructions to make more OSS friendly
Linking The Toolkit
You will need to tell the system that the CUDA toolkit is installed and make sure the cudart.dll.so file can find its required dependencies. Add /usr/local/cuda/lib to /etc/ld.so.conf and run as root
ldconfig
Then run
ldd /home/your_user/.wine/drive_c/windows/system32/cudart.dll
If you see any "not founds" then something has been setup wrong. Commonly we have seen this
user@box:~$ ldd /home/my_user/.wine/drive_c/windows/system32/cudart.dll linux-gate.so.1 => (0xffffe000) libcudart.so.2 => not found libwine.so.1 => /usr/local/lib/libwine.so.1 (0xf7e4b000) libm.so.6 => /lib32/libm.so.6 (0xf7e25000) libc.so.6 => /lib32/libc.so.6 (0xf7cd6000) libdl.so.2 => /lib32/libdl.so.2 (0xf7cd2000) /lib/ld-linux.so.2 (0xf7fac000)
If this happens then your CUDA toolkit has not been installed correctly or you installed the 64-bit version. You must install the 32-bit version of the toolkit regardless of what your system is. Remember this is independent of the Nvidia driver which can be 64-bit.
Also make sure the ldconfig method used above is the same for your distro. This method is currently used for Ubuntu.
Note: The LD_LIBRARY_PATH environment variable can be used as an alternative. Set it in your login script or before starting Wine.
user@box:~$ export LD_LIBRARY_PATH=/usr/local/cuda/lib:$LD_LIBRARY_PATH
Running The Client
Download Client
Go to the link and download the Windows Folding@Home console client.
http://folding.stanford.edu/English/DownloadWinOther
Extract the zip file into a new folder. Open a console, go to the folder you extracted the client to, and then run the client.
wine Folding@home-Win32-GPU.exe -forcegpu nvidia_g80
The first time you run this you should be prompted for your user information like with the other console clients. This version will accept the same console flags such as -config or -configonly. When running the config, set the priority to low. If you are also running a CPU client, set the CPU client priority to idle. Since the GPU client uses only a few percent it should minimally impact the CPU client while allowing the GPU client to get better PPD.
To stop the client, have the console selected and press
crtl + C
Important
Please read all of this before continuing to use the client.
Client Instability
I have been noticing the Fahcore_11.exe process still running after I stop the client. After you have stopped the client check your system monitor, or top, for running processes and kill the fahcore process if it is still running. The WU seems to start back up without an issue after doing this, but if you leave this process running and try to start GPU2 again then you will make your current WU EUE as well as all other WU's the client tries to start. The client will reach a limit eventually and freeze all further activity for 24 hours, or until the client has been restarted. Please be responsible and check for this issue so we can limit the amount of EUE's sent back.
3D desktop effects (compositing) may need to be disabled while running the client. This includes Compiz-Fusion and KDE 4 special effects or anything else like it. With a 3D desktop enabled I have seen random EUE's and Xorg reset after about ten minutes of use. This may no longer be the case with Nvidia driver 177.67, after installing this driver I have had no problems keeping KDE compositing on while running the GPU client, YMMV so keep an eye out for issues.
A switch to a login console (for example with ALT+F1), a X server shutdown or a X session logout can cause the core to hang.
System Lock Up
Only applies to the first generation wrapper.
For some reason while the client is running there are two processes taking up a large portion of processing power, Xorg and Fahcore_11.exe. The fahcore is a given but we can't seem to figure out why Xorg is using so much power. The assumption is that Xorg may be waiting on the Nvidia driver to finish what it is doing. On my dual core processor each process gets 50% CPU time or one core each. This causes the system to be unusable while the client is running. Ctrl + C to stop the client makes the system usable again. There have been reports that on Quad core processors this is not so much the case.
Another option is to edit the file
client.cfg
located in your F@H client folder. Then edit the "core" line
[core] cpuusage=100
and change it to
[core] cpuusage=50
(If you do not have this file in your F@H client folder or it doesn't contain a [core] section, run the client with the "-configonly" flag, answer the questions, and say "yes" to the question "Change advanced options?," and continue to answer all questions - this will generate a config.cfg with a [core] section). This will only let the client use up to 50% CPU power. This will increase your frame time, so in my case each 1% of a WU takes ~2 minutes to process, with the processor usage limit set to 50% it doubles to 4 minutes per 1%. You can place any percentage in that option depending on the load balancing you wish to use. On a quad core you could keep the client running on one core by setting 25%. Or you could set it to 50% and run the SMP client at the same time. I currently switch it back and forth. 100% when I am away from my computer and 50% plus one uni processor client while I am using the system. This method provides a minimal hit on system usability.
KDE 4.x notes with 1st Gen Wrapper
In the KDE 4.1.1 release with Special effects enabled, I have noticed ctrl + C is not as reponsive as it was in 4.0 and 4.1, sometimes taking up to five minutes to stop the client. I have been running the client at 95% CPU usage witn only a few seconds hit in performance but can now instantly stop the client with ctrl + C. I have also found my background apps function better (KOrganizer Calendar, System updates). The second generation wrapper eliminates the need for this.
Overclocking
To enable GPU overclocking, you need to add the "CoolBits" option to your X11 configuration file (for example /etc/X11/xorg.conf):
Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
Option "CoolBits" "1"
EndSection
Once enabled will you get access to the frequency settings in the NVIDIA X Server Settings:
user@box:~$ nvidia-settings
Multi-GPU Setup
Using multiple GPU's works the same way as it does in Windows. Make a new GPU2 folder, and follow the standard rules for running multiple clients, make sure your system ID is different for both clients. Add the -gpu N flag to the command when starting the client, where N is the number GPU you want to use for this client.
nice -n 19 wine Folding@home-Win32-GPU.exe -forcegpu nvidia_g80 -gpu 0
And the second client can be started with
nice -n 19 wine Folding@home-Win32-GPU.exe -forcegpu nvidia_g80 -gpu 1
The GPU number corresponds to the GPU number in nvidia-settings.
Unlike Windows, there is no need for monitors or "dummy plugs" to be connected to any of the VGA/DVI outputs.
GPU Pairing Rules
The same rules apply to CUDA in Linux as it does to CUDA in Windows when pairing GPU's. Ideally use the same GPU in all cards. If you must mismatch GPU's try to use GPU's that have integer multiples of stream processors (shaders) compared with each other. Such as if one card has 128 stream processors and the other has 64, this should be fine. If you do not follow this rule while mismatching then one card will run slower than the other. In the 170 drivers the faster card would run slower. In the 180 drivers the slower card now runs slower.
Do not use SLI mode while folding. When in SLI the cards appear as one card to the system but CUDA will only use one core.
Multi-GPU Xorg Config Howto
If you are having problems getting Xorg to see two or more cards here is a small howto to get you started.
The Nvidia driver only seems to see one GPU when being installed. If you let the driver setup your xorg.conf file you will see only one device present
Section "Device"
Identifier "8800GT 512"
Driver "nvidia"
VendorName "NVIDIA Corporation"
EndSection
You will need to manually add your other device or devices. The key part that lets the driver know one GPU from the next is the BusID. The Identifier can be what ever you want and the Driver must be "nvidia", if it is set to "nv" then you are not using the Nvidia supplied CUDA capable binary driver and this will need to be changed to "nvidia".
To find the BusID of your GPU's run:
lspci
Search for your GPU's in that list and take note of their pci bus ID's. Then add the bus ID for you first card and create another device entry in your /etc/X11/xorg.conf file with the bus ID for the second device.
Example
Section "Device"
Identifier "8800GT 512"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BusID "1:0:0"
EndSection
Section "Device"
Identifier "8800GT 512"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BusID "5:0:0"
EndSection
Running:
nvidia-settings
Should now show multiple GPU's.
