Making use of eddy_cuda of FSL 6.0.4 on Ubuntu 18.04

eddy implemented in FSL is time-consuming program. FSL recommends using eddy_cuda, GPU version of eddy. They ship eddy_cuda8.0 and edddy_cuda9.1. If you use Ubuntu 18.04, you can make use of eddy_cuda9.1 with only 4 commands.

Disclaimer: Installing nvidia-driver could cause display problem. I am not responsible for the problem…

Summary

Type the following 4 commands and you’re set to go.

sudo update-pciids
sudo ubuntu-drivers autoinstall
(reboot)
sudo apt install nvidia-cuda-toolkit
ln -sf ${FSLDIR}/bin/eddy_cuda9.1 ${FSLDIR}/bin/eddy_cuda

How to make use of eddy_cuda of FSL 6.0.4 on Ubuntu 18.04

running topup and eddy_openmp for comparison

I ran dwifslpreproc in mritrix3 to measure the processing time of topup and eddy for a dwi image with 30 axes.

real    17m25.986s
user    85m45.748s
sys 0m56.166s

It took 85min for one subject. It would be a long way to finish eddy for all subjects…

Download the latest PCI ID list

update-pciids will download the latest PCI ID list. This will help your PC identify the graphic board on your PC.

1	`sudo update-pciids`

Check your graphic board

The command below will show you installed graphic board.

1	`lspci \| grep -i nvidia`

Below is the result of mine. I use GeForce RTX 2080 Ti.

00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] (rev a1)
00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

Check GPU drivers

ubuntu-drivers devices shows you candidate drivers.

ubuntu-drivers devices
== /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0 ==
modalias : pci:v000010DEd00001E07sv000010DEsd000012FAbc03sc00i00
vendor   : NVIDIA Corporation
driver   : nvidia-driver-450 - distro non-free
driver   : nvidia-driver-460 - distro non-free recommended
driver   : nvidia-driver-450-server - distro non-free
driver   : nvidia-driver-418-server - distro non-free
driver   : xserver-xorg-video-nouveau - distro free builtin

As of Jan 22, 2021, nvidia-driver-460 is recommended. I decided to use this one.

In stall a GPU driver

ubuntu-drivers autoinstall will install the recommended driver.

1	`sudo ubuntu-drivers autoinstall`

Reboot

Reboot your machine to reflect the installation.

Check installation

You can check your driver installation by running nvidia-smi. nvidia-smi stands for “NVIDIA System Management Interface program.”

nvidia-smi
Fri Jan 22 19:53:41 2021      
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:01:00.0  On |                  N/A |
|  0%   42C    P8    22W / 260W |    230MiB / 11016MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
 
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1569      G   /usr/lib/xorg/Xorg                118MiB |
|    0   N/A  N/A      2602      G   ...gAAAAAAAAA --shared-files      106MiB |
+-----------------------------------------------------------------------------+

Note that it shows “CUDA Version: 11.2” though CUDA is not installed yet.

Installation of CUDA 9.1

FSL 6.0.4 ships eddy_cuda8.0 and eddy_cuda9.1. If you can install CUDA 8.0 or 9.1, you can use these programs. Ubuntu has nvidia-cuda-toolkit to install CUDA-related programs. Let’s check which version will be installed by apt show nvidia-cuda-toolkit.

apt show nvidia-cuda-toolkit
Package: nvidia-cuda-toolkit
Version: 9.1.85-3ubuntu1
Priority: extra
Section: multiverse/devel
Origin: Ubuntu
Maintainer: Ubuntu Developers &lt;ubuntu-devel-discuss@lists.ubuntu.com&gt;
Original-Maintainer: Debian NVIDIA Maintainers &lt;pkg-nvidia-devel@lists.alioth.debian.org&gt;
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 61.0 MB
Depends: nvidia-profiler (= 9.1.85-3ubuntu1), nvidia-cuda-dev (= 9.1.85-3ubuntu1), nvidia-opencl-dev (= 9.1.85-3ubuntu1) | opencl-dev, gcc-6 | clang-4.0 | clang-3.9 | clang-3.8 | clang (&lt;&lt; 1:5~) | gcc-5 | gcc-4.9 | gcc-4.8, g++-6 | clang-4.0 | clang-3.9 | clang-3.8 | clang (&lt;&lt; 1:5~) | g++-5 | g++-4.9 | g++-4.8, libc6 (&gt;= 2.4), libgcc1 (&gt;= 1:3.0), libnvvm3 (&gt;= 7.0), libstdc++6 (&gt;= 4.1.1)

We see cuda 9.1 will be installed, so we will use this.

1	`sudo apt install nvidia-cuda-toolkit`

Check version of CUDA

nvcc -V will show you which version of CUDA is used.

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85

We see CUDA 9.1 is used.

Prepare eddy_cuda

eddyのUsersGuide describes how to prepare eddy_cuda. We prepare symbolic link for eddy_cuda9.1. You need to use sudo to make a symbolic link.

1	`sudo ln -sf ${FSLDIR}/bin/eddy_cuda9.1 ${FSLDIR}/bin/eddy_cuda`

Now you’re ready to go!

Run topup and eddy_cuda

I ran dwifslpreproc for comparison with eddy_openmp

real    7m2.535s
user    7m3.179s
sys 0m16.221s

It’s 12 times faster!