CONDA configures the deep learning environment pytorch transformers

Preface

I want to learn about it recently Huggingface Of Transformers Library usage , Need to rebuild a virtual environment , Make a simple record for later work .

Text

This setup is mainly for testing and playing , Don't consider using a lower version compatible with XXX Application , Take a look at python Version of .

  It's almost a small version number a year ,3.7 Another year to stop maintenance , Intended use 3.8 了 .

conda create -n dev38 python=3.8

create Command creation environment . here miniconda Installation , also channel I won't repeat the setting of .

Tips conda You need to update , Update conveniently conda.

conda update -n base -c defaults conda

* python --version
Python 3.8.13

Look at the , Installed 3.8.13 Version of python.

Then install pytorch Besides, Transformers.PyTorch The installation of requires cudatoolkit To use the graphics card .

This machine has been installed before cudatoolkit, Check first .

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0

You can see that what is installed on this machine is cuda 11.1 .

however pytorch The official website doesn't seem to give this choice .

But take a look , It should be changed cudatoolkit That part is good .  Try installing first CUDA 11.1 .

conda install pytorch torchvision torchaudio cudatoolkit=11.1

Here is the -c The part of is removed , This part is about choosing the download source , In my impression pytorch Official sources are particularly slow , Try closing it here .

* conda install pytorch torchvision torchaudio cudatoolkit=11.1
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - cudatoolkit=11.1

Current channels:

  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/linux-64
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/noarch
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/linux-64
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/noarch
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro/linux-64
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro/noarch
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/linux-64
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/noarch
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/linux-64
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/noarch
  - https://mirrors.ustc.edu.cn/anaconda/pkgs/free/linux-64
  - https://mirrors.ustc.edu.cn/anaconda/pkgs/free/noarch
  - https://mirrors.ustc.edu.cn/anaconda/pkgs/main/linux-64
  - https://mirrors.ustc.edu.cn/anaconda/pkgs/main/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

Then glory reported wrong . Then try 11.1 Change to 11.3 .

  It's ready to install …… Just download a lot of things .

good , Can't use cuda, Probably because docker Of the external host driver Versions are also limited . Then we can only adjust cuda To 11.1 了 .

  I put the version back （ There is no -c Will report a mistake .

conda install cudatoolkit=11.1 -c nvidia

But I still can't find it gpu. Look at the , current torch The version is 1.10.2, In the usable environment installed before torch The version is 1.8.2, It's nothing more than torch Version of the problem ？

conda install pytorch=1.8.2 -c pytorch-lts

Back off again pytorch Version of , Note that there -c If pytorch Will make mistakes , Need to add lts （ Long term maintenance ）.

  And then you can use it cuda 了 …… It's probably a little new torch And what I use GPU Server's GPU Drive the five elements to disagree …… But because of GPU The server cannot update the driver version at will , May cause all users docker Collapse , So it can only be used like this LTS 了 .

Next install Transformers library .

conda install -c huggingface transformers

The installation here is also very smooth .

summary

Installing a deep learning environment is prone to various problems , If you have permission to update the graphics card driver of the server, you should be able to install a newer version of the Library , But like me in docker in , The scenario of sharing a server with many users , Maybe you can only use pytorch 1.8.2 Of LTS Version of the , Using the previous version will be acclimatized .