There are varieties of different Python virtual environment managers out there - conda, pipenv, poetry, virtualenv, etc. And from time to time, we would need to convert or switch between different virtual environment tools. For example, AWS SageMaker deals with specifically conda while most software engineering deals with pipenv, poetry, or virtualenv. In order to have interoperatible virtual environment specifications, we have to rely on requirements.txt format to ensure virtual environment reproducibility.

Using pip to install packages underneath conda will introduce undesirable namespace collusion problems thus it’s best to stick with using one single package manager. However, conda’s support for requirements.txt is extremely poor. The standard command as recommended by conda documentation:

conda install --file requirements.txt

will almost always incur some errors similar to the following:

CondaError: Error reading file, file should be a text file containing packages

To fix conda’s deficiencies, we can do the following algorithm:

while requirements.txt do
  if package in conda
    install the pacakage using conda
  else
    install the package using pip
end

The realization of the above algorithm using bash command is as follows.

while read requirement; do conda install python=3.7 --yes --force-reinstall -c conda-forge $requirement || pip install $requirement; done < requirements.txt

Alternatively, we can also use cat to pipe the requirements.txt to the while loop.

cat requirements.txt | while read requirement; do conda install python=3.7 --yes --force-reinstall -c conda-forge $requirement || pip install $requirement; done

To cite this content, please use:

@article{
    leehanchung,
    author = {Lee, Hanchung},
    title = {Syncing Conda Environments with requirements.txt},
    year = {2021},
    howpublished = {\url{https://leehanchung.github.io/}},
    url = {https://leehanchung.github.io/2021-08-04-conda-requirements/}
}