multiple parallel environments

[coldfix]

Hi,

thanks for this wonderful tool!! It's well crafted and extremely useful, but I feel that there is exactly one feature missing to make it just perfect for a workflow I frequently encounter while developing python packages:

One often needs to switch python versions in order to test behaviour on different python versions. Technically, pipenv --python=XXX can be used to switch between different python versions. However, this replaces the current environment and is therefore inconveniently slow in practice.

I would really like to be able to install multiple environments in parallel for the same project and select which one to use on the pipenv command line. This can speed up the time required to switch python versions for the same project significantly and therefore we would have a powerful (IMO alround better) replacement for virtualenvwrapper and the like.

This could work by e.g. providing an environment name on the command line:

# setup environments
pipenv --name 35 --python 3.5 install
pipenv --name 36 --python 3.6 install

# run commands
pipenv --name 35 run python

One issue could be that of different lockfiles in different python versions..

If you would be fine with such an option and think that it is possible to implement for s.o. unacquainted with the code, I'm of course willing to help on the implementation.

This issue is related to #368 and #137 who I think were trying to ask the same question, but I'm unsure whether you rejected the idea altogether or were happy with an intermediate solution at the time.

Best, Thomas

[techalchemy]

For automated testing many people use a pipeline similar to what you would find in our Travis config file. However you also have the option to use your own virtualenv when running pipenv. Most people don’t handle this by hand but if you do you probably want to make your own virtualenv for each or use pyenv

[techalchemy]

Thanks for the suggestion but deterministic virtualenv locations is an inherent part of what pipenv is, so I don't see us implementing this. I'm gonna close this out for now!

[coldfix]

Thanks for the answer.

For automated testing many people use a pipeline similar to what you would find in our Travis config file.

I was mainly thinking about interactive use, i.e. debugging, or stuff that's hard to test automatically (GUI, look&feel).

Thanks for the suggestion but deterministic virtualenv locations is an inherent part of what pipenv is, so I don't see us implementing this.

location can still be deterministic, based on path+name, so there is no real conflict here.

so I don't see us implementing this. I'm gonna close this out for now!

Fair enough, thanks anyway!

[alecbz]

I also don't understand what conflict this would create with deterministic virtualenv locations.

[jtratner]

To be super clear, you can still get your own custom environments set up just by sourcing virtualenvs.

virtualenv 35 --python=python3.5
virtualenv 36 --python=python3.6
source 35/bin/activate && pipenv install
source 36/bin/activate && pipenv install
source 35/bin/activate && pipenv run <whatever>

a tiny bit of additional visual clutter to the commands but is pretty straightforward

[alecbz]

Interesting, so if you run pipenv while already in a virtualenv, it will just use that virtualenv, and ignore the one it would use otherwise?

[techalchemy]

Yup

[moloney]

The workaround kind of defeats the point of using pipenv...

The argument about deterministic virtualenv locations is bunk, the python version would just be another factor in automatically selecting the correct virtualenv.

I am not saying there isn't possibly some valid technical reason for rejecting this feature, but it would be nice to actually know what it is.

[uranusjr]

@moloney Honest question out of curiosity: How would you design an interface if we are to start from the ground up for this one? How would you associate virtual environments to a given project without cluttering the usual workflow with --name arguments?

I ask because I’ve been looking into building my own home-baked Pipfile + virtual environment tool, and collecting feedbacks for a while. Any feedback would help me greatly toward my goal.

[moloney]

@uranusjr I should have been more precise. I agree that adding a --name option is probably too much. Instead of adding a --name option I would change the lookup behavior to use both the project and the python version to lookup the virtualenv to use.

I can understand not wanting to recreate the functionality of tox, but it is nice to be able to interactively switch between python versions and try things out without waiting for all the dependencies to reinstall.

[uranusjr]

That makes sense. I’ll ponder on this a bit, and maybe I can even come up with a backward-compatible way to do this in Pipenv.

[Cito]

Another use case for this is when I want to work on the same project in a shared folder used by different virtual machines. Currently I'm using the approach suggested by @jtratner.

[NicolasDutronc]

I have another use case: I'm working on a project with pytorch. The issue here is that the installation of pytorch is os specific, thus when I switch between linux and mac, the pytorch version is incompatible. So I was looking for a way to create an environment for each os.

[nathanielcompton]

@uranusjr I realize this issue has been closed for some time now, but I'm wondering if there is any progress regarding a solution/feature? If so, would you please point me in the right direction?

[uranusjr]

@nathanielcompton There is a (relatively recent) feature that adds support of an in-project .venv file to tell Pipenv to find the environment at a certain location. You can do things like this:

# Initialize the project.
mkdir my-project
cd my-project
echo "" > Pipfile

# Create some environments.
python3.5 -m venv .venv-3.5 --prompt=my-project
python3.6 -m venv .venv-3.6 --prompt=my-project

Now you can switch environments like this:

$ echo ./.venv-3.6 > .venv
$ pipenv --venv
/path/to/my/project/.venv-3.6
$ echo ./.venv-3.5 > .venv
$ pipenv --venv
/path/to/my/project/.venv-3.5

(Note: The ./ part is required. A string without path separators is interpreted as a directory name inside WORKON_HOME.)

So the functionality is there, all what’s left is to design a good UI around it. My suggestion would be to develop a standalone tool (called pipenv-switch, maybe?) and try figuring out how things can be wired together. Once it’s done, it’d be easy to propose a PEEP to try adding it into Pipenv proper.

[Jamim]

Hello @uranusjr,
A feature that you mentioned above looks like the reinventing of symlinks.
Is it a workaround for Windows or are there any hidden benefits from using this approach on UNIX-like operating systems?

[uranusjr]

Windows is certainly an important reason behind this design, where symlinking is not viable in general.

[HaaiHenkie]

Hello all,

Like @coldfix I am using pipenv for testing purposes. I tried @jtratner's solution, but then you use the same pipfile, while I had different requirements for my python 2.7 environment: an old wxPython version that is not pip installable.

I found a solution involving the PIPENV_PIPFILE environment variable. I am sharing it here in the hope it can be useful to others.

For the sake of this example let's say you are working under windows and your project folder is at: %USERPROFILE%\pythonprojects\project. Supposed that you need python 2.7 and 3.7 test environments, create subdirs 27 en 37 here. You need to put an initial Pipfile in these directories: they will not be created automatically.

For switching environment I wrote the batch file activate.cmd and put it somewhere in my windows path.

@if "%1"=="" (set PIPENV_PIPFILE=) else (set PIPENV_PIPFILE=%USERPROFILE%\pythonprojects\project\%1\Pipfile)

Now you can install your virtual environments and start using them:

activate 27 && pipenv --python C:\Python27\python.exe --site-packages install
activate 37 && pipenv --python 3.7 install
activate 27 && pipenv run pip list
activate 37 && python -m pipenv shell

If you are using activate in a batch file, use call activate.

If you need to use a Pipfile in the root of your project for distribution purposes, you can switch to that Pipfile by activate without any argument.

Cheers,

Henk

[amard33p]

Hello all,

Like @coldfix I am using pipenv for testing purposes. I tried @jtratner's solution, but then you use the same pipfile, while I had different requirements for my python 2.7 environment: an old wxPython version that is not pip installable.

I found a solution involving the PIPENV_PIPFILE environment variable. I am sharing it here in the hope it can be useful to others.

For the sake of this example let's say you are working under windows and your project folder is at: %USERPROFILE%\pythonprojects\project. Supposed that you need python 2.7 and 3.7 test environments, create subdirs 27 en 37 here. You need to put an initial Pipfile in these directories: they will not be created automatically.

For switching environment I wrote the batch file activate.cmd and put it somewhere in my windows path.

@if "%1"=="" (set PIPENV_PIPFILE=) else (set PIPENV_PIPFILE=%USERPROFILE%\pythonprojects\project\%1\Pipfile)

Now you can install your virtual environments and start using them:

activate 27 && pipenv --python C:\Python27\python.exe --site-packages install
activate 37 && pipenv --python 3.7 install
activate 27 && pipenv run pip list
activate 37 && python -m pipenv shell

If you are using activate in a batch file, use call activate.

If you need to use a Pipfile in the root of your project for distribution purposes, you can switch to that Pipfile by activate without any argument.

Cheers,

Henk

But this will always recreate the virtualenvs from scratch.

[HaaiHenkie]

But this will always recreate the virtualenvs from scratch.

Hi @amard33p,

That is not correct. The first two command lines create a virtualenv per PIPENV_PIPFILE environment variable. So no need to repeat those steps every time you want to switch virtualenv. Actually the example demonstrates it. On the third and fourth line you switch back to PIPENV_PIPFILE 27 and 37 respectively. There is no recreation of the virtualenvs there.