It took me a while to find out how to poetry install in Docker container with VSCode. So, I wrote this article for other people.
Creating a Docker contaienr for development
If creating a docker content by VSCode, the following Dockerfile is created.
ARG VARIANT="3.10-bullseye"
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}
# [Choice] Node.js version: none, lts/*, 16, 14, 12, 10
ARG NODE_VERSION="none"
RUN if [ "${NODE_VERSION}" != "none" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi
RUN pip3 install --disable-pip-version-check --no-cache-dir poetry==1.2.2I added the last line to install poetry. I did it this way but it seems to be better to use the official installer.
ARG VARIANT="3.10-bullseye"
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}
# [Choice] Node.js version: none, lts/*, 16, 14, 12, 10
ARG NODE_VERSION="none"
RUN if [ "${NODE_VERSION}" != "none" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi
ENV POETRY_HOME="/opt/poetry" \
POETRY_VERSION=1.2.2
ENV PATH="$POETRY_HOME/bin:$PATH"
RUN curl -sSL https://install.python-poetry.org | python3 -The POETRY_HOME needs to be added to PATH. Otherwise, poetry can’t be executed because the user who installed poetry is a root user. When you open the project in the container, the user is vscode by default.
With the current setting, it installs packages to a project-specific folder but not to the system. It’s ok for now for the preparation.
Create pyproject.toml with dependencies
There is no dependency at the moment. Let’s create pyproject.toml file where poetry manages dependencies.
Execute the following command in your project.
poetry initThen, pyproject.toml file can be created interactively.
This command will guide you through creating your pyproject.toml config.
Package name [blogpost-python]:
Version [0.1.0]:
Description []: Python repository for learning
Author [Nakatsu Yuto <yuto.nakatsu@dmgmori-digital.com>, n to skip]: yuto-yuto
License []: MIT
Compatible Python versions [^3.10]:
Would you like to define your main dependencies interactively? (yes/no) [yes] pyproject.toml file looks as follows.
[tool.poetry]
name = "blogpost-python"
version = "0.1.0"
description = "Python repository for learning"
authors = ["yuto-yuto"]
license = "MIT"
[tool.poetry.dependencies]
python = "^3.10"
[tool.poetry.dev-dependencies]
mockito = "^1.4.0"
pytest = "^7.1.3"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"Once it’s created, install the dependencies by the following command.
poetry installThen poetry.lock file is created. It is like package-lock.json file in Node.js. It fixes the versions. With this file, you can always install the same dependency versions.
Install dependencies to the Docker container
To execute poetry install in the container, it’s necessary to pass copy pyproject.toml and poetry.lock. It seems that the project files are automatically copied after Docker build finishes. it means that the two files don’t exist in the build time. So you need to copy them explicitly in the following way.
COPY pyproject.toml poetry.lock /tmp/poetry/In addition to that, poetry installs dependencies to a project-specific folder to make it virtual by default. It is written on the official page.
It means that VSCode doesn’t recognize the dependencies and thus, you will see lots of red lines under your code.
Moreover, commands can’t be executed directly. You have to call poetry run pytest instead of pytest for example.
But I think it does not make sense when using Docker. If multiple python versions are needed, I think the container should be split. For this reason, I set false to virtualenvs.create.
ENV POETRY_HOME="/opt/poetry" \
POETRY_VERSION=1.2.2
ENV PATH="$POETRY_HOME/bin:$PATH"
RUN curl -sSL https://install.python-poetry.org | python3 -
COPY pyproject.toml poetry.lock /tmp/poetry/
WORKDIR /workspaces/blogpost-python
RUN poetry config virtualenvs.create false && poetry installremoteUser is specified as vscode by default in devcontainer.json. So you might want to set the user here with USER vscode but it causes a permission error.
Docker image needs to be rebuilt whenever dependencies are updated
virtualenvs.create is set to false in the Dockerfile but the setting is for a root user. If you install a new package, it is installed into a project-specific folder. It means that you have to execute the command poetry run something instead of something.
If you want to use the new package without poetry run right after the installation, you need to add poetry.toml with the following content.
[virtualenvs]
create = falseBut you must always add sudo when executing a poetry command due to a permission error.
poetry update fails due to permission error
The permission error occurs on /usr/local/lib/python3.x/site-packages and /usr/local/something for example.
When virtualenvs.create is false, poetry needs write access to the directories where poetry does something.
To solve the problem, you needed to choose one of them depending on virtualenv.create.
- True (default) -> always use a command with
poetry run. e.g.poetry run pytest - False -> always use a poetry command with sudo. e.g.
sudo poetry install
But it leads to less usability of poetry, so I tried the following things to solve it.
Create a link (NOT work)
I created a link to the executable from /usr/local/bin in order to let a root user use it.
RUN update-alternatives --install /usr/local/bin/poetry poetry /opt/poetry/bin/poetry 900But sudo needs to be added when executing a poetry command.
Changing the owner of the /usr/local directory (NOT work)
I tried to change the owner of /usr/local directory.
RUN chown -R vscode:vscode /usr/localbut the permission error still occurs.
$ ls -l /usr | grep local
drwxr-xr-x 1 vscode vscode 4096 Oct 6 23:37 local
$ poetry update
...
...
Command ['/usr/local/bin/python3.10', '-m', 'pip', 'install', '--disable-pip-version-check', '--prefix', '/usr/local', '--upgrade', '--no-deps', '/home/vscode/.cache/pypoetry/artifacts/36/d6/d1/a41ef20e6edb20284ff2c5801156e91aa458d2b96d4cce3ed8c8758d6e/astroid-2.12.11-py3-none-any.whl'] errored with the following return code 1, and output:
Processing /home/vscode/.cache/pypoetry/artifacts/36/d6/d1/a41ef20e6edb20284ff2c5801156e91aa458d2b96d4cce3ed8c8758d6e/astroid-2.12.11-py3-none-any.whl
Installing collected packages: astroid
Attempting uninstall: astroid
Found existing installation: astroid 2.12.10
Uninstalling astroid-2.12.10:
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'METADATA'
Consider using the `--user` option or check the permissions.Changing remoteUser to root works
We don’t actually have to use non-root user in Docker container. Then, this is the easiest solution. Change the remoteUser in devcontainer.json.
"remoteUser": "vscode"to
"remoteUser": "root"Then, it works as expected because the root user executes all commands. It can do anything.
However, this change might not be a good idea from the point of security view. This is a development environment, so it might be no problem but it’s always better to choose a more secure way.
Final version of Dockerfile
The final code of Dockerfile looks like this below.
ARG VARIANT="3.10-bullseye"
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}
ENV POETRY_HOME="/opt/poetry" \
POETRY_VERSION=1.2.2
RUN curl -sSL https://install.python-poetry.org | python3 - \
&& update-alternatives --install /usr/local/bin/poetry poetry /opt/poetry/bin/poetry 900 \
# Enable tab completion for bash
&& poetry completions bash >> /home/vscode/.bash_completion \
# Enable tab completion for Zsh
&& mkdir -p /home/vscode/.zfunc/ \
&& poetry completions zsh > /home/vscode/.zfunc/_poetry \
&& echo "fpath+=~/.zfunc\nautoload -Uz compinit && compinit" >> /home/vscode/.zshrc
COPY pyproject.toml poetry.lock /tmp/poetry/
WORKDIR /tmp/poetry
RUN poetry config virtualenvs.create false && poetry install \
&& rm -rf /tmp/poetryThe removeUser in devcontainer.json is vscode.
virtualenvs.create is set to false in poetry.toml.
[virtualenvs]
create = falseIf you want to add a new package, you must execute a poetry command with sudo.
In case where lock file is NOT needed
In my working project, poetry.lock file is not necessary. The dependencies for production have fixed versions but not for dev dependencies. Therefore, I ignored poetry.lock file.
ARG VARIANT="3.10-bullseye"
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}
ENV POETRY_HOME="/opt/poetry" \
POETRY_VERSION=1.2.2
RUN curl -sSL https://install.python-poetry.org | python3 - \
&& update-alternatives --install /usr/local/bin/poetry poetry /opt/poetry/bin/poetry 900 \
# Enable tab completion for bash
&& poetry completions bash >> /home/vscode/.bash_completion \
# Enable tab completion for Zsh
&& mkdir -p /home/vscode/.zfunc/ \
&& poetry completions zsh > /home/vscode/.zfunc/_poetry \
&& echo "fpath+=~/.zfunc\nautoload -Uz compinit && compinit" >> /home/vscode/.zshrcBut the build fails on the pipeline because the necessary tools are not in the Docker image. For example, it can’t execute pytest. The tools need somehow to be installed.
To solve the problem, I added the following code to the pipeline before building the Docker image.
echo COPY pyproject.toml /tmp/poetry/ >> Dockerfile
echo WORKDIR /tmp/poetry >> Dockerfile
echo RUN poetry config virtualenvs.create false && poetry install && poetry update >> DockerfileAll tools are installed in this way and the build in the pipeline succeeds.
Comments