Chatmail self-hosted deployment never get success

duetg · December 15, 2023, 3:00pm

Hi there, I followed the instructions on Chatmail repo and tried to deploy self-hosted Chatmail server on an Ubuntu server 22.04 by another Ubuntu server yesterday and today eralier. But it never got success. It always failed at the step 4:

scripts/cmdeploy run

The following is the output. Has anyone encountered such a situation?

Command Line Output

root@xxx-xxx:~/chatmail# scripts/cmdeploy run
[$ pyinfra --ssh-user root chat.xxxxx.com /root/chatmail/cmdeploy/src/cmdeploy/deploy.py]

--> Loading config...

--> Loading inventory...

--> Connecting to hosts...
    [chat.xxxxx.com] Connected

--> Preparing Operations...
    Loading: /root/chatmail/cmdeploy/src/cmdeploy/deploy.py
--> An exception occurred in: /root/chatmail/cmdeploy/src/cmdeploy/deploy.py:

  File "/root/chatmail/venv/lib/python3.10/site-packages/pyinfra_cli/util.py", line 60, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "/root/chatmail/cmdeploy/src/cmdeploy/deploy.py", line 17, in <module>
    main()
  File "/root/chatmail/cmdeploy/src/cmdeploy/deploy.py", line 13, in main
    deploy_chatmail(config_path)
  File "/root/chatmail/cmdeploy/src/cmdeploy/__init__.py", line 380, in deploy_chatmail
    check_config(config)
  File "/root/chatmail/cmdeploy/src/cmdeploy/__init__.py", line 367, in check_config
    if any(x in value for x in blocked_words):
  File "/root/chatmail/cmdeploy/src/cmdeploy/__init__.py", line 367, in <genexpr>
    if any(x in value for x in blocked_words):
TypeError: argument of type 'int' is not iterable

Traceback (most recent call last):
  File "/root/chatmail/venv/bin/cmdeploy", line 8, in <module>
    sys.exit(main())
  File "/root/chatmail/cmdeploy/src/cmdeploy/cmdeploy.py", line 305, in main
    res = args.func(args, out, **kwargs)
  File "/root/chatmail/cmdeploy/src/cmdeploy/cmdeploy.py", line 59, in run_cmd
    out.check_call(cmd, env=env)
  File "/root/chatmail/cmdeploy/src/cmdeploy/cmdeploy.py", line 229, in check_call
    return subprocess.check_call(arg, shell=True, env=env)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'pyinfra --ssh-user root chat.xxxxx.com /root/chatmail/cmdeploy/src/cmdeploy/deploy.py' returned non-zero exit status 1.

duetg · December 15, 2023, 6:00pm

I reinstall the target machine into Debian 12, and use a Debian 12 virt-env to deploy chatmail. Still no good.
Here is the output of scripts/initenv.sh

root@debian:~/chatmail# scripts/initenv.sh
Obtaining file:///root/chatmail/chatmaild
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting aiosmtpd
  Using cached aiosmtpd-1.4.4.post2-py3-none-any.whl (154 kB)
Collecting iniconfig
  Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Collecting deltachat-rpc-server
  Using cached deltachat_rpc_server-1.131.4-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl (8.3 MB)
Collecting deltachat-rpc-client
  Using cached deltachat_rpc_client-1.127.0-py3-none-any.whl (26 kB)
Collecting atpublic
  Using cached atpublic-4.0-py3-none-any.whl (4.9 kB)
Collecting attrs
  Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Building wheels for collected packages: chatmaild
  Building editable for chatmaild (pyproject.toml) ... done
  Created wheel for chatmaild: filename=chatmaild-0.2-0.editable-py3-none-any.whl size=1522 sha256=e1b2615cb0a8ae8d9cf58377df9f95fb1714fb6f566c70af849a8d7ad2c15563
  Stored in directory: /tmp/pip-ephem-wheel-cache-a9enw36r/wheels/da/09/32/753913979ac85f148b55c6488f445ca4c964434ac4c4ff0e7c
Successfully built chatmaild
Installing collected packages: deltachat-rpc-server, deltachat-rpc-client, iniconfig, attrs, atpublic, aiosmtpd, chatmaild
Successfully installed aiosmtpd-1.4.4.post2 atpublic-4.0 attrs-23.1.0 chatmaild-0.2 deltachat-rpc-client-1.127.0 deltachat-rpc-server-1.131.4 iniconfig-2.0.0
Obtaining file:///root/chatmail/cmdeploy
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting pyinfra
  Using cached pyinfra-2.8-py2.py3-none-any.whl (249 kB)
Collecting pillow
  Using cached Pillow-10.1.0-cp311-cp311-manylinux_2_28_aarch64.whl (3.5 MB)
Collecting qrcode
  Using cached qrcode-7.4.2-py3-none-any.whl (46 kB)
Collecting markdown
  Using cached Markdown-3.5.1-py3-none-any.whl (102 kB)
Collecting pytest
  Using cached pytest-7.4.3-py3-none-any.whl (325 kB)
Collecting setuptools>=68
  Using cached setuptools-69.0.2-py3-none-any.whl (819 kB)
Collecting termcolor
  Using cached termcolor-2.4.0-py3-none-any.whl (7.7 kB)
Collecting build
  Using cached build-1.0.3-py3-none-any.whl (18 kB)
Collecting tox
  Using cached tox-4.11.4-py3-none-any.whl (153 kB)
Collecting ruff
  Using cached ruff-0.1.8-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (6.1 MB)
Collecting black
  Using cached black-23.12.0-py3-none-any.whl (194 kB)
Collecting pytest-xdist
  Using cached pytest_xdist-3.5.0-py3-none-any.whl (42 kB)
Collecting click>=8.0.0
  Using cached click-8.1.7-py3-none-any.whl (97 kB)
Collecting mypy-extensions>=0.4.3
  Using cached mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Collecting packaging>=22.0
  Using cached packaging-23.2-py3-none-any.whl (53 kB)
Collecting pathspec>=0.9.0
  Using cached pathspec-0.12.1-py3-none-any.whl (31 kB)
Collecting platformdirs>=2
  Using cached platformdirs-4.1.0-py3-none-any.whl (17 kB)
Collecting aiohttp>=3.7.4
  Using cached aiohttp-3.9.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB)
Collecting pyproject_hooks
  Using cached pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)
Collecting gevent>=1.5
  Using cached gevent-23.9.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.9 MB)
Collecting paramiko<3,>=2.7
  Using cached paramiko-2.12.0-py2.py3-none-any.whl (213 kB)
Collecting colorama<1
  Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting jinja2<4,>2
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting python-dateutil<3,>2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting configparser
  Using cached configparser-6.0.0-py3-none-any.whl (19 kB)
Collecting pywinrm
  Using cached pywinrm-0.4.3-py2.py3-none-any.whl (44 kB)
Collecting distro<2,>=1.6
  Using cached distro-1.8.0-py3-none-any.whl (20 kB)
Requirement already satisfied: iniconfig in ./venv/lib/python3.11/site-packages (from pytest->cmdeploy==0.2) (2.0.0)
Collecting pluggy<2.0,>=0.12
  Using cached pluggy-1.3.0-py3-none-any.whl (18 kB)
Collecting execnet>=1.1
  Using cached execnet-2.0.2-py3-none-any.whl (37 kB)
Collecting typing-extensions
  Using cached typing_extensions-4.9.0-py3-none-any.whl (32 kB)
Collecting pypng
  Using cached pypng-0.20220715.0-py3-none-any.whl (58 kB)
Collecting cachetools>=5.3.1
  Using cached cachetools-5.3.2-py3-none-any.whl (9.3 kB)
Collecting chardet>=5.2
  Using cached chardet-5.2.0-py3-none-any.whl (199 kB)
Collecting filelock>=3.12.3
  Using cached filelock-3.13.1-py3-none-any.whl (11 kB)
Collecting pyproject-api>=1.6.1
  Using cached pyproject_api-1.6.1-py3-none-any.whl (12 kB)
Collecting virtualenv>=20.24.3
  Using cached virtualenv-20.25.0-py3-none-any.whl (3.8 MB)
Requirement already satisfied: attrs>=17.3.0 in ./venv/lib/python3.11/site-packages (from aiohttp>=3.7.4->black->cmdeploy==0.2) (23.1.0)
Collecting multidict<7.0,>=4.5
  Using cached multidict-6.0.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (120 kB)
Collecting yarl<2.0,>=1.0
  Using cached yarl-1.9.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (324 kB)
Collecting frozenlist>=1.1.1
  Using cached frozenlist-1.4.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (273 kB)
Collecting aiosignal>=1.1.2
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting zope.event
  Using cached zope.event-5.0-py3-none-any.whl (6.8 kB)
Collecting zope.interface
  Using cached zope.interface-6.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (249 kB)
Collecting greenlet>=3.0rc3
  Using cached greenlet-3.0.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (654 kB)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (28 kB)
Collecting bcrypt>=3.1.3
  Using cached bcrypt-4.1.2-cp39-abi3-manylinux_2_28_aarch64.whl (695 kB)
Collecting cryptography>=2.5
  Using cached cryptography-41.0.7-cp37-abi3-manylinux_2_28_aarch64.whl (4.1 MB)
Collecting pynacl>=1.0.1
  Using cached PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (601 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting distlib<1,>=0.3.7
  Using cached distlib-0.3.8-py2.py3-none-any.whl (468 kB)
Collecting xmltodict
  Using cached xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)
Collecting requests>=2.9.1
  Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Collecting requests-ntlm>=1.1.0
  Using cached requests_ntlm-1.2.0-py3-none-any.whl (6.0 kB)
Collecting cffi>=1.12
  Using cached cffi-1.16.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (466 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (136 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.6-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1
  Using cached urllib3-2.1.0-py3-none-any.whl (104 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2023.11.17-py3-none-any.whl (162 kB)
Collecting pyspnego>=0.1.6
  Using cached pyspnego-0.10.2-py3-none-any.whl (129 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Building wheels for collected packages: cmdeploy
  Building editable for cmdeploy (pyproject.toml) ... done
  Created wheel for cmdeploy: filename=cmdeploy-0.2-0.editable-py3-none-any.whl size=1527 sha256=2becd99ab6be57c7248e76b935b3b44696ef1e307ba3e5efec0eca1cc1d5c1d3
  Stored in directory: /tmp/pip-ephem-wheel-cache-qm5vf5s1/wheels/c5/54/3a/aaec7f8f5af0db2fd23f6997913a289c044eaf92513fc88f2b
Successfully built cmdeploy
Installing collected packages: pypng, distlib, xmltodict, urllib3, typing-extensions, termcolor, six, setuptools, ruff, pyproject_hooks, pycparser, pluggy, platformdirs, pillow, pathspec, packaging, mypy-extensions, multidict, MarkupSafe, markdown, idna, greenlet, frozenlist, filelock, execnet, distro, configparser, colorama, click, charset-normalizer, chardet, certifi, cachetools, bcrypt, zope.interface, zope.event, yarl, virtualenv, requests, qrcode, python-dateutil, pytest, pyproject-api, jinja2, cffi, build, aiosignal, tox, pytest-xdist, pynacl, gevent, cryptography, aiohttp, pyspnego, paramiko, black, requests-ntlm, pywinrm, pyinfra, cmdeploy
  Attempting uninstall: setuptools
    Found existing installation: setuptools 66.1.1
    Uninstalling setuptools-66.1.1:
      Successfully uninstalled setuptools-66.1.1
Successfully installed MarkupSafe-2.1.3 aiohttp-3.9.1 aiosignal-1.3.1 bcrypt-4.1.2 black-23.12.0 build-1.0.3 cachetools-5.3.2 certifi-2023.11.17 cffi-1.16.0 chardet-5.2.0 charset-normalizer-3.3.2 click-8.1.7 cmdeploy-0.2 colorama-0.4.6 configparser-6.0.0 cryptography-41.0.7 distlib-0.3.8 distro-1.8.0 execnet-2.0.2 filelock-3.13.1 frozenlist-1.4.1 gevent-23.9.1 greenlet-3.0.2 idna-3.6 jinja2-3.1.2 markdown-3.5.1 multidict-6.0.4 mypy-extensions-1.0.0 packaging-23.2 paramiko-2.12.0 pathspec-0.12.1 pillow-10.1.0 platformdirs-4.1.0 pluggy-1.3.0 pycparser-2.21 pyinfra-2.8 pynacl-1.5.0 pypng-0.20220715.0 pyproject-api-1.6.1 pyproject_hooks-1.0.0 pyspnego-0.10.2 pytest-7.4.3 pytest-xdist-3.5.0 python-dateutil-2.8.2 pywinrm-0.4.3 qrcode-7.4.2 requests-2.31.0 requests-ntlm-1.2.0 ruff-0.1.8 setuptools-69.0.2 six-1.16.0 termcolor-2.4.0 tox-4.11.4 typing-extensions-4.9.0 urllib3-2.1.0 virtualenv-20.25.0 xmltodict-0.13.0 yarl-1.9.4 zope.event-5.0 zope.interface-6.1

Here is the output of scripts/cmdeploy run

root@debian:~/chatmail# scripts/cmdeploy run
[$ pyinfra --ssh-user root chat.xxxxx.com /root/chatmail/cmdeploy/src/cmdeploy/deploy.py]

--> Loading config...

--> Loading inventory...

--> Connecting to hosts...
    [chat.xxxxx.com] Connected

--> Preparing Operations...
    Loading: /root/chatmail/cmdeploy/src/cmdeploy/deploy.py
--> An exception occurred in: /root/chatmail/cmdeploy/src/cmdeploy/deploy.py:

  File "/root/chatmail/venv/lib/python3.11/site-packages/pyinfra_cli/util.py", line 60, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "/root/chatmail/cmdeploy/src/cmdeploy/deploy.py", line 17, in <module>
    main()
  File "/root/chatmail/cmdeploy/src/cmdeploy/deploy.py", line 13, in main
    deploy_chatmail(config_path)
  File "/root/chatmail/cmdeploy/src/cmdeploy/__init__.py", line 380, in deploy_chatmail
    check_config(config)
  File "/root/chatmail/cmdeploy/src/cmdeploy/__init__.py", line 367, in check_config
    if any(x in value for x in blocked_words):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/chatmail/cmdeploy/src/cmdeploy/__init__.py", line 367, in <genexpr>
    if any(x in value for x in blocked_words):
           ^^^^^^^^^^
TypeError: argument of type 'int' is not iterable

Traceback (most recent call last):
  File "/root/chatmail/venv/bin/cmdeploy", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/chatmail/cmdeploy/src/cmdeploy/cmdeploy.py", line 305, in main
    res = args.func(args, out, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/chatmail/cmdeploy/src/cmdeploy/cmdeploy.py", line 59, in run_cmd
    out.check_call(cmd, env=env)
  File "/root/chatmail/cmdeploy/src/cmdeploy/cmdeploy.py", line 229, in check_call
    return subprocess.check_call(arg, shell=True, env=env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'pyinfra --ssh-user root chat.xxxxx.com /root/chatmail/cmdeploy/src/cmdeploy/deploy.py' returned non-zero exit status 1.

missytake · December 15, 2023, 7:25pm

Ah, I found the error and should have fixed it. Can you run git pull origin main and try again?

Thank you very much for reporting!

duetg · December 16, 2023, 8:04pm

After many retries, the chatmail server was finally deployed. There are some suggestions to avoid people encountering the same problem in the future.

Install rsync on both target and operation virtualenv machines
Update DNS records before running script, not only A record for CHATMAIL_DOMAIN, for example chat.example.org, but also the CNAME for mta-sts.chat.example.org (that would be chat.example.org)
Install sudo on target machine for Generate OpenDKIM domain keys step, which I do not know why, as it is already root login

Although the setup script is complete and I uploaded the DNS records, I still cannot pass the test and bench steps. More than 6 failures prevent the test from continuing. But the server seems to be working fine as I can register a new account and receive messages from my account on nine.
The echobot did not reply my message.