Binaries-have-conflict

Here are some notes on how to deal with a certain problem in Debian packaging, which might be useful for people coming across the same issue. In certain cases you want to have two binary packages co-installable, which normally wouldn’t be because of the restriction that two packages can’t be installed at the same time which contain the same files: every single file in the filesystem have to belong to a single package1, and couldn’t be overwritten by another package.

This problem arises e.g. when packaging Python libraries which are shipped with executables like helper scripts or CLI tools. Usually, two packages for the two Python stacks resp. runtime systems are being build out of the source package, python-foo and python3-foo2. The included modules are going to be installed into different paths3, and there’s also other stuff in the packages which doesn’t conflict. No problem with that.

But, in both binary packages the same helper scripts usually are going to be installed into /usr/bin, which is a problem. Although the content of scripts might differ4, the two packages couldn’t installed at the same time because there is a binary conflict:

$ sudo debi citeproc-py_0.3.0-1_amd64.changes 
Selecting previously unselected package python-citeproc.
(Reading database ... 442052 files and directories currently installed.)
Preparing to unpack python-citeproc_0.3.0-1_all.deb ...
Unpacking python-citeproc (0.3.0-1) ...
Selecting previously unselected package python3-citeproc.
Preparing to unpack python3-citeproc_0.3.0-1_all.deb ...
Unpacking python3-citeproc (0.3.0-1) ...
dpkg: error processing archive python3-citeproc_0.3.0-1_all.deb (--install):
 trying to overwrite '/usr/bin/csl_unsorted', which is also in package python-citeproc 0.3.0-1
Setting up python-citeproc (0.3.0-1) ...
Processing triggers for man-db (2.7.2-1) ...
Errors were encountered while processing:
 python3-citeproc_0.3.0-1_all.deb
debi: debpkg -i failed

The all-around Lintian5 could be used to detect also this problem6 on the source package:

$ lintian citeproc-py_0.3.0-1_amd64.changes 
W: citeproc-py source: binaries-have-file-conflict python-citeproc python3-citeproc usr/bin/csl_unsorted

If two packages including the same file or files (meaning: filename and path) are supposed to be alternatively installable they need to have being declared as conflicting against each other7. But concerning Python libraries it’s definitely wanted that it is possible to have both runtime packages being installed together: people just want to run and develop things on both Python stacks on the same system. But how to handle the conflicting executables8?

There are comparatively easy ways to prevent rivaling binaries, but which all have some disadvantages. To begin with that, the scripts could be renamed (like by adding suffixes like -python2/-python3) to prevent filename duplication. But users which know the software from other setups might not find them, and that also probably does not correspond to what’s said in the documentation. And, this solution also needs duplicated manpages.

Another solution would be to build the scripts into /usr/share/<package>9 and to require the user to run the stuff from here or to add that to $PATH, but this isn’t really convenient for the user. However, if scripts are of minor importance, that approach saves the maintainer to write manpages (which are related to stuff in /usr/bin).

When it doesn’t matter on which Python it runs it’s also possible to contribute scripts only by one of the two packages. I’ve chosen this solution for Citeproc-py10, where I’ve removed csl_unsorted from the Python 2 package. This solutions means that one have to install also the whole, probably otherwise unused Python 3 stuff only for running the shipped helper script. But here it’s o.k. because this library isn’t that big for which it’s justified not to have set up a more complex solution for the binaries-have-conflict problem.

Most intricate to set up, but also most convenient for the user is to install an alternatives mechanism to handle conflicting executables for co-installable packages. I’ve did that for my package of Python-afl11. These two packages contain scripts (wrapper for afl), and Cython generated extensions in the public import paths12. The user has either one or both packages installed and could run the script /usr/bin/py-afl-fuzz in any case: that is very much like it’s wanted13.

Alternatives were created for the case that two different programs in two different binary packages, which aren’t handled by declaring “Conflicts” and therefore could be installed at the same time, offering the same functionality by different implementations14. Using the alternative system, the user can choose which implementation her prefers, and switch between them every time that’s wanted without installing or removing packages. I’ve got alternatives installed e.g. for Pybtex15, which is an alternative implementation of the bibliography preprocessor bibtex (package: texlive-binaries)16:

$ update-alternatives --display pybtex
pybtex - auto mode
  link best version is /usr/bin/bibtex.pybtex
  link currently points to /usr/bin/bibtex.pybtex
  link pybtex is /usr/bin/pybtex
  slave pybtex.1.gz is /usr/share/man/man1/pybtex.1.gz
/usr/bin/bibtex.pybtex - priority 200
slave pybtex.1.gz: /usr/share/man/man1/bibtex.pybtex.1.gz

$ sudo update-alternatives --config bibtex
There are 2 choices for the alternative bibtex (providing /usr/bin/bibtex).

  Selection    Path                      Priority   Status
------------------------------------------------------------
  0            /usr/bin/bibtex.original   100       auto mode
  1            /usr/bin/bibtex.original   100       manual mode
* 2            /usr/bin/bibtex.pybtex     10        manual mode

Press <enter> to keep the current choice[*], or type selection number: 

For Python-afl, alternatives have been set up like this: during build, the script(s) are put into /usr/share/<package> by dh_auto_install using custom settings for Pybuild. Thus, in the binary packages there are the files /usr/share/python-afl/py-afl-fuzz and /usr/share/python3-afl/py-afl-fuzz, which do not conflict.

The alternatives are put into action during the installation procedure of the packages by the postinst maintainer shell scripts17 in debian/, which run update-alternatives after the packages have been unpacked:

$ cat debian/python-afl.postinst
#!/bin/sh
set -e
if [ "$1" = configure ]
then
    update-alternatives \
    --install /usr/bin/py-afl-fuzz py-afl-fuzz /usr/share/python-afl/py-afl-fuzz 20
fi
#DEBHELPER#

The corresponding postinst script in the Python 3 package (both reduced here) installs another alternative for /usr/bin/py-afl-fuzz18:

$ cat debian/python3-afl.postinst 
#!/bin/sh
set -e
if [ "$1" = configure ]
then
    update-alternatives \
    --install /usr/bin/py-afl-fuzz py-afl-fuzz /usr/share/python3-afl/py-afl-fuzz 10
fi
#DEBHELPER#

The priorities (“20” and “10”) could have been chosen differently, this way the alternatives just remain unaltered if the Python 2 package has been installed first19. The alternatives are getting removed during the uninstall procedure by the corresponding prerm scripts, which trigger update-alternatives --remove20. Alternatives are handled by symbolic links, /usr/bin/py-afl-fuzz links to /etc/alternatives/py-afl-fuzz, and that again to /usr/share/python-afl/py-afl-fuzz, or the other one. Like said, with a setup like this the user always has py-afl-fuzz in $PATH, either if one of the two packages is installed, or both (to switch between the alternatives with --config isn’t really needed here because it’s the same script). A hint: the package installation testing tool Piuparts21 could be used to check over if maintainer scripts are running as they are supposed to22.

Manpages like /usr/share/man/man1/py-afl-fuzz.1.gz are also conflicting, but this problem could also be solved using alternatives: for Python-afl, two manpages having different file names are generated with txt2man for the two binary packages, py-afl-fuzz.py2.1 and py-afl-full.py3.1. And for update-alternatives, slaves could be used with --install23.

update-alternatives \
--install /usr/bin/py-afl-fuzz py-afl-fuzz /usr/share/python-afl/py-afl-fuzz 20 \
--slave /usr/share/man/man1/py-afl-fuzz.1.gz py-afl-fuzz.1.gz /usr/share/man/man1/py-afl-fuzz.py2.1.gz

update-alternatives \
--install /usr/bin/py-afl-fuzz py-afl-fuzz /usr/share/python3-afl/py-afl-fuzz 10 \
--slave /usr/share/man/man1/py-afl-fuzz.1.gz py-afl-fuzz.1.gz /usr/share/man/man1/py-afl-fuzz.py3.1.gz

Though, not so elegant is the fact that if both packages are installed at the same time, there are even three identical manpages on the system, which could be confusing for the user:

$ man py-afl-fuzz<TAB>
py-afl-fuzz   py-afl-fuzz.py2   py-afl-fuzz.py3

This isn’t really optimal, and I’m seeking out for a solution on this point of solving binaries-have-conflict with alternatives. I’m thinking about putting the manpages also into /usr/share, but that’s not so refined, either. Thus, this solution has its downside with manpages.


  1. Which could be detected with $ dpkg -S /path/to/file [return]
  2. https://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html#s-package_names [return]
  3. Like /usr/lib/python2.7/dist-packages/foo/ and /usr/lib/python3/dist-packages/foo/ [return]
  4. The Python helper tools for Debhelper (https://packages.debian.org/unstable/dh-python) probably have adjusted the Shebang lines, or Setuptools by console_script might have generated different scripts according to the runtime system. For that reason it’s not possible to put the scripts into a -common package. [return]
  5. https://lintian.debian.org/ [return]
  6. https://lintian.debian.org/tags/binaries-have-file-conflict.html. This affects also the corresponding manpages. [return]
  7. https://www.debian.org/doc/debian-policy/ch-relationships.html#s-conflicts [return]
  8. By the way, the same problem arises with doc-base files. [return]
  9. Using Pybuild this could be easily achieved with export PYBUILD_INSTALL_ARGS=--install-scripts=/usr/share/{package} [return]
  10. https://packages.qa.debian.org/c/citeproc-py.html [return]
  11. https://packages.qa.debian.org/p/python-afl.html. Thanks to Jakub Wilk for suggesting that. [return]
  12. Although having modules resp. object files in the public import path like a library, this package is more an application (which the user is going to run) than a library (which the user develops upon or needs to run an app). [return]
  13. These wrapper scripts are in fact identical by content and not Python runtime dependent, but this is irrelevant for alternatives. They could have been put into a separate -common package, though. [return]
  14. https://www.debian.org/doc/debian-policy/ch-files.html#s-binaries [return]
  15. https://packages.qa.debian.org/p/pybtex.html [return]
  16. https://lists.debian.org/debian-user/2002/08/msg02808.html [return]
  17. https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html [return]
  18. Some details are important for maintainer scripts: set -e for improved error handling is absolutely necessary, while it’s better to set this in the body instead in the shebang line (https://lintian.debian.org/tags/maintainer-script-without-set-e.html). To complete the shell script with exit 0 could be spared, but if it is written it’s important that the debhelper token (#DEBHELPER#) appears before that (it needs to expand to executable code, see https://bugs.debian.org/798216). The scripts are getting copied into the binary package as DEBIAN/postinst etc. [return]
  19. See update-alternatives(1) for detailled info. [return]
  20. https://sources.debian.net/src/python-afl/0.5.1-2/debian/python-afl.prerm [return]
  21. https://packages.qa.debian.org/p/piuparts.html [return]
  22. And, if you set up alternative for packages which are already in the archive, be sure to set up cross breaks against earlier versions which did not had alternatives to prevent that any of these packages are tried to installed together with the new ones (one having the script in /usr/bin, see https://bugs.debian.org/798354). [return]
  23. https://sources.debian.net/src/python-afl/0.5.1-2/debian/python-afl.postinst [return]