Snakes++: Anaconda goes Python 3

Anaconda is the OS installer used by Fedora and RHEL GNU/Linux
distributions and all their derivatives. It’s written in the Python programming
language and many people say it’s one of the biggest and most complex pieces of
software written in this dynamic programming language. At the same time it is
one oldest big Python projects. [1]

[1] the first commit in Anaconda’s git repository is from Apr 24 1999, but
that’s the beginning of the GUI code being developed so the core actually
predates even these "IT-old times"

Fedora, Anaconda, Python 3

Over the time, the Python language has been evolving and so has been Anaconda’s
codebase getting not only new features and bug fixes, but also code improvements
using new language features. [2] Such evolution have been happening in small
steps over the time, but in recent years the community around the Python
language have been slowly migrating to a new backwards-incompatible version of
the language — Python 3. Python 3 is the version of Python that will get
future improvements and generally the vast majority of focus and work. There
will only be bugfixes for Python 2 in the future. [3] Users of Fedora may have
noticed that there was a proposal for a major Python 3 as Default change that
suggested migrating core components (more or less everything that’s available on
a live media) to Python 3 for Fedora 22. Since some developers and partially
also QA ignored it (intentionally or not), the deadline was missed and the
change was postponed to Fedora 23. And together with this Anaconda’s deadline
for the "Python 3 switch" (see below) was postponed to the time when Fedora 22
gets released as we identified three key facts during the discussions about the
original feature proposal (for F22):

  1. no matter how much the target is clear, people ignore it if things are not broken (at which point they start complaining :-P)
  2. it’s hard and inefficient to maintain both Python 2 and Python 3 versions of such big project as anaconda is
  3. QA has not enough capacity to test both versions and thus switching between them during the release would make things broken moving the whole process at least few weeks back
[2] an interesting fact is that the beginning of history of the Anaconda’s
sources predate the existence of True and False (boolean) values
in Python
[3] https://www.python.org/dev/peps/pep-0373/#maintenance-releases
https://www.python.org/dev/peps/pep-0404/

"Python 3 only" vs. "Python 2/3 compa­tible"

Python 3 is a major step and making some code compatible with both Python 2 and
Python 3 usually requires adding if s checking which version of Python the
code is run in and doing one or another thing based on such check. There is a
very useful module called six [5] that provides many functions, lists and
checks that hide the if s, but even when using this module, the code gets
more complicated, worse readable and harder to debug (and thus maintain) by
making it Python 2/3 compatible. While for libraries (or more precisely python
packages), it is worth it as it makes them usable for a wider variety of user
code, for applications written in Python 2, it is easier and in many ways better
to just switch to Python 3.

[5] http://pythonhosted.org/six/

For the reasons described above the Red Hat’s Installer team as the group of
Anaconda’s developers and maintainers decided to make all their libraries Python
2/3 compatible and to move Anaconda to Python 3. The only exception is the
pyblock (CPython) library that was developed in 2005 to provide quite a wide
range of functionality (not only) for the Anaconda installer, but which has been
over time getting more and more replaced by other libraries, utilities and other
means and become only used by the installer. Thus instead of porting the whole
library to Python 3 we decided to drop it and implement the few required
functions in the (new) libblockdev library [6] that was being born at that
time.

[6] using GLib and GObject introspection as shown in some of my other
posts and thus being both Python 2 and Python 3 compatible

Yum vs. DNF

Not everything used by the Anaconda installer is, of course, developed and
maintained by the Installer team. There were few "external" python libraries
that were required to be made Python 2/3 compatible and then there was Yum,
used by Anaconda for one of the key things — package installation. Yum is
usually used as the yum utility, but it is also a Python package which is
the way Anaconda has been making use of it. However, Yum has been being slowly
replaced by a new project called DNF that started as a fork of Yum and that
has been replacing Yum code with either new code or calls to newly born (C)
libraries. It has been decided that Yum will never get the Python 3 support as
it will stay in the maintainance mode with new features and development being
focused on DNF. A result for Anaconda was that with the switch to Python 3 it
will also have to say "good bye" to Yum and use DNF instead. Fortunately, the
author and first maintainer of DNF — AleŇ° Kozumpl√≠k — gave his former team
great help and wrote the vast majority of the Anaconda’s DNFPayload
class. Still, the switch from Yum to DNF [7] in the installer was expected to
be a problematic thing and was one of the "official reasons" why the switch to
Python 3 was postponed to Fedora 23. [8]

[7] by default (previously only activated by the inst.dnf=1 boot option
[8] although so far the DNFPayload seems to be working great with only
few smaller (non-key) features being missing and added over time and
being the default for Fedora 22 (with inst.nodnf turning it off)

We need you

As can be seen from the above there are lots of things behing something that
might look like "just a switch to a newer version of Python". And where there is
lot of stuff behing something there’s also a lot of stuff that can go wrong. Be
it the Anaconda’s code (no, the 2to3 utility really doesn’t do it all
magically) or any of the libraries it uses, there is quite a lot to test and
check. That’s why we decided to give us a head start and did the switch to
Python 3 in a separate branch of the project using Copr to do unofficial
builds and composes. [9] At first, we had only been creating Rawhide composes,
but that turned out to be not enough as we spent the same time hitting and
debugging Python 3 related issues as with unrelated Rawhide issues. That’s why
we decided to spent extra time on it, ported the f22-branch code to Python 3
and started creating F22 Python 3 composes that are stable and do not suffer
with issues caused by unstable Rawhide packages.

The images are at https://m4rtink.fedorapeople.org/anaconda/images/python3/f22/
and we would like to encourage everybody having some free "testing time" to
download and test them by doing various more or less weird Fedora 22
installations. Please report issues at Anaconda’s GitHub project page and if
you have a patch, please a submit pull requests against our development
branch
.

Last but not least, big THANKS for any help!

[9] https://copr.fedoraproject.org/coprs/bkabrda/py3anaconda/