add "Basic Ubuntu packaging"

7 files changed, 378 insertions, 0 deletions

diff --git a/_posts/2021-03-10-ubuntu-packaging.md b/_posts/2021-03-10-ubuntu-packaging.md
new file mode 100644
index 0000000..431651e
--- /dev/null
+++ b/_posts/2021-03-10-ubuntu-packaging.md
@@ -0,0 +1,320 @@
+---
+title: Basic Ubuntu packaging
+excerpt: >
+  Packaging your software to be published in a PPA for the first time is a
+  pain.
+custom_css:
+  - snippets.css
+snippets_root_directory: snippets/ubuntu_packaging
+snippets_language: plain
+snippets:
+  basic:
+    - basic/changelog
+    - basic/control
+    - basic/copyright
+    - basic/rules
+  install:
+    - install/test.install
+  gbp:
+    - gbp/gbp.conf
+---
+It took me about an hour to make a PKGBUILD for my simple, non-compiled
+piece of software to be published on [AUR].
+In contrast, it took me a few days to figure out how to build suitable .deb
+packages for publishing in a PPA on [Launchpad].
+In this post, I'll try to describe some of the initial pain points of mine.
+
+[AUR]: https://aur.archlinux.org/
+[Launchpad]: https://launchpad.net/
+
+Basics
+------
+
+The Debian package format is really old, and it shows.
+There's a billion of metadata files to take care of, and barely any suitable
+tutorials for beginners.
+At best, you'll learn how to build _binary_ packages, not suitable for
+publishing in a PPA (which only accept _source_ packages and builds the
+binaries itself).
+
+First, you need to realise that there're source packages and binary packages.
+Binary packages are the .deb files that actually contain the software.
+A source package is, confusingly, multiple files, and you need to submit them
+all to Launchpad.
+You can distribute binary packages directly to your users, but they would have
+to fetch & install the new version manually every time there's an upgrade.
+If you could set up a repository and just point the users to it, they would get
+the new versions naturally via the package manager (`apt`).
+
+Canonical's Launchpad provides a very handy PPA (Personal Package Archive)
+service so that anyone can set up a repository.
+Users could then use `add-apt-repository ppa:...` and get the packages in a
+standard and convinient way.
+
+Tools
+-----
+
+There's a myriad of tools to build and maintain Debian packages.
+The [Debian New Maintainers' Guide] provides a [short summary] of how these
+tools interact.
+This tutorial assumes that your software lives in a Git repository and you'd
+like to use Git to maintain the packaging metadata in the same repository.
+This process is greatly aided by the [git-buildpackage] tool.
+We still need to install a bunch of other stuff though; the complete command
+line to install the required tools would be something like
+
+    sudo apt install -y build-essential devscripts dh-make git-buildpackage
+
+Many of the tools pick up particular metadata (like the maintainer name and
+email address) from environment variables.
+You can put something like
+
+    export DEBFULLNAME='John Doe'
+    epxort DEBEMAIL='John.Doe@example.com'
+
+in your .bashrc to set them globally.
+
+[Debian New Maintainers' Guide]: https://www.debian.org/doc/manuals/maint-guide
+[short summary]: https://www.debian.org/doc/manuals/maint-guide/build.en.html#hierarchy
+[git-buildpackage]: http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html
+
+Getting started
+---------------
+
+Let's create a repository to try things out.
+It'll contain a single executable shell script test.sh, which only outputs the
+string "test".
+
+    mkdir test
+    cd test
+    git init
+    cat <<'EOF' > test.sh
+    #!/usr/bin/env bash
+    echo test
+    EOF
+    chmod +x test.sh
+    git add .
+    git commit -m 'initial commit'
+
+This is going to be version 1.0 of our project, let's tag it as such.
+
+    git tag -a -m 'Release 1.0' v1.0
+
+All of the Debian packaging tools are tailored to the following use-case.
+
+1. There's an upstream distribution, which releases the software in tarballs.
+2. There's a maintainer (who's not the software author), who takes care of
+packaging and is disconnected from the development.
+
+This disconnect means that maintaining the Debian packaging files in the
+`master` branch is inconvinient using the existing tools.
+At the very least, you should create a separate branch for doing packaging
+work.
+
+In addition, Debian (and hence, Ubuntu) is not a rolling-release distribution.
+That means that there're regular releases, and the software version shouldn't
+change too much during a lifetime of a single release.
+Once Debian makes a release, the software version is more or less fixed, and
+security fixes from future versions should be backported separately for each of
+the supported Debian/Ubuntu releases.
+
+Except there _is_ a rolling-release distribution of Debian, and it's called
+"unstable" or "sid".
+The bleeding-edge packaging work should target the "unstable" distribution.
+
+So, let's create a new branch `debian` for our packaging work:
+
+    git checkout -b debian
+
+All the packaging tools assume there's a separate folder "debian" that contains
+the package metadata files.
+There's a handy tool `dh_make` that creates the directory and populates it with
+a number of template metadata files.
+Using it is not so simple though.
+First of all, it assumes that there's a properly named tarball with the project
+sources available in the parent directory.
+Why?
+Who knows.
+Let's create said tarball:
+
+    git archive --format=tar --prefix=test_1.0/ v1.0 | gzip -c > ../test_1.0.orig.tar.gz
+
+The tarball name should follow the NAME_VERSION.orig.tar.gz pattern exactly!
+Anyway, now is the time to run `dh_make`:
+
+    dh_make --indep --copyright mit --packagename test_1.0 --yes
+
+I'm using the MIT License for our little script, hence the `--copyright mit`
+argument.
+In addition, every package in Debian is either "single", "arch-independent",
+"library" or "python".
+I'm not sure what the exact differences between those are, but a shell script
+is clearly CPU architecture-independent, hence the `--indep` argument.
+If it was a compiled executable, it would be a "single" (`--single`) package.
+
+`dh_make` created the "debian" directory for us, filled with all kinds of
+files.
+The only required ones are "changelog", "control", "source", "rules" and the
+"source" directory.
+Let's remove every other file for now:
+
+    rm -f -- debian/*.ex debian/*.EX debian/README.* debian/*.docs
+
+You can study the exact format of the metadata files in the [Debian New
+Maintainers' Guide], but for now let's make keep it simple:
+
+{% include snippets/section.html section_id='basic' %}
+
+The "control" and "copyright" files are fairly straighforward.
+The "changelog" file has a strict format and is supposed to be maintained using
+the `dch` tool (luckily, git-buildpackage helps with that; more on that later).
+
+The "rules" file is an _executable_ Makefile, and actually controls how the
+software is built.
+Building a package involves invoking many predefined targets in this Makefile;
+for now, we'll resort to delegating everything to the `dh` tool.
+It's the Debhelper tool; it's a magic set of scripts that contain an
+unbelievable amount of hidden logic that's supposed to aid package maintainers
+in building the software.
+For example, if the package is supposed to be built using the standard
+`./configure && make && make install` sequence, it'll do this automatically.
+If it's a Python package with setup.py, it'll use the Python package-building
+utilities, etc.
+We don't want any of that, we just want to copy test.sh to /usr/bin.
+It can be taked care of using the `dh_install` script.
+While building the package, it'll get executed by `dh`, read the
+"debian/test.install" file and copy the files listed there to the specified
+directories.
+Our test.install should look like this:
+
+{% include snippets/section.html section_id='install' %}
+
+At this point, we can actually build a proper Debian package!
+
+    dpkg-buildpackage -uc -us
+
+This command will generate a bunch of files in the parent directory.
+The one of interest to us is "test_1.0-1_all.deb".
+We can install it using `dpkg`:
+
+    sudo dpkg -i ../test_1.0-1_all.deb
+
+We can now execute `test.sh`, and it'll hopefully print the string "test".
+
+This .deb file can be distributed to other users, but is no good for uploading
+to Launchpad.
+For one, it's a binary package, and we need source packages for Launchpad to
+build itself.
+Second, it's unsigned, which is also a no-no.
+
+I'm actually not going to describe how to set up a GnuPG key and upload it to
+the Ubuntu keyserver (keyserver.ubuntu.com), but it's pretty straightforward
+once you know the basics of GnuPG key handling.
+
+One disadvantage of the `dpkg-buildpackage` tool is that it creates a lot of
+files in the "debian" directory; their purpose is unclear to me.
+For now, you can delete them, leaving only the original "changelog", "control",
+"copyright", "rules", "test.install" and the "source" directory.
+
+git-buildpackage
+----------------
+
+git-buildpackage is a wonderful tool that helps with managing the packaging
+work in the upstream repository.
+Please refer to its manual to learn how to use it properly.
+We need to configure it so that it knows how the release tags look like
+(`vVERSION`), how the packaging branch is called (`debian`) and where to put
+the generated files.
+Create "debian/gbp.conf" with the following contents:
+
+{% include snippets/section.html section_id='gbp' %}
+
+One unclear line here is `pristine-tar = False`.
+It turns out, a lot of Debian package maintainers use the `pristine-tar` tool
+to create "pristine", byte-for-byte reproducible tarballs of the upstream
+software.
+This is just more headache for us, so we're not going to use that;
+git-buildpackage will just use the normal `git archive` to create tarballs.
+
+First, commit the packaging work we just made:
+
+    git add debian/
+    git commit -m 'initial Debian release'
+
+We can now build the package using git-buildpackage:
+
+    gbp buildpackage
+
+The tool will try to sign the packages, so this assumes that you have your
+GnuPG key set up!
+
+If all went right, it just built the packages in the ../build-area directory.
+And it hasn't crapped all over the working directory too!
+Similar to `dpkg-buildpackage`, it builds binary packages by default.
+To build _source_ packages, it needs to be invoked with the `-S` argument:
+
+    gbp buildpackage -S
+
+It'll build the source package in the same directory (you'll notice a lot of
+files having the "_source" suffix).
+If all is well, we can tag the packaging work we've just completed:
+
+    gbp buildpackage --git-tag-only
+
+This will create the `debian/1.0-1` tag in the repository.
+
+We are now ready to upload the source package to Launchpad.
+It's done using the `dput` tool.
+The naive way would fail:
+
+    dput ppa:john-doe/test ../build-area/test_1.0-1_source.changes
+
+This is due to the fact that we've specified that we're targetting the
+"unstable" distribution in debian/changelog.
+There's no "unstable" distribution of Ubuntu though; we need to manually
+specify the minimal-supported version (e.g. "bionic"):
+
+    dput ppa:john-doe/test/ubuntu/bionic ../build-area/test_1.0-1_source.changes
+
+What about other distributions?
+Well, if the binary package doesn't need recompiling, we can use Launchpad's
+"Copy packages" feature; this is well-described in this [Ask Ubuntu question].
+
+[Ask Ubuntu question]: https://askubuntu.com/q/23227/844205
+
+New versions
+------------
+
+When a new version is released, git-buildpackage helps to integrate it to the
+packaging branch.
+Let's say the new version is tagged `v1.1`:
+
+    git checkout debian
+    git merge v1.1
+    gbp dch
+
+The above command will update debian/changelog; modify it manually to target
+the usual "unstable" distribution instead of "UNRELEASED" and update the
+version to something like "1.1-1".
+
+    git add debian/
+    git commit -m 'Debian release 1.1'
+    gbp buildpackage -S
+
+This will build the source package for the new version in the ../build-area
+directory; you can then upload it Launchpad and copy the built binary packages.
+
+Aftermath
+---------
+
+This fucking sucks.
+What's the way to sanely manage the repository if the build/runtime
+dependencies are different for different Ubuntu versions?
+I have no idea.
+Some pointers to help you understand what's going on in this tutorial more
+deeply:
+
+* [When upstream uses Git: Building Debian Packages with git-buildpackage](https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.import.upstream-git.html)
+* [Using Git for Debian packaging](https://www.eyrie.org/~eagle/notes/debian/git.html)
+
+Good luck with this, because I'm definitely overwhelmed.
diff --git a/_posts/snippets/ubuntu_packaging/basic/changelog b/_posts/snippets/ubuntu_packaging/basic/changelog
new file mode 100644
index 0000000..6b7131e
--- /dev/null
+++ b/_posts/snippets/ubuntu_packaging/basic/changelog
@@ -0,0 +1,5 @@
+test (1.0-1) unstable; urgency=medium
+
+  * Initial release.
+
+ -- John Doe <John.Doe@example.com>  Wed, 10 Mar 2021 16:15:19 +0000
diff --git a/_posts/snippets/ubuntu_packaging/basic/control b/_posts/snippets/ubuntu_packaging/basic/control
new file mode 100644
index 0000000..55f8252
--- /dev/null
+++ b/_posts/snippets/ubuntu_packaging/basic/control
@@ -0,0 +1,13 @@
+Source: test
+Section: utils
+Priority: optional
+Maintainer: John Doe <John.Doe@example.com>
+Build-Depends: debhelper-compat (= 12)
+Standards-Version: 4.4.1
+Homepage: https://example.com/test
+
+Package: test
+Architecture: all
+Depends: ${misc:Depends}
+Description: This is a test package.
+ This is a test package, just trying out Debian packaging.
diff --git a/_posts/snippets/ubuntu_packaging/basic/copyright b/_posts/snippets/ubuntu_packaging/basic/copyright
new file mode 100644
index 0000000..8a67e52
--- /dev/null
+++ b/_posts/snippets/ubuntu_packaging/basic/copyright
@@ -0,0 +1,31 @@
+Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Upstream-Name: test
+Upstream-Contact: John Doe <John.Doe@example.com>
+Source: https://example.com/test
+
+Files: *
+Copyright: 2021 John Doe <John.Doe@example.com>
+License: MIT
+
+Files: debian/*
+Copyright: 2021 John Doe <John.Doe@example.com>
+License: MIT
+
+License: MIT
+ Permission is hereby granted, free of charge, to any person obtaining a
+ copy of this software and associated documentation files (the "Software"),
+ to deal in the Software without restriction, including without limitation
+ the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ and/or sell copies of the Software, and to permit persons to whom the
+ Software is furnished to do so, subject to the following conditions:
+ .
+ The above copyright notice and this permission notice shall be included
+ in all copies or substantial portions of the Software.
+ .
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+ OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+ IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+ CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+ TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
diff --git a/_posts/snippets/ubuntu_packaging/basic/rules b/_posts/snippets/ubuntu_packaging/basic/rules
new file mode 100644
index 0000000..cbe925d
--- /dev/null
+++ b/_posts/snippets/ubuntu_packaging/basic/rules
@@ -0,0 +1,3 @@
+#!/usr/bin/make -f
+%:
+	dh $@
diff --git a/_posts/snippets/ubuntu_packaging/gbp/gbp.conf b/_posts/snippets/ubuntu_packaging/gbp/gbp.conf
new file mode 100644
index 0000000..d9f7e5f
--- /dev/null
+++ b/_posts/snippets/ubuntu_packaging/gbp/gbp.conf
@@ -0,0 +1,5 @@
+[DEFAULT]
+upstream-tag = v%(version)s
+debian-branch = debian
+pristine-tar = False
+export-dir = ../build-area/
diff --git a/_posts/snippets/ubuntu_packaging/install/test.install b/_posts/snippets/ubuntu_packaging/install/test.install
new file mode 100644
index 0000000..6222235
--- /dev/null
+++ b/_posts/snippets/ubuntu_packaging/install/test.install
@@ -0,0 +1 @@
+test.sh usr/bin