-termux-packages
+Termux packages
===============
-This project contains scripts and patches to cross compile and package packages for
-the [Termux](http://termux.com/) Android application.
+[](https://travis-ci.org/termux/termux-packages)
+[](https://gitter.im/termux/termux)
+This project contains scripts and patches to build packages for the [Termux](https://termux.com/) Android application. Note that packages are cross compiled and that on-device builds are not currently supported.
+
+Setting up a build environment using Docker
+===========================================
+For most people the best way to obtain an environment for building packages is by using Docker. This should work everywhere Docker is supported (replace `/` with `\` if using Windows) and ensures an up to date build environment that is tested by other package builders.
+
+Run the following script to setup a container (from an image created by [scripts/Dockerfile](scripts/Dockerfile)) suitable for building packages:
+
+ ./scripts/run-docker.sh
+
+This source folder is mounted as the `/root/termux-packages` data volume, so changes are kept
+in sync between the host and the container when trying things out before committing, and built
+deb files will be available on the host in the `debs/` directory just as when building on the host.
+
+The docker container used for building packages is a Ubuntu 17.10 installation with necessary packages
+pre-installed. The default user is a non-root user to avoid problems with package builds modifying the system
+by mistake, but `sudo` can be used to install additional Ubuntu packages to be used during development.
+
+Build commands can be given to be executed in the docker container directly:
+
+ ./scripts/run-docker.sh ./build-package.sh libandroid-support
+
+will launch the docker container, execute the `./build-package.sh libandroid-support`
+command inside it and afterwards return you to the host prompt, with the newly built
+deb in `debs/` to try out.
+
+Note that building packages can take up a lot of space (especially if `build-all.sh` is used to build all packages) and you may need to [increase the base device size](http://www.projectatomic.io/blog/2016/03/daemon_option_basedevicesize/) if running with a storage driver using a small base size of 10 GB.
+
+Build environment without Docker
+================================
+If you can't run Docker you can use a Ubuntu 17.10 installation (either by installing a virtual maching guest or on direct hardware) by using the below scripts:
+
+- Run `scripts/setup-ubuntu.sh` to install required packages and setup the `/data/` folder.
+
+- Run `scripts/setup-android-sdk.sh` to install the Android SDK and NDK at `$HOME/lib/android-{sdk,ndk}`.
+
+There is also a [Vagrantfile](scripts/Vagrantfile) available as a shortcut for setting up an Ubuntu installation with the above steps applied.
Building a package
==================
-In a non-rooted Android device an app such as Termux may not write to system locations,
-which is why every package is installed inside the private file area of the Termux app:
- PREFIX=/data/data/com.termux/files/usr
+The basic build operation is to run `./build-package.sh $PKG`, which:
+
+1. Sets up a patched stand-alone Android NDK toolchain if necessary.
-For simplicity while developing and building, the build scripts here assume that a /data
-folder is reserved for use on the host builder, which requires setup:
- sudo mkdir /data
- sudo chown $USER /data
+2. Reads `packages/$PKG/build.sh` to find out where to find the source code of the package and how to build it.
-The basic flow is then to run "./build-package.sh $PKG", which
-- Sets up a patched stand-alone Android NDK toolchain
+3. Extracts the source in `$HOME/.termux-build/$PKG/src`.
-* Reads packages/$PKG/build.sh to find out where to find the source code of the package and how to build it.
+4. Applies all patches in packages/$PKG/\*.patch.
-* Applies all patches in packages/$PKG/\*.patch
+5. Builds the package under `$HOME/.termux-build/$PKG/` (either in the build/ directory there or in the
+ src/ directory if the package is specified to build in the src dir) and installs it to `$PREFIX`.
-* Builds the package and installs it to $PREFIX
+6. Extracts modified files in `$PREFIX` into `$HOME/.termux-build/$PKG/massage` and massages the
+ files there for distribution (removes some files, splits it up in sub-packages, modifies elf files).
-* Packages the package in one or more .dpkg files for distribution
-Reading and following build-package.sh is the best way to understand what's going on here.
+7. Creates a deb package file for distribution in `debs/`.
+Reading [build-package.sh](build-package.sh) is the best way to understand what is going on.
Additional utilities
====================
-* build-all.sh: used for building all packages in the correct order (using buildorder.py)
+* build-all.sh: used for building all packages in the correct order (using buildorder.py).
-* check-pie.sh: Used for verifying that all binaries are using PIE, which is required for Android 5+
+* clean.sh: used for doing a clean rebuild of all packages.
-* detect-hardlinks.sh: Used for finding if any packages uses hardlinks, which does not work on Android M
+* scripts/check-pie.sh: Used for verifying that all binaries are using PIE, which is required for Android 5+.
-* check-versions.sh: used for checking for package updates
-
-* clean-rebuild-all.sh: used for doing a clean rebuild of all packages (takes a couple of hours)
+* scripts/check-versions.sh: used for checking for package updates.
-* list-packages.sh: used for listing all packages with a one-line summary
-
-
-Resources about cross-compiling packages
-========================================
-* [Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/index.html)
-
-* [Beyond Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/)
+* scripts/list-packages.sh: used for listing all packages with a one-line summary.
-* [Cross-Compiled Linux From Scratch](http://cross-lfs.org/view/svn/x86_64-64/)
-* [OpenWrt](https://openwrt.org/), an embedded Linx distribution, contains [patches and build scripts](https://dev.openwrt.org/browser/packages)
+Resources
+=========
+* [Android changes for NDK developers](https://android.googlesource.com/platform/bionic/+/master/android-changes-for-ndk-developers.md)
-* http://dan.drown.org/android contains [patches for cross-compiling to Android](http://dan.drown.org/android/src/) as well as [work notes](http://dan.drown.org/android/worknotes.html), including a modified dynamic linker to avoid messing with LD_LIBRARY_PATH.
+* [Linux From Scratch](http://www.linuxfromscratch.org/lfs/view/stable/)
-* [CCTools](http://cctools.info/index.php?title=Main_Page) is an Android native IDE containing [patches for several programs](https://code.google.com/p/cctools/source/browse/#svn%2Ftrunk%2Fcctools-repo%2Fpatches) and [a bug tracker](https://code.google.com/p/cctools/issues/list).
+* [Beyond Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/stable/)
-* [BotBrew](http://botbrew.com/) was a package manager for rooted devices with [sources on github](https://github.com/jyio/botbrew). Based on opkg and was transitioning to apt.
+* [OpenWrt](https://openwrt.org/) as an embedded Linx distribution contains [patches and build scripts](https://dev.openwrt.org/browser/packages)
-* [Kivy recipes](https://github.com/kivy/python-for-android/tree/master/recipes) contains recipes for building packages for Android.
+* [Kivy recipes](https://github.com/kivy/python-for-android/tree/master/pythonforandroid/recipes) contains recipes for building packages for Android.
Common porting problems
=======================
-* The Android bionic libc does not have iconv and gettext/libintl functionality built in. A package from the NDK, libandroid-support,
-contains these and may be used by all packages.
+* The Android bionic libc does not have iconv and gettext/libintl functionality built in. A `libandroid-support` package contains these and may be used by all packages.
* "error: z: no archive symbol table (run ranlib)" usually means that the build machines libz is used instead of the one for cross compilation, due to the builder library -L path being setup incorrectly
-* rindex(3) is defined in <strings.h> but does not exist in NDK, but strrchr(3) from <string.h> is preferred anyway
+* rindex(3) does not exist, but strrchr(3) is preferred anyway.
* <sys/termios.h> does not exist, but <termios.h> is the standard location.
* <sys/fcntl.h> does not exist, but <fcntl.h> is the standard location.
-* glob(3) system function (glob.h) - not in bionic, but use the libglob package
+* <sys/timeb.h> does not exist (removed in POSIX 2008), but ftime(3) can be replaced with gettimeofday(2).
-* undefined reference to 'rpl_malloc' and/or 'rpl_realloc': These functions are added by some autoconf setups
- when it fails to detect 0-safe malloc and realloc during cross-compilating. Avoided by defining
- "ac_cv_func_malloc_0_nonnull=yes ac_cv_func_realloc_0_nonnull=yes".
- See http://wiki.buici.com/xwiki/bin/view/Programing+C+and+C%2B%2B/Autoconf+and+RPL_MALLOC
+* <glob.h> does not exist, but is available through the `libandroid-glob` package.
-* cmake and cross compiling: http://www.cmake.org/Wiki/CMake_Cross_Compiling
- CMAKE_FIND_ROOT_PATH=$TERMUX_PREFIX to search there.
- CMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY and
- CMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY
- for only searching there and don't fall back to build machines
-
-* Android is removing sys/timeb.h because it was removed in POSIX 2008, but ftime(3) can be replaced with gettimeofday(2)
-
-* mempcpy(3) is a GNU extension. We have added it to <string.h> provided TERMUX_EXPOSE_MEMPCPY is defined,
- so use something like CFLAGS+=" -DTERMUX_EXPOSE_MEMPCPY=1" for packages expecting that function to exist.
+* SYSV shared memory is not supported by the kernel. A `libandroid-shmem` package, which emulates SYSV shared memory on top of the [ashmem](http://elinux.org/Android_Kernel_Features#ashmem) shared memory system, is available. Use it with `LDFLAGS+=" -landroid-shmem`.
+* SYSV semaphores is not supported by the kernel. Use unnamed POSIX semaphores instead (named semaphores are unimplemented).
dlopen() and RTLD_* flags
=================================
-<dlfn.h> declares
+<dlfcn.h> declares
- enum { RTLD_NOW=0, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=2, RTLD_NOLOAD=4}; // 32-bit
- enum { RTLD_NOW=2, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=0x00100, RTLD_NOLOAD=4}; // 64-bit
+ RTLD_NOW=0; RTLD_LAZY=1; RTLD_LOCAL=0; RTLD_GLOBAL=2; RTLD_NOLOAD=4; // 32-bit
+ RTLD_NOW=2; RTLD_LAZY=1; RTLD_LOCAL=0; RTLD_GLOBAL=0x00100; RTLD_NOLOAD=4; // 64-bit
These differs from glibc ones in that
-1. They are not preprocessor #define:s so cannot be checked for with #ifdef RTLD_GLOBAL (dln.c in ruby does this)
-2. They differ in value from glibc ones, so cannot be hardcoded in files (DLFCN.py in python does this)
-3. They are missing some values (RTLD_BINDING_MASK, RTLD_NOLOAD, ...)
+1. They differ in value from glibc ones, so cannot be hardcoded in files (DLFCN.py in python does this)
+2. They are missing some values (`RTLD_BINDING_MASK`, ...)
+Android Dynamic Linker
+======================
+The Android dynamic linker is located at `/system/bin/linker` (32-bit) or `/system/bin/linker64` (64-bit). Here are source links to different versions of the linker:
-RPATH, LD_LIBRARY_PATH AND RUNPATH
-==================================
-On desktop linux the linker searches for shared libraries in:
-1. RPATH - a list of directories which is linked into the executable, supported on most UNIX systems. It is ignored if RUNPATH is present.
-2. LD_LIBRARY_PATH - an environment variable which holds a list of directories
-3. RUNPATH - same as RPATH, but searched after LD_LIBRARY_PATH, supported only on most recent UNIX systems, e.g. on most current Linux systems
-The Android linker (/system/bin/linker) does not support RPATH or RUNPATH, so we set LD_LIBRARY_PATH=$USR/lib and try to avoid building
-useless rpath entries with --disable-rpath configure flags.
-Another option to avoid depending on LD_LIBRARY_PATH would be supplying a custom linker - this is not done due to the overhead of maintaining a custom linker.
+- [Android 5.0 linker](https://android.googlesource.com/platform/bionic/+/lollipop-mr1-release/linker/linker.cpp)
+- [Android 6.0 linker](https://android.googlesource.com/platform/bionic/+/marshmallow-mr1-release/linker/linker.cpp)
+- [Android 7.0 linker](https://android.googlesource.com/platform/bionic/+/nougat-mr1-release/linker/linker.cpp)
+Some notes about the linker:
-Warnings about unused DT entries
-================================
-Starting from 5.1 the Android linker warns about VERNEED (0x6FFFFFFE) and VERNEEDNUM (0x6FFFFFFF) ELF dynamic sections:
-
- WARNING: linker: $BINARY: unused DT entry: type 0x6ffffffe arg ...
- WARNING: linker: $BINARY: unused DT entry: type 0x6fffffff arg ...
-These may come from version scripts in a Makefile such as:
-
- -Wl,--version-script=$(top_srcdir)/proc/libprocps.sym
-The termux-elf-cleaner utilty is run from build-package.sh and should normally take care of that problem.
-
-Obtaining shell access on an emulator
-=====================================
-First install and start sshd on device:
-
- apt install openssh
- sshd
-Then follow the below steps:
-
- # Find out the linux user for the package to use in the chown command later:
- adb shell dumpsys package com.termux | grep userId=
- # Push your public ssh key:
- adb push $HOME/.ssh/id_dsa.pub /data/data/com.termux/files/home/.ssh/authorized_keys
- # Use the linux user for the package, 10053 below, to set ownerhip and permissions:
- adb shell chown -R 10053 /data/data/com.termux/files/home/.ssh/
- adb shell chmod -R 0700 /data/data/com.termux/files/home/.ssh/
- # Forward port 8022 to the emulator:
- adb forward tcp:8022 tcp:8022
- # Finally connect with ssh:
- ssh -p 8022 localhost
-
-Bootstrapping on device
-=======================
-To get files on device one option is:
-
- udpsvd -vE 0.0.0.0 8069 tftpd -c . # Run on device. -c arg to allow file uploading
- printf "mode binary\nput out.md\nquit" | tftp 192.168.0.12 8069 # on computer
-Another is with ftp:
- tcpsvd -vE 0.0.0.0 8021 ftpd -w . # Run on device. -w arg to allow file uploading
- printf "put tmp.c\nquit" | ftp -n 192.168.0.12 8021 # Run on computer. -n arg to use anonymous login
-NOTE: The ftpd and tftpd programs has been patched to run without chroot. This means that the directory serving is only the starting point and clients may cd out of if the access the whole system!
+- The linker warns about unused [dynamic section entries](https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-42444.html) with a `WARNING: linker: $BINARY: unused DT entry: type ${VALUE_OF_d_tag}` message.
+
+- The supported types of dynamic section entries has increased over time.
+
+- The Termux build system uses [termux-elf-cleaner](https://github.com/termux/termux-elf-cleaner) to strip away unused ELF entries causing the above mentioned linker warnings.
+
+- Symbol versioning is supported only as of Android 6.0, so is stripped away.
+
+- `DT_RPATH`, the list of directories where the linker should look for shared libraries, is not supported, so is stripped away.
+
+- `DT_RUNPATH`, the same as above but looked at after `LD_LIBRARY_PATH`, is supported only from Android 7.0, so is stripped away.
+
+- Symbol visibility when opening shared libraries using `dlopen()` works differently. On a normal linker, when an executable linking against a shared library libA dlopen():s another shared library libB, the symbols of libA are exposed to libB without libB needing to link against libA explicitly. This does not work with the Android linker, which can break plug-in systems where the main executable dlopen():s a plug-in which doesn't explicitly link against some shared libraries already linked to by the executable. See [the relevant NDK issue](https://github.com/android-ndk/ndk/issues/201) for more information.
~mdw / termux-packages / blobdiff