site_treelogo
  -   Terms of Use and Privacy
Apps | GNU/Linux | Meta
rss
site_treelogo
  -   Terms of Use and Privacy
Apps | GNU/Linux | Meta
rss

<<   <   >   >>

2020-07-11 | Apps | Overlay Partition

Thumbsup will create a workable photo site with modern mobile views. Knoppix 8.6.1 has a 64 bit version of Docker that it runs outside of the 32 bit install of Debian Buster. Since Knoppix will run on just about any x86 system from the last couple of decades, it is a decent choice to use as an OS to create something like a photo site, where you want to make sure you don't have to go through too many shenanigans to get the software to run, particularly since it runs Docker. There is a problem with Docker in that it doesn't deal well with aufs, at least with the version that is used on Knoppix 8.6.1. Further, if you want to share files outside of the chroot through a bind, you can't use Reiserfs. Because of all this, I modified the knoppix-newroot command to bind a subdirectory off of backups in my home directory:

for k in /KNOPPIX /KNOPPIX[1-7]*; do
 if mountpoint -q "$k"; then
  mkdir -p "/NEWROOT$k"
  mount --bind "$k" "/NEWROOT$k" && MOUNTS="/NEWROOT$k $MOUNTS"
 fi
done
mkdir -p /NEWROOT/bk
mount --bind "/home/knoppix/backups/bk" "/NEWROOT/bk"&& MOUNTS="/NEWROOT/bk $MOUNTS"
# Link first level
ln -snf /KNOPPIX/{sbin,bin,lib} /NEWROOT

To run Docker, start up the chroot environment and exit the default loaded container:

$ /usr/local/bin/knoppix-newroot
Setting up chroot-environment with symlinks, patience...
Chroot environment set up in /NEWROOT
Do 'sudo chroot /NEWROOT'
$ sudo chroot /NEWROOT
# knocker start
Knocker - Knoppix meets Docker V1.0
Checking architecture... 64bit Kernel, OK.
.
.
.
bash-5.0# exit
.
.
.

I imagine there is a way to modify this so I don't exit each time. I believe this is just an example of running knoppix within docker. I have simpler plans.

Pull down the docker image:

# docker pull thumbsupgallery/thumbsup
Using default tag: latest
latest: Pulling from thumbsupgallery/thumbsup
e7c96db7181b: Pull complete 
bbec46749066: Pull complete 
.
.
.
0d0527056aaf: Pull complete 
8eee12e81b69: Pull complete 
ea8ae027aa51: Pull complete 
Digest: sha256:9d6b104e4d4d531daf0e749ac31d47eed914b429754317b2c9136482db8784ef
Status: Downloaded newer image for thumbsupgallery/thumbsup:latest

The filename and other exif data shows up in the album on information. Depending on the exif data, it might not look good, and you may want to remove for privacy. If you prefer to strip all exif information, remember that this also has the rotation information on the image, so capture that first:

mogrify -auto-orient pic*.jpg

Then strip all exif info if you don't want it:

exiftool -all= -overwrite_original pic*.jpg

Run the docker program that creates the album. Notice I'm using the bk mount point:

# docker run -t --mount type=bind,source=/bk,target=/bk  thumbsupgallery/thumbsup thumbsup --input /bk/met/media/ --output /bk/met/site --sort-albums-by title --sort-albums-direction asc --sort-media-by filename --sort-media-direction desc

When you are done and happy with the program, create a backup of the docker image:

# docker save thumbsupgallery/thumbsup > /bk/thumbsup.tar

To restore in the future, just use docker load < /bk/thumbsup.tar. Here is how the bk partition is mounted within the chroot:

# mount
.
.
.
/dev/sdb1 on /bk type ext4 (rw,relatime)
.
.
.

Outside the chroot:

$ mount | grep bk
/dev/sdb1 on /NEWROOT/bk type ext4 (rw,relatime)
$ grep sdb1 /etc/fstab
/dev/sdb1 /home/knoppix/backups ext4 noauto,users,exec 0 0

What I find most fascinating about all of this, is from a conceptual perspective this is extremely simple, yet the details rapidly become complicated. The operating system, including Docker, can be installed on almost any system with five or so clicks and entries. Just boot up Knoppix and click the "install knoppix to disk" option. After you start your new system, just load up the Docker image and run it with some options that include source and destination locations for the images and website storage, respectively. This is complicated because the "ease of install and use" Knoppix conflicts somewhat with the "ease of install and use" Docker. Another complication is how you share the actual files. Now, while it is true that we have a pretty great snapshot of how to do this, we are fairly stuck with this procedure, as it is unique to the constraints of "ease of install and use". Say we wanted to install this on Ubuntu 20.04. Much of this would be different, and we would need to know quite a bit to change this. True, Docker would likely run in a more standard way, but we would still have to install the OS and remap our file shares. We could also use the npm method of install npm install -g thumbsup, but, well, the npm package management platform is owned by a large corporate entity. Besides the corporate ownership parts, package management tends to rot over time. With the method described here, we have a tarball of the Docker image, an OS that can be installed on anything, and a few commands and filesystem tweaks to run. Just a side note here. The Thumbsup Docker image is using Alpine Linux. It turns out that the latest iteration of Alpine didn't provide the correct features for Thumbsup, so the author recompiled ImageMagick by pulling a source tree from github and recompiling with HEIC support using this Dockerfile:

FROM node:10-alpine as base
RUN apk add --no-cache libgomp zlib libpng libjpeg-turbo libwebp tiff lcms2 x265

# --------------------------------------------- # Build ImageMagick with HEIC support # Based on https://github.com/jarnoh/imagemagick-docker # Shouldn't be needed anymore when the APK package includes HEIC support (https://bugs.alpinelinux.org/issues/9555) # --------------------------------------------- # Copied from https://raw.githubusercontent.com/thumbsup/docker/master/runtime-alpine/Dockerfile FROM base as magick

RUN apk add --no-cache alpine-sdk automake autoconf libtool bash RUN apk add --no-cache zlib-dev libpng-dev libjpeg-turbo-dev libwebp-dev tiff-dev lcms2-dev x265-dev

WORKDIR /work RUN git clone -b frame-parallel https://github.com/strukturag/libde265.git WORKDIR libde265 RUN ./autogen.sh && ./configure && make -j8 install

WORKDIR /work RUN git clone https://github.com/strukturag/libheif.git WORKDIR libheif RUN ./autogen.sh && ./configure && make -j8 install

WORKDIR /work RUN git clone https://github.com/ImageMagick/ImageMagick.git WORKDIR ImageMagick RUN ./configure --with-heic --with-jpeg --with-lcms2 --with-png --with-gslib --with-tiff --with-zlib --with-threads --with-webp --without-x --disable-cipher --without-magick-plus-plus --without-pango --without-perl RUN make -j8 install

# --------------------------------------------- # Final Node image with all dependencies # ---------------------------------------------

FROM base RUN apk add --update --no-cache ffmpeg graphicsmagick exiftool gifsicle

# Copy ImageMagick libs and binary COPY --from=magick /usr/local/lib/libde265.so.1 /usr/local/lib/libde265.so.1 COPY --from=magick /usr/local/lib/libheif.so.1 /usr/local/lib/libheif.so.1 COPY --from=magick /usr/local/lib/libMagickCore-7.Q16HDRI.so.6 /usr/local/lib/libMagickCore-7.Q16HDRI.so.6 COPY --from=magick /usr/local/lib/libMagickWand-7.Q16HDRI.so.6 /usr/local/lib/libMagickWand-7.Q16HDRI.so.6 COPY --from=magick /usr/local/lib/ImageMagick-7.0.8/ /usr/local/lib/ImageMagick-7.0.8/ COPY --from=magick /usr/local/etc/ImageMagick-7/ /usr/local/etc/ImageMagick-7/ COPY --from=magick /usr/local/bin/magick /usr/local/bin/magick RUN ln -s /usr/local/bin/magick /usr/local/bin/convert

This further complicates re-use from a certain perspective. Let me put this another way. Resilience from an English language perspective is "The capacity to recover quickly from difficulties; toughness". If the subject is a human faced with difficulties, a human will use intelligence to adapt, whether the problem is psychological, social, or simply a need for food. From a compute perspective, though, resilience means the ability to recover quickly from difficulties, but in this case the difficulties are a broken system. What kinds of things break an IT system? For a distributed system, any of the fallacies of distributed computing might break it. Perhaps the network is down. Perhaps the topology changed. There are also other usual suspects like hard disk failure or an OS upgrade that is incompatible with the application.

This last point is exactly what Docker is for. Alpine, for instance, is incompatible with Thumbsup. The author fixes this with a custom compile of ImageMagick. But, notice how specific this has become within the system? If we look at this from the simple perspective, we are just running a container on Knoppix, and could conceivably run it anywhere. When we dig in, though, we realize that we are really running a customized version of ImageMagic that is inserted into an Alpine OS that is running in Docker that is running on Knoppix, that has some specific, unique hacks to get the file system to bind so that all works together. Notice how in the compile command there are other libraries linked, like zlib? These kind of libraries can change over time, and zlib is particularly famous for this. So, sure, we can ensure that we can run this Thumbsup app always, but if we have to modify it in five years, it could be quite difficult. This means that we are not resilient from that perspective. We cannot easily change the system. We have a Docker resiliency paradox.

How do we get out of the paradox? The only way I know how to do this is to map systems so they can easily be translated at the time of crisis to new systems. This article, for instance, provides a list of the key technical considerations so that the software could be ported. It is a mistake, though, to assume that pushing the OS libraries and frameworks (ImageMagick is a graphics framework) into the container is necessarily a means to resilience. It will solve some problems, but it creates others. Further, a modern system like what is documented here relies on Github, the Alpine Linux distribution, the npm package manager/service, and many other distributed efforts that all have equally obscure abstractions and gotchas. This doesn't mean it is wise to build your own mobile-happy photo app, but it does mean that there are concerns of resilience that can only be solved with some sweat and documentation.

image docker knocker

Articles tagged with image on SysAdminTools:

2020-07-11: Overlay Partition
2019-07-30: XFig Diagramming Tool
2019-07-20: Removing Image Metadata

Articles tagged with docker on SysAdminTools:

2020-07-11: Overlay Partition

Articles tagged with knocker on SysAdminTools:

2020-07-11: Overlay Partition