` as an anchor point, which will later be filled with content by JavaScript: ```html

``` ## 3. Fetch and process JSON data This is the main part. On the client side, the JSON data is fetched and for every language it contains, a child `

` is created inside the anchor point. The original JSON response from WakaTime includes an _Other_ item (which represents unrecognized terminal/directory activity) and items with very low usage. I get more than 30 entries, which is far too much to display. So I remove the _Other_ category entirely and filter out languages below 1% of usage, then normalize the remaining percentages to 100%. Here is the code: ```javascript document.addEventListener("DOMContentLoaded", () => { const element = document.getElementById("wakatime"); if (element) fetch("/languages.json") .then((response) => response.json()) .then(({ data }) => drawChart(element, fixOther(data))); }); /* Draw chart for the given data */ function drawChart(element, data) { data.forEach((item) => { const childElement = document.createElement("div"); childElement.className = "flex justify-center py-1"; childElement.style.backgroundColor = item.color; childElement.innerText = item.name; element.appendChild(childElement); }); // Let the CSS grid do the magic, see the blog post // of Marco Slooten for details: // https://marcoslooten.com/blog/using-css-grid-to-create-a-bar-chart/ element.style.gridTemplateColumns = data .map((item) => `${item.percent}fr`) .join(" "); } /* Remove "Other" and languages below 1%, normalize to 100% */ function fixOther(wakatimeData) { const filtered = wakatimeData.filter( (item) => item.name !== "Other" && item.percent >= 1, ); const total = filtered.reduce((sum, item) => sum + item.percent, 0); return filtered.map((item) => ({ ...item, percent: (item.percent / total) * 100, })); } ``` That's it. In my actual implementation, I added tooltips and language logos. But to keep this blog post short, I left that out. --- # Photovoltaik-Dashboard als Web-Applikation _Visualisierung von Messwerten einer Solaranlage_ Author: Georg Ledermann Published: 2021-02-03 Tags: Ruby on Rails, InfluxDB, Raspberry Pi, SENEC, SOLECTRUS Canonical: https://ledermann.dev/blog/2021/02/03/photovoltaik-dashboard-als-web-applikation/ Meine Solaranlage sammelt im Minutentakt Hunderte Messwerte, aber die Hersteller-App machte zu wenig daraus. Also habe ich mein eigenes Echtzeit-Dashboard dafür gebaut. --- Die Energiewende findet seit kurzem auch im eigenen Heim statt - auf dem Dach unseres Hauses befindet sich nun eine Photovoltaik-Anlage. Damit kommt der im Haus benötigte Strom größtenteils aus der Solarenergie und nicht mehr aus der Verbrennung von Kohle, was unabhängig von der Frage nach der finanziellen Rentabilität eine gute Sache ist - und außerdem für einen Web-Entwickler eine interessante Herausforderung darstellt, über die ich berichten möchte. Im Keller befindet sich jetzt ein _SENEC.Home V3 hybrid duo_, der in der Größe eines Kühlschranks neben dem Wechselrichter auch einen Batteriespeicher enthält, sodass überschüssige Energie bevorzugt nicht in die wenig lukrative Einspeisung, sondern in den eigenen Verbrauch der kommenden Nacht und des nächsten Tages fließen kann. Die SENEC-Box sammelt nebenbei auch eine große Anzahl an Messwerten und übermittelt diese in kurzen Zeitabständen über das Internet an den Hersteller, wo ich als Anlagenbetreiber Einblick in die eigene Energieerzeugung nehmen kann. SENEC stellt hierfür ein Web-Portal sowie eine Mobile-App bereit, sodass man ohne weiteres Zutun eine mehr oder weniger hübsche Visualisierung erhält. Diese Einblicke sind ganz nützlich, lassen sich aber auch besser machen. Hier möchte ich skizzieren, wie ich mit Ruby on Rails ein eigenes Photovoltaik-Dashboard erstellt habe. Vorab ein Blick auf das Endergebnis: Solectrus Screenshot Live: [https://demo.solectrus.de](https://demo.solectrus.de) ## Warum eine eigene Applikation? An den SENEC-Angeboten stören mich einige Dinge: - **Fünf-Minuten-Intervall:** Die Messwerte werden zwar kontinuierlich ermittelt, aber nur alle fünf Minuten an SENEC übertragen. Blickt man also in deren Mobile-App oder ins Web-Portal, sieht man für die Jetzt-Situation Werte, die bis zu fünf Minuten alt sein können. Das kann irritierend sein, weil sich z.B. längst eine große Wolke vor die Sonne geschoben haben könnte und die Energieausbeute im Moment eine ganz andere ist als dargestellt. - **Rentabilität:** SENEC lässt den finanziellen Aspekt komplett außen vor. Inwieweit sich die Investition in die Anlage bereits amortisiert hat, erfährt man dort nicht. Es gibt keine Gegenüberstellung von Investitionsaufwand und Ertrag. - **Keine Vorhersage:** Mittels solarspezifischer Wettervorhersage ließe sich prinzipiell die zu erwartende Strommenge prognostizieren, was beispielsweise interessant sein könnte, wenn es um das "Betanken" des Elektromobils geht: Wenn ich weiß, dass es morgen viel Sonnenenergie geben wird, lade ich das Auto eher morgen auf als heute. SENEC unterstützt dies aber nicht. - **Visuelle Darstellung:** Die Benutzeroberfläche des Web-Portals ist recht altbacken. Die App hingegen sieht zwar besser aus, ist aber auch nicht das Gelbe vom Ei. - **Stabilität**: Die von SENEC bereitgestellten Dienste scheinen des öfteren Lastprobleme zu haben. Die Antwortzeiten sind oft recht lang und manchmal fällt der Server auch ganz aus, was dann so aussieht: Fehleranzeige in der SENEC App ## Erste Schritte Da die SENEC-Box auch einen Web-Server enthält und über das lokale Ethernet erreichbar ist, besteht prinzipiell die Möglichkeit, selbst aktiv zu werden. Nach ersten Experimenten mit dem lokalen SENEC-Webserver wird schnell klar, dass sich wesentliche Messwerte leicht auslesen lassen: Kommunikation mit dem integrierten Web-Server des SENEC-Stromspeichers Damit beginnt die Sache aus Sicht eines Web-Entwicklers interessant zu werden. Ein paar Wochen später ist die Web-Applikation _SOLECTRUS_ entstanden, deren Komponenten ich allesamt im Quellcode veröffentliche. An dieser Stelle sei darauf hingewiesen, dass ich außer meiner Eigenschaft als Kunde in keinerlei Verbindung zum Unternehmen [SENEC](https://senec.com) stehe. ## Die Komponenten von SOLECTRUS ### Einzelne Messwertabfrage Um die Messwerte für einen einzelnen Zeitpunkt abzufragen, habe ich einen kleinen Ruby-Client geschrieben. Dieser kontaktiert den lokalen SENEC-Webserver über dessen IP-Adresse, fragt die wichtigsten Messwerte über einen POST-Request ab und decodiert die Antwort. Den Ruby-Client habe ich als Gem [senec](https://github.com/solectrus/senec) veröffentlicht. Die Ermittlung der aktuellen Messwerte erfolgt damit sehr einfach: ```ruby request = Senec::Request.new host: '10.0.1.18' request.inverter_power # => 6302 request.house_power # => 560 ``` ### Regelmäßige Datenermittlung und -übertragung Der SENEC-Webserver lässt sich nur über das lokale Netzwerk erreichen. Das soll auch so bleiben, eine Erreichbarkeit von außen kommt aus Sicherheitsgründen nicht in Frage. Die Messwerte müssen also regelmäßig abgefragt und in eine externe Datenbank übertragen werden. Um dies stromsparend und zuverlässig rund um die Uhr zu erledigen, verwende ich den Einplatinencomputer _Raspberry Pi_. Für diesen habe ich das kleine Ruby-Script [SENEC collector](https://github.com/solectrus/senec-collector) geschrieben, das regelmäßig (alle 5 Sekunden) die Messwerte abfragt und weitergibt. Bei der Suche nach einem Anbieter für Solar-Prognosen bin ich auf [forecast.solar](https://forecast.solar) gestoßen: Dort steht eine öffentliche API bereit, mit der man automatisiert unter Angabe von Geo-Koordinaten, Hausausrichtung, Dachneigung und Maximal-Leistung (kWp) die in den nächsten Tagen voraussichtlich produzierte Energiemenge der eigenen Photovoltaikanlage abrufen kann. Diese API wird vom gleichen Script ebenfalls regelmäßig (alle 15 Minuten) abgefragt und die erhaltenen Werte weitergegeben. Da es sich bei den entstehenden Daten um Zeitreihen handelt und voraussichtlich größere Mengen anfallen werden, bietet sich die Verwendung einer darauf spezialisierten Datenbank wie [InfluxDB](https://www.influxdata.com/) an. Damit lassen sich spätere Auswertungen leicht und effizient durchführen. Den _SENEC collector_ habe ich als Docker-Container verpackt, sodass er leicht auf einem _Raspberry Pi_ betrieben werden kann. Insgesamt entsteht damit eine Datensammlung in InfluxDB, die permanent mit aktuellen Werten befüllt wird. Um historische Daten zu ergänzen, die **vor** Inbetriebnahme des Collectors entstanden sind, gibt es auch eine Möglichkeit: SENEC selbst stellt auf ihrem Web-Portal die erhaltenen Daten (naja, nicht alle, aber die wichtigsten) im CSV-Format wochenweise zum Download bereit. Um diese ebenfalls nach InfluxDB zu übernehmen, habe ich noch das kleine CLI-Tool [SENEC importer](https://github.com/solectrus/importer) geschrieben. Lässt man dieses über die heruntergeladenen CSV-Dateien laufen, ist die Influx-Datenbank komplett und enthält sämtliche Werte seit Inbetriebnahme der Photovoltaik-Anlage (anfangs im 5-Minuten-Takt, später dann im 5-Sekunden-Takt). ### Visualisierung der Daten als Dashboard Die Rails-Applikation [SOLECTRUS](https://github.com/solectrus/solectrus/) entnimmt die Daten aus InfluxDB und stellt sie über eine ansprechende Oberfläche dar. Dies ist die umfangreichste Komponente und erledigt folgende Aufgaben: - Dargestellt werden die Messwerte sowie der Stromfluss zwischen Stromerzeugung, Stromverbrauch und Speicher - Es wird unterschieden zwischen aktueller Messung (_Jetzt_) und vergangenen Zeiträumen (_Tag_, _Woche_, _Monat_, _Jahr_ und _Gesamt_) - Berücksichtigt werden die Messwerte _Erzeugung_, _Hausverbrauch_, _Einspeisung_, _Netzbezug_, _Akku-Beladung_, _Akku-Entnahme_, _Ladezustand_ und _Wallbox_ - Der zeitliche Verlauf von Messwerten wird als Diagramm dargestellt - Bei der Darstellung der Stromerzeugung eines Tages wird im Diagramm zusätzlich die Prognose visualisiert - Errechnet wird der Autarkie-Grad sowie der Gewinn des gewählten Zeitraums. Letzterer ergibt sich aus der Einsparung aufgrund der Stromerzeugung zuzüglich der Einspeisevergütung. - Die Darstellung ist responsive und funktioniert sowohl im Desktop-Browser als auch auf Smartphone und Tablet Ich verwende folgenden Tech-Stack: - [Ruby on Rails](https://rubyonrails.org/) - [Hotwire](https://hotwire.dev/) - [ViewComponent](https://viewcomponent.org/) - [Tailwind CSS](https://tailwindcss.com/) - [Chart.js](https://www.chartjs.org/) - [InfluxDB](https://www.influxdata.com/products/influxdb/) ## Veröffentlichungen Alle Komponenten stehen auf GitHub im Quelltext bereit: [https://github.com/solectrus](https://github.com/solectrus) Das Dashboard kann mit echten Daten hier öffentlich eingesehen werden: [https://demo.solectrus.de](https://demo.solectrus.de) Gut erkennbar ist zur Zeit, dass Photovoltaik im deutschen Winter ein limitiertes Vergnügen ist - im Rest des Jahres dürfte das aber ganz anders aussehen. **Update im Januar 2023:**\ Das Projekt SOLECTRUS nimmt an Fahrt auf und wird von immer mehr Nutzern eingesetzt. Ich habe daher eine separate Website eingerichtet, auf der zukünftig alle Informationen zu finden sind: [https://solectrus.de](https://solectrus.de) --- # Progressive image loading with BlurHash _Recipe for use with Ruby on Rails and Vue.js_ Author: Georg Ledermann Published: 2020-08-01 Tags: Ruby on Rails, Active Storage, Vue.js Canonical: https://ledermann.dev/blog/2020/08/01/progressive-image-loading-with-blurhash/ A whole photo, squeezed into a few dozen characters that render instantly as a blurred preview. Rails does the encoding, Vue.js the decoding. --- A [BlurHash](https://blurha.sh) is a compact representation of a placeholder for an image. It can be used to display a preview while the browser is loading the entire image. In this article, I want to show how to encode the BlurHash in a Ruby on Rails application and how to use it for progressive image loading in a Vue.js frontend. To start off, here is an example of a BlurHash string: ``` LlNA}44TN{kqyEtls:xux^tRRjRi ``` Because it's a small string (typically about 30 chars), it can be stored **inside the database** right along with other image metadata like width, height, geocoding, etc. The implementation requires two steps: First, the BlurHash string needs to be encoded from the original image and then the string representation can be decoded to draw the placeholder. It looks like this: There are open-source libraries in several languages for both steps. ## The Rails backend: Encoding the BlurHash First, we need to add the [blurhash](https://github.com/Gargron/blurhash) gem to the Rails application: ```bash bundle add blurhash ``` In this example, I want to use Active Storage for file storage, so extracting file metadata is done with analyzers. For images, there is a [built-in analyzer](https://api.rubyonrails.org/classes/ActiveStorage/Analyzer/ImageAnalyzer.html) which relies on MiniMagick for image processing. To keep this example simple, I've built a custom analyzer based on this class. For better performance, I recommend to use [ruby-vips](https://github.com/libvips/ruby-vips) and rewrite the analyzer from scratch. ```ruby # app/analyzers/my_image_analyzer.rb class MyImageAnalyzer < ActiveStorage::Analyzer::ImageAnalyzer def metadata read_image do |image| if rotated_image?(image) { width: image.height, height: image.width } else { width: image.width, height: image.height } end.merge blurhash(image) end end private def blurhash(image) # Create a thumbnail first, otherwise the BlurHash encoding is very slow thumbnail = image.resize('200x200>').auto_orient { blurhash: BlurHash.encode( thumbnail.width, thumbnail.height, thumbnail.get_pixels.flatten ) } rescue MiniMagick::Invalid => e logger.error "Error while encoding BlurHash: #{e}" {} end end ``` The new analyzer needs to be registered, which is typically done with an initializer: ```ruby # config/initializers/active_storage.rb require 'my_image_analyzer' Rails.application.configure do config.active_storage.analyzers = [ MyImageAnalyzer, ActiveStorage::Analyzer::VideoAnalyzer ] end ``` When the analyzer processes a file, it adds the BlurHash string to the file metadata, which is stored in the `active_storage_blobs` table of the database. Try it out: ```ruby class Post < ApplicationRecord has_one_attached :image end post = Post.create! post.image.attach( io: URI.open('https://images.unsplash.com/photo-1451153378752-16ef2b36ad05'), filename: 'example.jpg' ) post.image.analyze post.image.metadata # => { # identified: true, # width: 3872, # height: 2592, # blurhash: 'LlNA}44TN{kqyEtls:xux^tRRjRi', # analyzed: true # } ``` That's it for the backend. The next step is how to use the BlurHash string in the frontend. ## The Vue.js frontend: Using the BlurHash as a placeholder while loading For decoding pixels from a BlurHash string, we need to install the [blurhash](https://github.com/woltapp/blurhash) package: ```bash yarn add blurhash ``` Now we can build a simple Vue component that draws the pixels on a canvas. It takes a hash and an aspect ratio as props. To keep things performant, a minimal canvas (32 × 32 pixels) is used and resized via CSS. For styling stuff, I chose the wonderful [Tailwind CSS](https://tailwindcss.com/) library. ```vue ``` Now we can build a `LazyImage` component, which takes an image URL, a BlurHash string, and width & height. Via the IntersectionObserver API, it starts loading the image when it scrolls into the viewport. While loading, the BlurHash placeholder is displayed. First, I add [vue-intersect](https://github.com/heavyy/vue-intersect) which simplifies handling the observer: ```bash yarn add vue-intersect ``` This is the central part of the implementation: The `` component comes **without** the `src` attribute, which will be added later when the image enters the viewport. With opacity transition, the placeholder is faded into the fully loaded image: ```vue ``` Using the `LazyImage` component is simple: ```vue ``` ## More I have pushed the full source code to a [GitHub repository](https://github.com/ledermann/blurhash-vue) and created a [live demo](https://blurhash-vue.vercel.app/). ## Changelog - **2021-04-15**: I've updated the frontend example to Vue.js **3** and removed the `vue-intersect` library. Please look at the [commit](https://github.com/ledermann/blurhash-vue/commit/0d748220396325073185b952d8ccb4f1ac762353) in the GitHub repository for the changes. --- # Building Docker images, the performant way _Fasten your seat belt!_ Author: Georg Ledermann Published: 2020-01-29 Tags: Docker, Ruby on Rails Canonical: https://ledermann.dev/blog/2020/01/29/building-docker-images-the-performant-way/ Two to three times faster CI builds, and the trick is not clever caching. It is baking every dependency into a prepared base image once. --- Many of the Rails applications I build these days are deployed as [Docker](https://www.docker.com/) images. Unfortunately, building Docker images usually takes a long time. This article describes how the build process can be accelerated by a factor of 2-3. This text is a follow-up to my article [Dockerize Rails, the lean way](/blog/2018/04/19/dockerize-rails-the-lean-way/), where I described how to build small images. Now I want to write about how to reduce the time Docker needs to build such images. Based on the official Ruby images, building a Docker image for a Rails application takes several minutes. For an application I'm currently working on, executing `docker build .` takes about **5 minutes**. Because this is part of a CI process, this effort is repeated over and over again. This slows down the deployment process and costs money when it comes to external CI tools like [GitHub Actions](https://github.com/features/actions) or [CircleCI](https://circleci.com/), which are usually paid on a time basis. Much of the build time is used for installing dependencies, such as Linux packages, Ruby gems and Node.js modules. Some Ruby gems (e.g. Nokogiri or Puma) contain native extensions that have to be compiled first during installation, which takes additional time. Since these third-party components do not change constantly in the app, there is some room for optimization here. The idea is to create a base image with **pre-installed dependencies**. As a result, I published a repo called [**DockerRailsBase**](https://github.com/ledermann/docker-rails-base), a set of multi-stage base images for Rails applications using pre-installed dependencies. ## Performance comparison Before going into the details, I want to show the numbers. I compared build times using a typical Rails application. This is the result on my local machine, measured by executing `time docker build .` - Based on the official Ruby Alpine image: **4:50 min** - Based on DockerRailsBase: **1:57 min** As you can see, using DockerRailsBase is more than **2 times faster** compared to the official Ruby image. It saves nearly **3min** on every build. Of course, this may be different for your application, but it shows the potential. Note: Before I started timing, the base image was not present on my machine, so it was downloaded first, which took some time. If the base image is already downloaded, the build time is only 1:18min (**3 times faster**). ## Creating the base image I first made the following assumptions about the Rails application: - The app is compatible with [Ruby for Alpine Linux](https://github.com/docker-library/ruby/blob/master/3.0/alpine3.14/Dockerfile) - The app uses [Ruby on Rails](https://rubyonrails.org/) 6 - The app uses [PostgreSQL](https://www.postgresql.org/) database - The app installs Node modules with [Yarn](https://yarnpkg.com/) - The app compiles JS with [Webpacker](https://github.com/rails/webpacker) and/or [Asset pipeline (Sprockets)](https://github.com/rails/sprockets-rails) This is the case for most of my Rails applications. If your apps differ, a modified base image may help. To build a very small production image, [multi-stage building](https://docs.docker.com/develop/develop-images/multistage-build/) is used. There are two Dockerfiles in this repo, one for the first stage (called "Builder") and one for the resulting stage (called "Final"). ### First: Builder stage In this stage, the Ruby gems and Node modules are installed and the assets will be compiled. For doing this, some Alpine packages are installed first. Then some standard Ruby gems and Node modules are installed and some ONBUILD triggers are added to install the app's dependencies (by re-using the standard ones – this is the most important thing). Here are the main parts of this Dockerfile: ```docker # Builder/Dockerfile FROM ruby:alpine # Add basic packages RUN apk add build-base postgresql-dev git nodejs yarn tzdata file # Install standard Node modules COPY package.json yarn.lock /app/ RUN yarn install # Install standard gems COPY Gemfile* /app/ RUN bundle install #### ONBUILD: Add triggers to the image, executed later while building a child image # Install Ruby gems (for production only) # This will be fast because most of the gems are already installed ONBUILD COPY Gemfile* /app/ ONBUILD RUN bundle install --without development:test && \ bundle clean --force # Remove unneeded gems # Install Node modules (for production only) # This will be fast because most of the modules are already installed ONBUILD COPY package.json yarn.lock /app/ ONBUILD RUN yarn install # Copy the whole application folder into the image ONBUILD COPY . /app # Compile assets with Webpacker and/or Sprockets ONBUILD RUN bundle exec rails assets:precompile ``` **Some details removed**, so don't copy this! See the full and up-to-date Dockerfile in the GitHub repo: [https://github.com/ledermann/docker-rails-base/blob/master/Builder/Dockerfile](https://github.com/ledermann/docker-rails-base/blob/master/Builder/Dockerfile) For the selected Ruby gems and Node modules, see the Gemfile and packages.json: - [https://github.com/ledermann/docker-rails-base/blob/master/Builder/Gemfile](https://github.com/ledermann/docker-rails-base/blob/master/Builder/Gemfile) - [https://github.com/ledermann/docker-rails-base/blob/master/Builder/package.json](https://github.com/ledermann/docker-rails-base/blob/master/Builder/package.json) The result is two folders: One with the gems (`/usr/local/bundle`) and one with the app (`/app`). They will be copied into the final image in the final stage. ### Second: Final stage The final stage is used to build the production image. It installs just the bare minimum to keep the image small. Based on the Ruby Alpine image, it adds some packages needed for production and adds some ONBUILD triggers to copy the app and the gems from the "Builder" stage. Here are the main parts of this Dockerfile: ```docker # Final/Dockerfile FROM ruby:alpine # Add basic packages RUN apk add postgresql-client tzdata file # Copy app with gems from former build stage ONBUILD COPY --from=Builder /usr/local/bundle/ /usr/local/bundle/ ONBUILD COPY --from=Builder /app /app ``` **Some details removed**, so don't copy this! See the full and up-to-date Dockerfile in the GitHub repo: [https://github.com/ledermann/docker-rails-base/blob/master/Final/Dockerfile](https://github.com/ledermann/docker-rails-base/blob/master/Final/Dockerfile) ## How to use the base images The two DockerRailsBase images are published to Docker Hub. They can be used from the application's `Dockerfile`, which now can be very short and simple: ```docker # Builder stage FROM ledermann/rails-base-builder:latest AS Builder # Final stage FROM ledermann/rails-base-final:latest # Additional setup your production image requires, e.g. additional Alpine packages # RUN apk add ffmpeg vips USER app CMD ["bundle", "exec", "puma", "-C", "config/puma.rb"] ``` Yes, this is the complete Dockerfile of the Rails app. It is so simple because the main work is done by ONBUILD triggers. There are some interesting parts: - Using ONBUILD triggers allows us to move lots of standard stuff into the base image, so the app's Dockerfile can be very simple. Remember: With [ONBUILD](https://docs.docker.com/engine/reference/builder/#onbuild), a Dockerfile command is defined in the base image, but will be processed later, when the image is used as the base for another build. - The builder image includes lots of Ruby gems I'm using in many apps. If a particular gem is **not** used in one app, it will be wiped out by the `bundle clean` command and so will **not** be included in the resulting image. Of course, this enlarges the builder image (**not** your resulting image), but downloading is faster than installing. - If the app requires a Ruby gem or Node module in a different version, this doesn't matter. They will be installed, so the pre-installed version will not be used. Of course, the more of the pre-installed dependencies are used, the better. Hope this article is helpful for other developers. --- # Using UltraHook to receive webhooks at development _Alternative to ngrok.com_ Author: Georg Ledermann Published: 2019-03-17 Tags: Ruby on Rails, Stripe, UltraHook Canonical: https://ledermann.dev/blog/2019/03/17/using-ultrahook-to-receive-webhooks-at-development/ Testing webhooks locally is tricky: the sender cannot reach your laptop behind a firewall. UltraHook provides a free tunnel for incoming POST requests, so you can debug them on your own machine. --- When building web applications, it may be necessary to process incoming webhooks. With UltraHook these requests can also be received during development on the local computer. I'm currently working on a shop-like web application using Ruby on Rails that can process credit card payments with [Stripe](https://www.stripe.com/). Stripe sends out webhooks during certain events, so I need a way to test them on my local computer. With [UltraHook](https://www.ultrahook.com/) they can be received simply from behind a firewall. It is like ngrok.com, but for POST requests only. In contrast to [ngrok.com](https://ngrok.com/), the service is completely free and offers static endpoint URLs. In my Rails application I'm using [foreman](https://github.com/ddollar/foreman), [dotenv](https://github.com/bkeepers/dotenv) and [puma-dev](https://github.com/puma/puma-dev). It is simple to add UltraHook to this setup: **Step 1:** We need to register at [ultrahook.com](https://www.ultrahook.com/register) to get a personal API key and a namespace. Also, a small Ruby gem needs to be installed. **Step 2:** The API key needs to be added to the local `.env` file: ```bash STRIPE_PUBLIC_KEY=pk_test_12345678 STRIPE_PRIVATE_KEY=sk_test_12345678 STRIPE_WEBHOOK_SECRET=whsec_12345678 ULTRAHOOK_API_KEY=my-ultrahook-api-key # <= This is the added line ``` **Step 3:** Assuming the local URL (served by puma-dev) is `http://my-shop.test`, we need to add this line to the `Procfile`: ```bash backend: bin/rails s -p 3000 frontend: bin/webpack-dev-server ultrahook: ultrahook my-shop http://my-shop.test # <= This is the added line ``` **Step 4:** At the external service (e.g. Stripe), we add this endpoint URL to send webhooks to: ```bash http://my-shop.my-namespace.ultrahook.com ``` Finally, the application will be started as usual in the development environment. Now it handles incoming webhooks, too. ```bash foreman start ``` That's it. Just fire and forget. --- # Updating Docker services with Portainer webhooks _Continuous Delivery_ Author: Georg Ledermann Published: 2018-09-17 Tags: Docker, Portainer, Deployment, CI/CD Canonical: https://ledermann.dev/blog/2018/09/17/updating-docker-services-with-portainer-webhooks/ Deploying a new Docker image by hand means logging in, pulling, restarting. With a Portainer webhook, every successful CI build deploys itself. --- Some time ago I wrote about [using Portainer for Docker hosting](/blog/2018/03/29/migration-from-docker-cloud-to-portainer/) and [using GitLab as container registry](/blog/2018/04/05/switching-from-docker-hub-to-gitlab-container-registry/). Due to the further development of Portainer there is an interesting improvement regarding the installation of updates. To update my Docker services I've been using [Shepherd](https://github.com/djmaze/shepherd), a small tool that constantly polls the registry to see if a new Docker image is available and then fetches it from there, updating the running container with it. This works, but is not perfect: It leads to a basic load of the host and updates are delayed noticeably, especially if there are lots of services to check. It would be better if a **push** mechanism could be used instead of polling, so that action is taken only when an updated image is available. Exactly this is possible with the recently released version [1.19.2](https://github.com/portainer/portainer/releases/tag/1.19.2) of Portainer: Now you can enable a webhook for each service. This webhook reacts to POST requests and performs an update. This can easily be integrated into a CI/CD pipeline, I want to describe it using _GitLab CI_: ## Step 1: Activate webhook in Portainer First, the webhook for the desired service needs to be activated in Portainer. This generates an endpoint we will use later: Enabling webhooks in Portainer ## Step 2: Add CI variable in GitLab Second, this endpoint must be made known to the CI/CD. In _GitLab_ I'm adding a variable for this. Caution: If the same image is used for several services, several webhooks are required accordingly. Setting GitLab variables ## Step 3: Enhance .gitlab-ci.yml In the CI script `.gitlab-ci.yml` this variable needs to be used to post a request to the webhook after the image was successfully pushed to the Docker registry. The script will look like this: ```yaml release: script: - ... - docker build ... - docker push ... - curl -X POST $PORTAINER_HOOK_APP - curl -X POST $PORTAINER_HOOK_WORKER ``` That's it. Now, after a new image is released, the Portainer host gets informed about this, so an update can be installed immediately. BTW: I'm still using Shepherd, but for external images only, where an update check at longer intervals is sufficient, e.g. once a day. --- # Swagger with Rails: Know your options _How to document your API_ Author: Georg Ledermann Published: 2018-06-21 Tags: Ruby on Rails, API Canonical: https://ledermann.dev/blog/2018/06/21/swagger-with-rails-know-your-options/ Hand-written API docs are already wrong by the time you merge. Three Rails gems generate them from the source instead, and they disagree about how. --- Creating API documentation is an essential task that should not be done separately from implementation. The risk that implementation and documentation may diverge should not be underestimated. One way to document a RESTful API is to use the [OpenAPI specification](https://swagger.io/resources/open-api/), also known as _Swagger_. This way, every detail of the API is noted in a precisely defined JSON file. There are many tools available to process this kind of JSON file – e.g. with [Swagger UI](https://swagger.io/tools/swagger-ui/) there is an interactive tool to read the documentation in the browser and test the API (by building and executing `curl` commands). The JSON file can be created manually, but of course, an automated generation is a more elegant way. In the world of Ruby on Rails there are two popular approaches of doing this: - Enhance the controller via an additional DSL, so the JSON file can be generated from the controller - Leave the controller unchanged, but enhance the (integration) tests so that the documentation can be generated from it I found the following gems to support this: - [Swagger::Docs](https://github.com/richhollis/swagger-docs) - [Swagger::Blocks](https://github.com/fotinakis/swagger-blocks) - [rswag](https://github.com/domaindrivendev/rswag) **Swagger::Docs** is the oldest tool I found, it was born in 2013 and therefore only supports the old version v1.2 of the Swagger specification. There is no effort to support v2 or newer, which is probably the biggest disadvantage of this gem. With _Swagger::Docs_ you write your documentation directly to the API controllers via a custom DSL. The JSON file is generated via a rake task. **Swagger::Blocks** was inspired by _Swagger::Docs_ and has been developed since 2014. It supports v2 of the specification. The documentation is added to the controller. The special feature is that the JSON file is generated on-the-fly: Instead of a rake task, the JSON file is generated at runtime. **rswag** is the new kid in town (started 2016), supports v2 of to the specification and is integrated with RSpec, so the documentation is added to the integration tests. I like this approach because it forces you to test the various response options (_ok_, _not authorized_, _bad request_, etc.) and to ensure that the response body matches the schema. The JSON file is created by a rake task. As a bonus, Swagger UI is included. From my point of view, _rswag_ is the tool of choice because it best ensures that documentation and implementation fit together. --- # Exif analyzer for Active Storage _Extracting GPS location from uploaded images_ Author: Georg Ledermann Published: 2018-05-15 Tags: Ruby on Rails, Active Storage Canonical: https://ledermann.dev/blog/2018/05/15/exif-analyzer-for-active-storage/ Most photos your users upload know exactly where they were taken. Active Storage throws that away, so I wrote a small analyzer that keeps it. --- Rails 5.2 introduces [Active Storage](https://github.com/rails/rails/tree/master/activestorage), which can replace external file uploading gems like CarrierWave, PaperClip or Shrine. There are some helpful articles and tutorials about it, e.g. by [Evil Martians](https://evilmartians.com/chronicles/rails-5-2-active-storage-and-beyond), [GoRails](https://gorails.com/episodes/file-uploading-with-activestorage-rails-5-2) or [Drifting Ruby](https://www.driftingruby.com/episodes/in-depth-look-into-activestorage). I want to demonstrate how to add one more feature. Active Storage in Rails 5.2.0 is just the beginning. Recently, [Pull Request #32471](https://github.com/rails/rails/pull/32471) by Janko Marohnić (the author of Shrine) was merged, which allows us to use the [ImageProcessing](https://github.com/janko-m/image_processing) gem instead of mini_magick. This results in faster image processing, automatic orientation, thumbnail sharpening and more. Active Storage can extract metadata from uploaded images, but currently, this means `width` and `height` only. One thing I'm missing so far is extracting GPS location (latitude / longitude / altitude) from the Exif part. The following describes how this feature can be added by using the gem [exifr](https://github.com/remvee/exifr). In Active Storage there are _Analyzers_ to extract metadata – for every file type there is a separate analyzer – currently for images and videos only, but you can add your own. For images, there is the [ImageAnalyzer](https://github.com/rails/rails/blob/v5.2.0/activestorage/lib/active_storage/analyzer/image_analyzer.rb). Because of the internal structure of this class, the only way of enhancement seems to be monkey patching. First, add this to your Gemfile: ```ruby gem 'exifr' ``` Then, add the file `config/initializers/exif.rb` with this content: ```ruby require 'exifr/jpeg' module ActiveStorage class Analyzer::ImageAnalyzer < Analyzer def metadata read_image do |image| if rotated_image?(image) { width: image.height, height: image.width } else { width: image.width, height: image.height } end.merge(gps_from_exif(image) || {}) end rescue LoadError logger.info "Skipping image analysis because the mini_magick gem isn't installed" {} end private def gps_from_exif(image) return unless image.type == 'JPEG' if exif = EXIFR::JPEG.new(image.path).exif if gps = exif.fields[:gps] { latitude: gps.fields[:gps_latitude].to_f, longitude: gps.fields[:gps_longitude].to_f, altitude: gps.fields[:gps_altitude].to_f } end end rescue EXIFR::MalformedImage, EXIFR::MalformedJPEG end end end ``` That's all. Now for every processed image the GPS location data (if there is any) will be extracted as metadata and stored to the database in the table `active_storage_blobs`. --- # Dockerize and configure a JavaScript single-page application _A way of accessing environment variables_ Author: Georg Ledermann Published: 2018-04-27 Tags: JavaScript, Docker, nginx Canonical: https://ledermann.dev/blog/2018/04/27/dockerize-and-configure-javascript-single-page-application/ Packing a compiled single-page app into a Docker image is easy. The harder part is making one image behave differently in staging and production without rebuilding it. --- Building a lean Docker image for delivering a single-page JavaScript application is simple. But it is not that easy when it comes to configuring with environment variables. Let's say we have a [Vue.js](https://vuejs.org/) application which is compiled with `yarn build` for production. The files to be delivered are placed in the `/dist` folder with a simple `index.html` to load the JS code. It doesn't matter if your JavaScript application is based on React, Angular or something else, the following steps would be similar. ## Building the smallest possible Docker image After compilation with `yarn build`, we don't need the large `node_modules/` folder with its hundreds of MB. We even don't need Node.js or Yarn. In production, we just need a web server to deliver the compiled files. So, to build the smallest possible Docker image for production, a [multi-stage](https://docs.docker.com/develop/develop-images/multistage-build/) `Dockerfile` is recommended: First, the build process is performed by Node.js and Yarn. The resulting artifacts are then copied to a new image based on the official `nginx` image: ```docker # First step: Build with Node.js FROM node:alpine AS Builder WORKDIR /app COPY package.json yarn.lock /app/ RUN yarn install COPY . /app RUN yarn build # Use plain nginx to deliver the dist folder only FROM nginx:stable-alpine COPY nginx.conf /etc/nginx/conf.d/default.conf COPY --from=Builder /app/dist /usr/share/nginx/html ``` We also need an `nginx` simple configuration file named `nginx.conf`: ```nginx server { listen 80 default_server; listen [::]:80 default_server; root /usr/share/nginx/html; index index.html; location / { # Support the HTML5 History mode of the vue-router. # https://router.vuejs.org/en/essentials/history-mode.html try_files $uri $uri/ /index.html; } } ``` Okay, that's all. The result is a very small image ([~ 8MB](https://microbadger.com/images/ledermann/docker-vue) download size for my example application) and can be used in production. ## Configure the container with environment variables With Docker, containers are configured by environment variables. For example, this is used to define backend URLs, API access tokens etc. Assume we want to use the following `docker-compose.yml`: ```docker version: '3.4' services: app: image: 'ledermann/docker-vue' ports: - '80' environment: - VUE_APP_BACKEND_HOST=backend.example.com - VUE_APP_MATOMO_HOST=matomo.example.com - VUE_APP_MATOMO_ID=42 ``` Now we have a problem: We deliver compiled JavaScript with `nginx`. There is no `Node.js`, so we have no access to Docker's environment variables. However, the following approach allows configuration via environment variables: 1. In your JS code, use strings like `'$VUE_APP_BACKEND_HOST'` and assume they contain the configuration value 2. On container startup, modify the existing JS files with search & replace to set values 3. In development, just use `process.env` ### Step 1: Add Configuration class to the JS code In development, we use Node.js, so [process.env](https://nodejs.org/api/process.html#process_process_env) has access to the local environment. There is a package called [dotenv](https://github.com/motdotla/dotenv) to load env vars from a file. Add this to your project: ```bash $ yarn add dotenv ``` Remember: We need this in development only. In production, there is no Node.js at runtime, so `process.env` doesn't include any environment variable! To encapsulate application configuration, I have created a simple class named `Configuration` to access environment variables both in development and production. It includes config strings named `$VUE_APP_XXX` to be replaced later at container startup (see step 2): ```javascript import dotenv from "dotenv"; dotenv.config(); export default class Configuration { static get CONFIG() { return { backendHost: "$VUE_APP_BACKEND_HOST", matomoHost: "$VUE_APP_MATOMO_HOST", matomoId: "$VUE_APP_MATOMO_ID", }; } static value(name) { if (!(name in this.CONFIG)) { console.log(`Configuration: There is no key named "${name}"`); return; } const value = this.CONFIG[name]; if (!value) { console.log(`Configuration: Value for "${name}" is not defined`); return; } if (value.startsWith("$VUE_APP_")) { // value was not replaced, it seems we are in development. // Remove $ and get current value from process.env const envName = value.substr(1); const envValue = process.env[envName]; if (envValue) { return envValue; } else { console.log( `Configuration: Environment variable "${envName}" is not defined`, ); } } else { // value was already replaced, it seems we are in production. return value; } } } ``` Usage example: ```javascript import Configuration from "configuration"; var backendHost = Configuration.value("backendHost"); console.log(backendHost); ``` ### Step 2: Replace vars on container startup First, we need a bash script named `entrypoint.sh` to run on every container startup: ```bash #!/bin/sh # Replace env vars in JavaScript files echo "Replacing env vars in JS" for file in /usr/share/nginx/html/js/app.*.js; do echo "Processing $file ..."; # Use the existing JS file as template if [ ! -f $file.tmpl.js ]; then cp $file $file.tmpl.js fi envsubst '$VUE_APP_BACKEND_HOST,$VUE_APP_MATOMO_HOST,$VUE_APP_MATOMO_ID' < $file.tmpl.js > $file done echo "Starting nginx" nginx -g 'daemon off;' ``` What this script does: - The first time the container is started, the existing `app.*.js` files are copied and used as a template for the following search & replace (line 9-12). - The main part (line 14) is to use [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html), which is included in the `nginx` Docker image. It replaces strings in a file with the values of the given environment variables. To use this script as entrypoint, add this lines to the `Dockerfile` described above: ```docker COPY entrypoint.sh / ENTRYPOINT ["/entrypoint.sh"] ``` ### Step 3: Make use of dotenv in development It is more simple to use environment variables in development. Because we have included the `dotenv` package, we can place a file called `env.local` with this content: ```bash VUE_APP_BACKEND_HOST="backend.my-site.dev" VUE_APP_MATOMO_HOST="matomo.my-site.com" VUE_APP_MATOMO_ID="42" ``` ## Result You find the complete code in my [DockerVue](https://github.com/ledermann/docker-vue) example application on GitHub. --- # Dockerize Rails, the lean way _Size matters_ Author: Georg Ledermann Published: 2018-04-19 Tags: Docker, Ruby on Rails Canonical: https://ledermann.dev/blog/2018/04/19/dockerize-rails-the-lean-way/ My Rails Docker image started at 1.6 GB. With an Alpine base and multi-stage builds I got it down to 329 MB. --- Building a Docker image for a given Rails application is easy – unless you want the Docker image as small as possible. Docker is awesome, but handling large files is annoying. Read how I have reduced the image size of a Rails application from 1.6GB to 329MB. There are several ways to reduce the size of a Docker image. Here I want to describe some of them: 1. Use Alpine as the base image 2. Use Multi-stage building 3. Beware of the `chown` pitfall 4. Remove Bundler cache 5. Remove parts of the app not needed in resulting image ## 1. Use Alpine as the base image The [official Ruby Docker image](https://hub.docker.com/r/library/ruby/tags/2.5.1/) is based on Debian _Stretch_. It is about 860MB in size, which is quite a lot. But there is help in the form of an [Alpine image](https://hub.docker.com/r/library/ruby/tags/2.5.1-alpine/), which is only 55 MB. Great, but there are some things different: Except Ruby, almost nothing is included, so you have to add the required Linux packages by yourself. Note: Currently (April 2018) there is an [open issue](https://github.com/docker-library/ruby/issues/196) with the Alpine image for Ruby 2.5, which leads to some **Stack level too deep** errors. Until this issue is resolved, you must use the Ruby 2.4 Alpine image, which works well. **Update 2018-10-03:** The issue is [fixed](https://github.com/docker-library/ruby/pull/237). ## 2. Use multi-stage building Last year [Docker 17.05](https://docs.docker.com/release-notes/docker-ce/#17050-ce-2017-05-04) introduced [multi-stage builds](https://docs.docker.com/develop/develop-images/multistage-build/), which are a great way to get lean images by separating the building steps from the bundling steps. The building steps require all the build tools to be present (e.g. git, nodejs, yarn, development packages). In the resulting image, they are not needed. The idea is to first build the stuff and then copy only the resulting artifacts into the final image. Think about separate Dockerfiles with a feature to copy some files from one image to another. Simplified example: ```docker FROM ruby:alpine as Builder RUN apk add build-base yarn git postgresql-dev . . RUN bundle install RUN rake assets:precompile . . FROM ruby:alpine RUN apk add postgresql-client . . COPY --from=Builder /compiled-files /compiled-files ``` ## 3. Beware of the chown pitfall It is a common practice to set the owner of some files/folders via `chown` after copying the files. But with Docker this can unintentionally increase image size. See this fragment of a Dockerfile: ```docker COPY . /dest-folder RUN chown -R someuser:somegroup /dest-folder ``` This works, but the result is two layers with the same size (use `docker history` to see the size added by each layer). Since [Docker 17.09](https://docs.docker.com/release-notes/docker-ce/#17090-ce-2017-09-26) there is a way to do the same things without wasting space: ```docker COPY --chown=someuser:somegroup . /dest-folder ``` Just one layer, no size duplication. Read more about this here: [https://blog.mornati.net/docker-images-and-files-chown](https://blog.mornati.net/docker-images-and-files-chown) ## 4. Remove Bundler cache Bundler installs some files not needed in production, so you can delete them: - Cache folder - C source files and compiled object files (for gems with native extensions) ```docker RUN rm -rf /usr/local/bundle/cache/*.gem \ && find /usr/local/bundle/gems/ -name "*.c" -delete \ && find /usr/local/bundle/gems/ -name "*.o" -delete ``` ## 5. Remove parts of the app not needed in resulting image If your Rails application uses _Yarn_ to manage JavaScript packages, you will find lots of MB (usually 100MB and more) in the folder `node_modules/`. After the assets are precompiled, they are not needed anymore. Delete them in the build stage to save space in the resulting image. Besides that, precompiling assets leave a large number of files behind in `tmp/cache/`, which are not needed in production. While you're at it, there are some more files you don't need in production: The `spec/` (or `test/`) folder and the `assets` folders. ```docker # Remove folders not needed in resulting image RUN rm -rf node_modules tmp/cache app/assets vendor/assets lib/assets spec ``` # Result For the implementation of the described actions I used my all-time example application [DockerRails](https://github.com/ledermann/docker-rails). Here is the resulting Dockerfile: ```docker ###################### # Stage: Builder FROM ruby:2.5.1-alpine as Builder RUN apk add --update --no-cache \ build-base \ postgresql-dev \ git \ imagemagick \ nodejs-current \ yarn \ tzdata WORKDIR /app # Install gems ADD Gemfile* /app/ RUN bundle config --global frozen 1 \ && bundle install --without development test -j4 --retry 3 \ # Remove unneeded files (cached *.gem, *.o, *.c) && rm -rf /usr/local/bundle/cache/*.gem \ && find /usr/local/bundle/gems/ -name "*.c" -delete \ && find /usr/local/bundle/gems/ -name "*.o" -delete # Install yarn packages COPY package.json yarn.lock /app/ RUN yarn install # Add the Rails app ADD . /app # Precompile assets RUN RAILS_ENV=production SECRET_KEY_BASE=foo bundle exec rake assets:precompile # Remove folders not needed in resulting image RUN rm -rf node_modules tmp/cache app/assets vendor/assets lib/assets spec ############################### # Stage wkhtmltopdf FROM madnight/docker-alpine-wkhtmltopdf as wkhtmltopdf ############################### # Stage Final FROM ruby:2.5.1-alpine LABEL maintainer="mail@georg-ledermann.de" # Add Alpine packages RUN apk add --update --no-cache \ postgresql-client \ imagemagick \ tzdata \ file \ # needed for wkhtmltopdf libcrypto1.0 libssl1.0 \ ttf-dejavu ttf-droid ttf-freefont ttf-liberation ttf-ubuntu-font-family # Copy wkhtmltopdf from former build stage COPY --from=wkhtmltopdf /bin/wkhtmltopdf /bin/ # Add user RUN addgroup -g 1000 -S app \ && adduser -u 1000 -S app -G app USER app # Copy app with gems from former build stage COPY --from=Builder /usr/local/bundle/ /usr/local/bundle/ COPY --from=Builder --chown=app:app /app /app # Set Rails env ENV RAILS_LOG_TO_STDOUT true ENV RAILS_SERVE_STATIC_FILES true ENV EXECJS_RUNTIME Disabled WORKDIR /app # Expose Puma port EXPOSE 3000 # Save timestamp of image building RUN date -u > BUILD_TIME # Start up ENTRYPOINT ["docker/startup.sh"] ``` Now the resulting image is 329MB, its (compressed) download size is [121MB](https://microbadger.com/images/ledermann/docker-rails). **Update 2020-01-29:** There is a follow-up article in which I show how to reduce the building time: [Building Docker images, the performant way](/blog/2020/01/29/building-docker-images-the-performant-way/) --- # Switching from Docker Hub to GitLab Container Registry _Continuous Delivery_ Author: Georg Ledermann Published: 2018-04-05 Tags: Docker, Deployment, CI/CD Canonical: https://ledermann.dev/blog/2018/04/05/switching-from-docker-hub-to-gitlab-container-registry/ Docker Hub had become the bottleneck in my pipeline: slow builds and large files shuffled across the Atlantic. Moving to GitLab's container registry fixed both. --- An automated build process is essential for software development. After every single change in the source code – no matter how small – the software is completely assembled and tested automatically. In my case, the result is usually a Docker image. So far I've used [Docker Hub](https://hub.docker.com/) to manage my Docker images. This was quite comfortable but had some disadvantages: - It took up to 30 minutes between the commit and the built image. This is because I used Docker Hub not only as a registry (to store the images) but also to build the images: A build was initiated by a webhook at the GitHub repo. Because the CPU resources provided by Docker Hub are limited, the process of building images was very slow. - Docker images are quite large and are stored in the USA. The frequent transfer of large files between Germany and the USA causes a lot of traffic. - More and more often the building failed because of an **Internal server error** at Docker Hub. This is very bad when it comes to production deployment. - The service is not free, Docker charges a small fee for hosting of private images. After I recently [said goodbye to DockerCloud](/blog/2018/03/29/migration-from-docker-cloud-to-portainer/), I don't want to use Docker Hub for private repositories anymore. Since I've been using GitLab CE for several years for Continuous Integration, it is about time to use the **GitLab Container Registry** for storing Docker images. ## 1. Setup GitLab with Container Registry The GitLab Container Registry was [introduced in 2016 with GitLab 8.8](https://about.gitlab.com/2016/05/23/gitlab-container-registry/). It just needs to be enabled. I'm using the [official Docker image for GitLab CE](https://hub.docker.com/r/gitlab/gitlab-ce/) to run GitLab on my own server behind [nginx-proxy](https://github.com/jwilder/nginx-proxy) with the [letsencrypt-nginx-proxy-companion](https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion). To install GitLab with the enabled registry, I use the following Docker Compose file (reduced to the relevant parts): ```yaml version: "3.4" services: web: environment: GITLAB_OMNIBUS_CONFIG: | external_url 'https://gitlab.example.com' registry_external_url 'http://registry.gitlab.example.com' nginx['listen_port'] = 80 nginx['listen_https'] = false nginx['proxy_set_headers'] = { 'X-Forwarded-Proto' => 'https', 'X-Forwarded-Ssl' => 'on' } VIRTUAL_HOST: gitlab.example.com,registry.gitlab.example.com VIRTUAL_PORT: 80 LETSENCRYPT_EMAIL: info@example.com LETSENCRYPT_HOST: gitlab.example.com,registry.gitlab.example.com CERT_NAME: gitlab.example.com image: "gitlab/gitlab-ce:latest" ports: - "22" - "80" ``` Now Gitlab is available with the browser at **https://gitlab.example.com**, the registry API is available at **https://registry.gitlab.example.com**. After enabling the registry for a given project, we can push an image to our own registry like this: ```bash ~ $ docker login registry.gitlab.example.com -u myusername -p mypassword ~ $ cd myrepo ~ $ docker build -t registry.gitlab.example.com/name/repo:latest . ~ $ docker push registry.gitlab.example.com/name/repo:latest ``` Nice. Next step is automation. ## 2. Setting up a GitLab Runner For the GitLab Runner I'm using a small virtual machine at [Hetzner](https://www.hetzner.de/cloud/). For about five bucks per month, you get the CX21, a virtual Linux host with 2 vCPU, 4GB RAM and 40GB SSD - enough for GitLab Runner. [Installing GitLab Runner](https://docs.gitlab.com/runner/install/linux-repository.html) is quite simple, I'll skip this here. ## 3. Configure project For each project to be tested with GitLab CI, a file **.gitlab-ci.yml** is required in the root of the repo. It also covers the pushing of the image. The following minimalistic one creates the image, performs the tests and pushes the image to the registry. ```yaml test: before_script: - echo $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY script: - docker build -t $CI_REGISTRY_IMAGE:latest . - docker run --rm $CI_REGISTRY_IMAGE:latest bundle exec rake test - docker push $CI_REGISTRY_IMAGE:latest ``` Read on for [details about the available CI/CD variables](https://docs.gitlab.com/ce/ci/variables/) or the [configuration of the jobs](https://docs.gitlab.com/ce/ci/yaml/). ## The result Overall, my build process has improved in several ways: - Now it takes less than 10 minutes from commit to the finished Docker image - The images are transferred within the Hetzner data center only (fast and without traffic costs) - There are no costs for an external registry service --- # Migration from Docker Cloud to Portainer _Bye, Bye, Docker Cloud_ Author: Georg Ledermann Published: 2018-03-29 Tags: Docker, Portainer, Hosting, Deployment Canonical: https://ledermann.dev/blog/2018/03/29/migration-from-docker-cloud-to-portainer/ Docker shut down Docker Cloud, leaving more than fifty of my containers without a home. I moved them to Portainer: lightweight, self-hosted, and easier to switch to than I expected. --- Nothing lasts forever: A few days ago, Docker Inc. announced the [end of "Docker Cloud"](https://success.docker.com/article/cloud-migration). This article describes the migration of my single node installation to [Portainer](https://portainer.io) – an open-source lightweight management UI for Docker. Docker Cloud (formerly known as "Tutum" before the acquisition by Docker Inc.) is an orchestration tool to manage a container infrastructure. With a nice web interface, it supports the administration of stacks, services, and containers – including automatic updates and monitoring. This also works well with a single docker node (slogan: "Bring your own host"), which is my use case. I'm running multiple applications in more than 50 containers on a bare-metal Linux server. Docker gives its users 60 days to switch to another software. Since I've managed all my services and applications in production with Docker Cloud since 2016, I had to look for another tool right now. ## What exactly is a replacement needed for? I want to continue running my containers on my own host, which is a dedicated Linux machine at [Hetzner](https://www.hetzner.com/de/dedicated-rootserver/matrix-ex) in a German data center. In summary, I'm looking for a solution for the following tasks: - **Web interface:** Docker can be managed by the command line, but I want to use a web browser. - **Auto-Redeploy:** If a new image is available, the dependent containers must be redeployed automatically. - **Monitoring:** In case of relevant events (e.g. failures, container updates), a Slack notification should be performed There are lots of alternative products out there, I've chosen Portainer. BTW, Portainer wants to be a [replacement for Rancher](https://x.com/portainerio/status/814542425410576386). An essential requirement for Portainer, however, is that the _Swarm Mode_ is required to use "Stacks". The swarm mode can be activated on a single host, too. But all other services will have to deal with it – there are some pitfalls. ## 1. Getting started Because a very old version of Docker was used on my host (due to the requirements of Docker Cloud) and an update would cause downtimes, I decided to build the setup from scratch on a new host. ### Install Docker The first steps are simple: First, you have to complete the [default installation](https://docs.docker.com/install/linux/docker-ce/ubuntu/) of Docker. At the time of this writing, this is `18.03.0-ce`. ### Add registry authentication For the use of private images, login to your registry is required: ```bash ~ $ docker login ``` This results in the `/root/.docker/config.json` file, which will be needed later for automatic image updates. ### Prepare persistent volumes Because in my case there is only one node in the swarm, I store all persistent volumes locally on the host in a specific folder: ```bash ~ $ mkdir /my-volumes ``` ## 2. Install Shepherd for automatic updates With Docker Cloud, the availability of a new image can auto-upgrade the corresponding containers: Placing an `autoredeploy=true` is all you need (if the image is stored at Docker Hub). Now, without Docker Cloud, you have to handle this by yourself. One solution is [WatchTower](https://github.com/v2tec/watchtower/), but this is not suitable for the swarm mode. Fortunately, there is another tool named [Shepherd](https://github.com/djmaze/shepherd) – which runs `docker service update` for all existing services via a simple bash script in an endless loop. There is a small problem here with private images, which can easily be corrected (see [PR#10](https://github.com/djmaze/shepherd/pull/10) by me). Because it is not merged yet, I'm using my own image that contains this fix. Install it as a service: ```bash ~ $ docker service create --name shepherd \ --env SLEEP_TIME="3m" \ --env BLACKLIST_SERVICES="shepherd" \ --env WITH_REGISTRY_AUTH="true" \ --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock,ro \ --mount type=bind,source=/root/.docker/config.json,target=/root/.docker/config.json,ro \ ledermann/shepherd ``` By default, the old containers are still present (as `stopped`) after a service update. To change this, we can tell Docker to use a history limit of `1`, so the old containers are always removed: ```bash docker swarm update --task-history-limit 1 ``` # 3. Install Slack-Notifier for monitoring To be notified about Docker events on your hosts (e.g. starting container, stopping container etc.), there is a Slack integration tool called [slack-docker](https://github.com/int128/slack-docker). Install it as as service, too: ```bash ~ $ docker service create \ --name slack-notifier \ --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \ --env webhook=https://hooks.slack.com/services/some/secret/hook \ int128/slack-docker ``` There is another useful tool to mention: [monitor-docker-slack](https://github.com/DennyZhang/monitor-docker-slack) permanently queries the status of all containers and sends a notification if a stopped or unhealthy container is found. # 4. Install nginx-proxy as reverse proxy A reverse proxy is required to route incoming requests to virtual hosts. So far I was very satisfied with the [nginx-proxy](https://github.com/jwilder/nginx-proxy), which can be used in conjunction with the [letsencrypt-nginx-proxy-companion](https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion) to get certificates by "Let's Encrypt". Nice and stable. I'll install the [alpine](https://github.com/jwilder/nginx-proxy#jwildernginx-proxyalpine) image because it allows using `http/2` out of the box. First we create some folders for persisting volumes: ```bash ~ $ mkdir -p /my-volumes/proxy/certs ~ $ mkdir -p /my-volumes/proxy/vhost.d ~ $ mkdir -p /my-volumes/proxy/html ``` Then we save this as `docker-compose.yml`: ```bash version: '3.4' services: nginx-proxy: image: 'jwilder/nginx-proxy:alpine-0.7.0' ports: - target: 80 published: 80 protocol: tcp mode: host - target: 443 published: 443 protocol: tcp mode: host restart: always volumes: - '/var/run/docker.sock:/tmp/docker.sock:ro' - '/my-volumes/proxy/certs:/etc/nginx/certs:ro' - '/my-volumes/proxy/vhost.d:/etc/nginx/vhost.d' - '/my-volumes/proxy/html:/usr/share/nginx/html' labels: com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true" letsencrypt: image: 'jrcs/letsencrypt-nginx-proxy-companion:stable' restart: always volumes: - '/var/run/docker.sock:/var/run/docker.sock:ro' - '/my-volumes/proxy/certs:/etc/nginx/certs' - '/my-volumes/proxy/vhost.d:/etc/nginx/vhost.d' - '/my-volumes/proxy/html:/usr/share/nginx/html' ``` Important notes: - The current release `0.7.0` of nginx-proxy supports the Docker swarm mode, the older ones are not compatible. - Publishing the ports in `host` mode is needed to get the real IP in the containers To create a stack named `reverse-proxy`, run this: ```bash cat docker-compose.yml | docker stack deploy --compose-file - reverse-proxy ``` ## 5. Install Portainer First, create a folder for the volume: ```bash ~ $ mkdir -p /my-volumes/portainer/data ``` Then install Portainer as a docker service: ```bash ~ $ docker service create \ --name portainer \ --publish 80:9000 \ --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \ --mount type=bind,src=/my-volumes/portainer/data,dst=/data \ --env VIRTUAL_HOST=portainer.example.org \ --env VIRTUAL_PORT=9000 \ portainer/portainer \ -H unix:///var/run/docker.sock ``` For serving Portainer behind nginx-proxy, we need to add an [additional config](https://portainer.readthedocs.io/en/stable/faq.html#how-can-i-configure-my-reverse-proxy-to-serve-portainer). Append this to `/my-volumes/proxy/vhost.d/myportainer.example.org`: ```bash # Allow https via nginx-proxy location /portainer/ { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass http://portainer/; } # Allow Container console location /portainer/api/websocket/ { proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_http_version 1.1; proxy_pass http://portainer/api/websocket/; } ``` ## 6. Install application stacks Finally, the applications you want to host can be added as separate stacks from within Portainer. ## Result After all, this is my Portainer dashboard as seen in the browser: Portainer Dashboard The migration took me a few days and drops the dependency on an external service. --- # Starting a blog – again _Next try_ Author: Georg Ledermann Published: 2018-03-28 Tags: Self Canonical: https://ledermann.dev/blog/2018/03/28/starting-a-blog-again/ This is not my first blog; the earlier ones quietly faded away. This time I want a lasting place to write about software development and what I learn along the way. --- Once again, after several attempts in the last century, I'm starting a blog about my life in the world of software development. I'll write about lessons learned, opinions and maybe more. Don't expect daily posts or perfect English.