Tag Archives: openbsd

Migrating from Gallery Menalto 1.x to Piwigo: An Open Source Solution

Automation, Database, Python, , ,

Introduction

Migrating a large collection of photos and albums from an outdated photo gallery software can be a daunting task. With the aim of addressing this challenge, I have developed a robust framework to facilitate the migration from Gallery Menalto to Piwigo. This open-source project, licensed under GPL 3, not only aids in migrating to Piwigo but also provides the flexibility to migrate to any photo gallery software with an open API.

Background

Around 2004, I deployed a self-hosted photo gallery to organize, store, and maintain digital photos. At the time, this was a new area for me, and I had little experience managing digital images. The motivation was clear: my first daughter had just been born, and I wanted to ensure that these precious memories were easily accessible not only for myself but also for other family members, including grandparents.

For various reasons, I chose Gallery Menalto as my gallery software and ran it on OpenBSD 4.0 with a Geeklog 1.3 website. This setup was hosted in my closet in an apartment in Finland. Ten years later, several more daughters had been born, and I had moved to Canada. Despite the passage of time, both GeekLog and Gallery Menalto were still running on the same old OpenBSD system, which I had virtualized on top of VMware Server 2.0. The whole system had moved from the closet to the basement.

By then, I had amassed over 10,000 photos, and I was increasingly stressed about how to migrate them from the legacy system. Although I had attempted several times to upgrade from OpenBSD to Linux, the process was far from simple.

Another ten years passed, and the system was back in Finland, still running on a virtualized OpenBSD on top of VMPlayer. I couln’t even get the VM to run on libvirt because the OpenBSD 4.0 can’t handle that. Upgrading had become even more challenging; what was a legacy system ten years ago had become practically prehistoric.

I searched extensively for a solution, but nothing seemed to work. The Gallery version was 1.4, which couldn’t be upgraded to 2.x and then to 3.x, making the transition to something like Piwigo impossible. OpenBSD hadn’t received any updates either, but the environment was jail-rooted, and Gallery 1.x, with its known internals, didn’t pose significant security risks – it was wide open anyway.

Since no ready-made solution was available, I decided to create my own. I started by migrating all the articles from Geeklog to WordPress, a task that took about two nights and was relatively straightforward – if you can call something where you need to have gateway MySQL to run on docker container because of the legacy source system and bunch of python scripts straight forward, it was straightforward. Next, I delved into the internals of Gallery 1.4, experimenting through trial and error. Within a few nights, I had a plan.

Instead of attempting to migrate everything at once, I decided to collect the metadata from Gallery 1.x in one phase and perform the actual migration in a second, separate phase. This approach meant that the first phase was not dependent on the second, allowing me to choose any target gallery software if I decided against Piwigo later on.

Once this idea took shape, I managed to get everything working relatively quickly. I started the migration yesterday, and as of now, I have successfully completed it. I had to make a few changes to the code along the way, but the final version is now available in my GitHub repository.

Project Overview

The project comprises two main Python scripts:

  1. Collect Gallery Metadata (collect_gallery_meta_data.py): This script gathers meta information for all albums and photos from Gallery Menalto.
  2. Execute Migration (execute_migration.py): This script performs the actual migration of albums and photos, along with their metadata, to Piwigo.

Collecting Gallery Metadata

The collect_gallery_meta_data.py script is designed to traverse through the albums and photos in a Gallery Menalto installation, collecting crucial metadata and storing it in a MySQL database (or any SQLAlchemy-supported database). This metadata includes:

  • Album name
  • Parent album
  • Album caption
  • Album title
  • Album description
  • Photo filename
  • Photo caption
  • Photo title

How It Works

  1. Command-line Argument: The script takes an album located in the Gallery 1.x root as a command-line argument.
  2. Traversal: It recursively goes through all sub-albums and photos, collecting metadata.
  3. Storage: The collected data is stored in database tables (albums and photos)

Executing the Migration

The execute_migration.py script utilizes the metadata collected to perform the migration to Piwigo. It:

  • Creates the root album and sub-albums in Piwigo.
  • Uploads photos along with their metadata (capture date, upload date, caption, title, description) to Piwigo.

Key Features

  1. Download Photos: Photos are downloaded from Gallery Menalto only once. The field downloaded is set to 1 when the download is complete.
  2. Upload Photos: Once a photo is successfully uploaded to Piwigo, the field uploaded is set to 1.
  3. Track Migration: The albums table has a column migrated, which is set to 1 once an album is migrated. This ensures the process can be safely resumed if interrupted.

Advantages and Applications

This framework is highly beneficial for users who wish to transition from Gallery Menalto to modern photo gallery software. Its primary advantages include:

  1. Flexibility: While designed for Piwigo, the framework can be adapted for any photo gallery software with an open API.
  2. Efficiency: Metadata and photos are efficiently handled and migrated, ensuring data integrity and completeness.
  3. Open Source: As an open-source project under the GPL 3 license, it encourages collaboration and customization.

Getting Started

Clone the Repository:

git clone https://github.com/elsonico/gallery-piwigo-migration.git
cd gallery-piwigo-migration

Set Up the Database:

Ensure you have a MySQL database (or any SQLAlchemy-supported database) ready. Here’s the DDL for the two tables:

albums

CREATE TABLE `albums` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` varchar(255) NOT NULL,
  `parent_id` int DEFAULT NULL,
  `metadata` text,
  `meta` text,
  `title` varchar(255) DEFAULT NULL,
  `caption` text,
  `description` text,
  `migrated` tinyint(1) DEFAULT '0',
  `created` tinyint(1) DEFAULT '0',
  PRIMARY KEY (`id`),
  KEY `parent_id` (`parent_id`),
  CONSTRAINT `albums_ibfk_1` FOREIGN KEY (`parent_id`) REFERENCES `albums` (`id`)
)

photos

CREATE TABLE `photos` (
  `id` int NOT NULL AUTO_INCREMENT,
  `album_id` int DEFAULT NULL,
  `filename` varchar(255) NOT NULL,
  `caption` text,
  `metadata` text,
  `meta` text,
  `type` varchar(10) DEFAULT NULL,
  `description` text,
  `url` varchar(255) DEFAULT NULL,
  `downloaded` tinyint(1) DEFAULT '0',
  `uploaded` tinyint(1) DEFAULT '0',
  `capturedate` datetime DEFAULT NULL,
  `uploaddate` datetime DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `album_id` (`album_id`),
  CONSTRAINT `photos_ibfk_1` FOREIGN KEY (`album_id`) REFERENCES `albums` (`id`)
)

Collect Metadata

Before starting the metadata collection you need to have few environment parameters set, since the code expects to have them:

  • GALLERY_BASE_URL
    • This is your gallery URL including your album location: https://[gallery_host]/[albums]
    • This is the location you can find your albums.dat
  • DATABASE_URL
    • This is your MySQL URL: mysql+pymysql://[user]:[password@[host]/[database]

Once all above including the database tables are created your are good to go:

python collect_gallery_meta_data.py [root_album]

Execute Migration

As for collecting the metadata for doing the migration you need several environment variables set:

  • GALLERY_BASE_URL
    • Same as above depending on your configuration: https://[gallery_host]/[albums]
  • PIWIGO_API_URL
    • This is: https://[piwigo_host]/piwigo/ws.php
  • PIWIGO_USERNAME / PIWIGO_PASSWORD
    • Piwigo credentials with privileges to create albums and add photos
  • MIG_DB_HOST / MIG_DB_NAME
    • The database host and database your store the albums and database tables
  • MIG_DB_USER / MIG_DB_PASSWORD
    • Credentials for the migration database
  • PW_DB_HOST / PW_DB_USER / PW_DB_PASSWORD / PW_DB_NAME
    • The Piwigo database host, database and credentials

Once you are certain above environmental variables are correctly set you are good to go and migrate the Gallery albums from root album level album by album to Piwigo.

The migration start with below command:

python execute_migration.py [root_album]

The program expects to find album information collected for [root_album] so you must ensure you have data collected with collect_gallery_meta_data.py before getting to this phase. Note also, that you can edit the meta information through database tables if you want to change anything there.

Conclusion

Migrating from Gallery Menalto to Piwigo, or any other modern photo gallery software, is now more manageable with this open-source framework. By leveraging the power of Python and SQLAlchemy, this project ensures a seamless transition, preserving all your precious photo memories. I warmly welcome contributions and collaboration to further enhance and extend this tool.