This post originated from an RSS feed registered with Web Buzz by Stuart Langridge. Original Post: Managing my books Feed Title: as days pass by Feed URL: http://feeds.feedburner.com/kryogenix Feed Description: scratched tallies on the prison wall Latest Web Buzz Posts Latest Web Buzz Posts by Stuart Langridge Latest Posts From as days pass by

I read a lot of books, and these days quite a few of them are ebooks. I keep a list of books I’ve read, and I try to review a reasonable proportion of those when I get time. Jono Lange asked how I maintain this list, so I said I’d write it up.

First, books are in Calibre.

I keep the metadata about each book up to date there, and I write my reviews in the “Comments” section.

I then have a little script I run called calibre-dump-library. It looks like this:

1
2
3
4
5
6
7

#!/bin/bash
calibredb export --all --dont-save-cover --dont-write-opf \
    --formats=EPUB --to-dir=/home/aquarius/Documents/Books \
    --template="{authors}/{series:|(| }{series_index:0>2s| |) }{title}"
python ~/bin/calibre-dump-comments.py > ~/Documents/Books/comments.json
scp ~/Documents/Books/comments.json kryogenix.org:public_html/booklist/
python ~/bin/calibre-export-link-recent.py

This does four things:

Export the books from Calibre into an external folder

This is done in quite a careful way; my Documents/Books folder has all my books in it, organised by author¹. It’s synced elsewhere, and also available to my phone so I can fetch any book from my collection. The --template stuff is somewhat laboriously worked out so that The Name of the Wind by Patrick Rothfuss is in a folder named Patrick Rothfuss and is itself named (Kingkiller Chronicles 01) Name of the Wind, The.epub, so all the books in a series stay together and in order.

It uses calibredb, which is the command line tool to access, export, work with, and edit calibre’s database. Calibre does a pretty impressive job of making all its stuff available to external tools and scripts; you do not need to run the app itself to fiddle with the database. Here we’re using calibredb export to export a bunch of books to an external folder.

Export my reviews and some book metadata to a JSON file

I need the JSON file so I can drop it onto my website and build the booklist. So, this relatively trivial Python² script runs calibredb list to dump the metadata contents out into a pipe-separated text list; we then parse that list in Python so that it’s sorted and any missing metadata is added, and then dump it as JSON.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

#!/usr/bin/env python
import subprocess, json
out = subprocess.check_output(["calibredb", "list", "-f",
    "authors,title,comments,timestamp,series,series_index", 
    "--separator", "|", "-w", "10000"])
out = out.split("\n")
data = []
for line in out[1:]:
    if line:
        (bookid, authors, title, comments, 
            timestamp, series, series_index, _) = [x.strip() for x in line.split("|")]
        if comments == "None": comments = None
        if series != "None" and series_index:
            series_index = int(float(series_index))
            anti_series_index = 999 - series_index
        else:
            series = None
            series_index = None
            anti_series_index = None
        if title:
            data.append({"authors": authors, "title": title,
                "comments": comments, "timestamp": timestamp, "bookid": bookid,
                "series": series, "series_index": series_index,
                "ymd": timestamp[:10], "anti_series_index": anti_series_index})
data.sort(key=lambda k:(k["ymd"], k["series"], k["anti_series_index"]), reverse=True)
print json.dumps(data, indent=2)

Copy the JSON file to my website

The actual book list is a reasonably simple PHP script which reads the JSON file and displays the list of books, or displays a list of reviews if that’s what you picked. Not very complicated; if you’re desperate for the source, ask and I’ll send it over, but it’s really not hard.

Set up the recent list

It’s handy, when browsing my Books folder from my phone, to be able to see the books I added most recently to my list. So as a final step we create a folder called 000RECENT (so it is at the top of the list) and copy the ten most recently added books into it.³ This is also not complex stuff; walk through all the list, sort them in last-modified time order, pick the top 10, copy them.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#!/usr/bin/env python
import os, shutil
RECENT = "/home/aquarius/Documents/Books/000RECENT"
def getfiles(path):
    epubs = []
    for root, dirs, files in os.walk(path):
        these = [os.path.join(root, f) for f in files if f.endswith(".epub") and "000RECENT" not in root]
        epubs += [(x, os.stat(x).st_mtime) for x in these]
    return epubs
files = getfiles("/home/aquarius/Documents/Books")
files.sort(cmp=lambda b,a: cmp(a[1], b[1]))
files = [x[0] for x in files[:9]]
shutil.rmtree(RECENT)
os.mkdir(RECENT)
counter = 1
for source in files:
    target = os.path.join(RECENT, str(counter) + ". " + source.split("/")[-1])
    shutil.copyfile(source, target)
    counter += 1
print "Recent files now are:"
print "\n".join([x.split("/")[-1] for x in files])

So, that’s what I do. It means that I have a list of all my books, I have the ability to review books, my reviews are published on my own website (and on Goodreads if you prefer to follow them there; Calibre takes care of syncing my book list with Goodreads), and I can get at any of my books easily from wherever I am on whatever device I want, as long as it can do ssh. This has all evolved in fits and starts over a few years, but… I like books, and spending some time to make my access to them easier is a good thing in, er, in my book.

1. Do not access books inside calibre’s own Calibre Library folder. That is calibre’s private property and should not be fiddled with; in particular, books are often in a sort of “unpacked” state inside there ↩
2. Python 2. Yes. No upgrading in Castle Langridge ↩
3. You might be saying: why not symlink them? And the answer is that AndFTP, which is the SFTP app I use from Android to copy books from my Books folder onto my phone, doesn’t like symlinks and thinks they’re a folder. So, we copy them. Low-tech. ↩