In January 2007 I started playing with the Flickr API - the HTTP-based web service that lets you manipulate Flickr. At that point I was using it to play with machine tags and to see how the most popular Web Service API worked, especially in the area of authentication. This was in the days before OAuth if you can remember that far back.
I started with a test program in C that called libcurl and did some of the signing and parameter marshaling of the
flickr.photos.getInfo call which is where all the juicy metadata about photos is. I started thinking about ways to map photo metadata into RDF for manipulating and querying; there is an existing Perl Flickr RDF mapping but it didn't contain everything. This state of sources was useful; it contained a small library with the one API method plus a command-line utility to call it. Since I was using libCurl to call Flickr, I named it Flickcurl. Also CFlickr was taken! (Flickcurl also uses libxml but flickcurlibxml is just nuts).
Apart from playing with photo metadata I also had some personal reasons
to make something new. I wanted a lighter weight and less formal project
than the way I had been building the Redland RDF
Libraries. More of it compiles, ship it model and
less of the unit tests, test cases and continual make check, worrying
about portability approach. Maybe more fun would be a way to put it.
I'm happiest as a free software / open source software tool-builder and
at this point in 2007 I was spending a lot of time at work doing
non-coding things such as designing specifications and doing technical
leadership and the chance to work on some different code now and then
was appealing to counterpoint the work stuff.
Redland is a set of libraries that have been growing since mid-2000 with more and more features as the semantic web technology stack grows so at any point in time there is no clear end state. For this project I wanted a clear goal to reach so I could be clearly done at some point. This is possible with the Flickr API since there are at any time a finite number of API calls (something like 100) so progress can be measured... although Flickr did add API calls while I was working on it. The result was I made a Flickcurl API coverage page with embedded API changelog (automatically generated from source code comments).
Flickcurl 0.1 was "released" 2007-01-21 although I didn't announce it to anyone at that point. It was more of a tarball than an actual release.
One more thing I wanted to do was to experiment with different ways to tell people about software, compared to the ways I as using with Redland which was mostly email based but also via SourceForge and Freshmeat. So for Flickcurl I tried a bunch of different ways:
- Mentioned 0.2 on Twitter 24 Jan 2007. (3% of API)
- Announced 0.5 (and all later versions) on the yws-flickr Flickr API mailing list in a single paragraph email on 4 Feb 2007. (3% of API)
- Announced 0.6 on the #swig IRC channel 11 Feb 2007. (6% of API)
- Announced 0.7 (and all later versions) via Freshmeat releases of the Flickcurl on freshmeat project 18 Feb 2007. (10% of API)
- Announced 0.11 in a blog post Flickcurl - C API to Flickr on 3 August 2007. (50% of API)
- Announced 0.12 to the Flickr API group on 11 August 2007. (72% of API)
That was kind of fun, and I also followed a similar light weight process with Triplr but that's another story. I think caring less worked out fine; people did use it and submit patches. Right now I still use the Flickr mailing list, API group, and freshmeat project.
As the library headed towards 100% of the API and beyond it did get a bit more formal and I imported what I think are the best practices from the Redland libraries:
- objects in C design
- always refactoring the source code: refactoring is not just for dynamic languages
- source code docu-comments generating an HTML API reference via gtk-doc
- folding in portability fixes
- make it work with optional libraries for extra functionality (Raptor in this case, to allow serialising to other RDF syntaxes)
- built in portable ANSI C
- taking care about memory leaks with valgrind
- comes with a utility program able to exercise the entire API (called
flickcurl) - Debian packages (created by somebody else, yay!)
- manual pages for the command line utilities
The general aim was to get 100% of the Flickr API done by the end of 2007 and I actually reached it for Flickcurl 1.0 on 2008-01-12 which was pretty close.
So right now the library has gone beyond 1.0; the latest release is Flickcurl 1.4 which was released last Tuesday 24th June (see release notes) which primarily added video support but I also updated the photo metadata mapping to RDF by adding a serializer class for abstracting the photo-to-triples process.
The RDF triple mappings is something that has always been there but not part of the core library. It could be optionally used inside Raptor to automatically read Flickr photo URIs as RDF data sources. I doubt it'll ever be presented inside a public web service like Triplr since it would require passing in Flickr API authentication tokens and user credentials.
The RDF triples mapping I've made for the Flickr photo metadata has mixture of vocabularies which are in 3 buckets:
- Existing Vocabularies: well known RDF schemas (class and properties) that have been developed over many years by multiple people and organisations, sometimes with a lot of formality.
- Flickr-specific Vocabularies: vocabularies I made up mostly for Flickr video and places API terms.
- Machine Tag Vocabularies: I made them up using machinetags.org/ns URIs as a root for the namespaces associated with the vocabularies. The terms in the vocabularies come from how people used machine tags on Flickr and are not always defined.
This is a range of what might be called semantic web heavy to light although there is absolutely nothing wrong with mixing things up if you are not worried about inference. This is OK! I should probably put some html/schema documents at the vocabularies and get the redirects and all that # and / business sorted so that the linked data works out properly but what I have now is just a start and I'd be interested to see what people think. There are more details of the vocabularies and terms I'm using in the Flickcurl 1.4 release notes although I should probably add vocabulary information to the documentation too.
That's all for now but I'll expand some more in another post about the Flickr API itself and my experience with it and impressions of it as a both a software developer and HTTP Web Service designer.