Michael Medin

Random thoughts on development, SOA and monitoring…

Thumbnail 1000:s of photos on a Synology NAS (in hours not months)

synology_ds212I moved all my photos over to my Synology NAS (which I really enjoy) just to discover that it would take months to index all photos. So I started to look for options and while most people managed to increase the speed a bit by reducing the quality of the thumbnails I figured why not utilize some of the CPU power I have at home?

So I ended up using my Linux box to do the thumbnailing for me instead which reduced the time to thumbnail 30.000 photos down to hours.

The process

The process is simple and straight forward and requires NFS as well as ImageMagic on the Linux box I also disabled the indexer on the Synology DS (using the pause feature) but I don’t think this is strictly required.

The first step is to mount the photo share:

mkdir /mnt/photo
sudo mount DS_IP:/volume1/photo /mnt/photo/

Second step is to create a script to generate thumbnails.

This will for all *.jog create the required thumbnails for you if you have other files you might need to tweak this.

#!/bin/bash
pushd "$1"
shopt -s nocaseglob
if [ ! -d @eaDir ] ; then mkdir @eaDir ; fi
for f in *.jpg ; do
if [ "$f" == "*.jpg" ] ; then break ; fi
echo "$1 - $f..."
if [ ! -d @eaDir/$f ] ; then mkdir @eaDir/$f ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ] ; then convert $f -resize 1280x1280\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ] ; then convert @eaDir/$f/SYNOPHOTO:THUMB_XL.jpg -resize 800x800\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_L.jpg ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ] ; then convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 320x320\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_M.jpg ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ] ; then convert @eaDir/$f/SYNOPHOTO:THUMB_M.jpg -resize 120x120\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_S.jpg ; fi
if [ ! -f @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ] ; then convert @eaDir/$f/SYNOPHOTO:THUMB_L.jpg -resize 640x640\> -quality 90 -unsharp 0.5x0.5+1.25+0.0 @eaDir/$f/SYNOPHOTO:THUMB_B.jpg ; fi
done
popd

Last step is to run the script in the folder you want to thumbnail (or the root if you want to thumbnail it all):

Replace ~/tn.sh with the location of the script you created before.

find . -type d -name @eaDir -prune -o ! -name @eaDir -type d -exec ~/tn.sh {} \;

A big warning

Now this may void you warranty break your NAS and generally end the world as you know it. I tried it on my NAS and it works but there are no guarantees. But I hope that the people at Synology will listen to this and create a sanctioned way to remotely create thumbnails as it is much better (especially as upgrading photo station requires re-indexing) than the rather cheep re-upload photos using a tool solution they have today.

Some theory

So what does this do? And how do the thumbnails work?

Well that is pretty straight forward.

If you create a folder called folder 1 with two photos “photo 1.jpg” and “photo 2.jpg” the indexer will create the following folder structure:

image

In other words a folder called @eaDir with a subfolder for each photo. The subfolder representing a photo in turn contain the various thumbnails.

The thumbnail process is configured in the following file /usr/syno/etc/thumb.conf or /usr/syno/etc/thumb_high.conf depending on which setting you have. I use the normal one (low resolution thumbnails) which is defined like so:

size=120, quality=90, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_S.jpg
size=320, quality=90, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_M.jpg
size=640, quality=80, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_B.jpg
size=800, quality=90, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_L.jpg
size=1280,quality=90, unsharp=0.5x0.5+1.25+0.0, filename=SYNOPHOTO:THUMB_XL.jpg

Which is essentially just command line arguments for ImageMagicks convert command (which is what I call in my script above).

Samba anyone?

A quick note is that the reason this doesn’t work via samba (smb) I think is the colon character. A colon in a smb filename (at least in windows) is a stream A pretty cool but very neglected feature of NTFS. A stream is a file in a file meaning you can put more than one stream of data in each file. This was a cool way to “joke” with co-workers back in the Windows NT4 days as Explorer did not see streams. If you created a file somewhere with 0 bytes and then put megabytes of data in a stream the file would still be listed as 0 bytes yet take up all the space on the hard disk Ler

Monitoring Conferences 2012!

Well, since I got an email from Würth Phoenix asking me to plug their conference Which has been renamed again this year calling itself Open Source System Management Conference 2012. Ironically enough they forgot to include the link making it a bit difficult to give some information but a quick google reviled this page http://www.wuerth-phoenix.com/en/company/event/nagios-event/. Since I figured it would be a dreadfully short blog post with just a single conference I figured I’d mention the other once I know about as well. Continue reading

Real time event-log monitoring with NSClient++

Monitoring the event log can quickly become straining for both the computer as well as the administrator as the event log grows and grows. To make this simpler for both the administrator and the computer NSClient++ 0.4.0 introduced real-time event log monitoring. This means we no longer scan the event log instead we simply scan events as they come in. The benefit, in addition to lowering the resources required, is that we can also get notified instantly when an error occurs instead of every 5 minutes or however often we check the log. Another addition is a simple client o generate event log message to help administrators debug event log filters. This is a quick introduction to event log monitoring and real-time event log monitoring showing how to set up real-time event log monitoring both for active and passive use via NSCA and NRPE.

Continue reading

Using WMI with NSClient++ 0.4.0 Part 1: Command line tools

This is a series detailing how you can leverage WMI to monitor you Computers from a monitoring tool such as Nagios or Icinga. Since I decided to clean up the command line syntax of the WMI plugin for NSClient++ for the up-coming 0.4.0 version a few days ago I will start by showing how you can use what has become an almost full featured WMI client.

Continue reading

Vissa saker ändras aldrig…

DSC03137Idag på vägen hem så passerade jag central stationen som de håller på att bygga om. Och jag är väldigt entusiastisk över den “nya” central stationen som är mera “europeisk”. Tidigare var det mest 2-3 kiosker och ett stort väntrum men nu blir det en massa små mack barer, restauranger och affärer. Något som är väldigt vanligt i t.ex. Tyskland där tåg stationerna blir lite som en mini galleria där man kan plocka med sig allt det där som man inte har tid med annars.

Så när jag passerade passade jag på att flanera runt lite och titta på de (fortfarande bara en handfull) små “mini restauranger” som öppnat på nedre plan.

Skrot-nisse

Och döm om min förvåning när jag gick förbi Pocket shop (som ju faktiskt fanns redan innan ombyggnaden). Men där på bordet står en bok som jag läst mycket i mina dar.

IMAG0181

Det var väldigt nära att jag köpte den, för 79 kronor hade jag kunnat bli ägare till vad jag förutsätter är “The 2011 definitive edition” av min gamla trotjänare från 1981. Dock bortsätt från lite mindre justeringar så verkar min andra utgåva vara snarlik den moderna nya versionen och gissningsvis då även originalet från 1976.

Den största förändringen jag noterade var att den numera inte är lika hög min version är “nästan” A4 storlek den nya verkade vara lite lägre.

Vissa saker ändras aldrig…och tur är väldet!

DSC03139

Embedding Python inside a multithreaded C++ program

This is a tutorial for how to embed python correctly inside a multi threaded program. Python is a very neat language which is very easy to embed inside C++ thanks to the boost::python library. But there are some crucial parts which is missing from boost:python in regards to how to manage GIL and thread state which I introduce here. Since this is my first C++ tutorial I will include boost:thread as well as a quick hello world application as well.

Continue reading

Efficient Work: Context switching via the Keyboard

One of the most common perceptions of windows is that it is a PUI (PARC User interface) oriented operation system. In other words the mouse is required to do everything. This is in many places true but fortunately Microsoft has recently changed this and incorporated the keyboard more and more into what was previously a mouse only domain. With Vista and Windows 7 a whole range of new keyboard shortcuts emerged and slapping on some third-party tools on top of that and we get a full-fledged keyboard experience.

This is a short introduction to how I use Modern Windows features and various third-party tools to optimize my workflow using the Keyboards in Windows..

Continue reading

The end of an era: Apoteket!

apotekten_ny-logga_D_Rylander.mediumIs it possible to summarizing two years of Apoteket with a handful of music tracks?

Music has always been an important influence to me. I tend to emote strongly by listening to music. Unfortunately emotions are not, for me, really in the public domain so I will probably never share that here.

So what can I do instead?

Well, taking the easy (read ironic) door I decided to list some music which in various ways represents much of what happened at Apoteket. The idea was born really from the first segment as it is a nice mix of “my music” and what I expect would be considered “their music” (With them being the team from Oracle SSI India).

Offshore, near-shore, no-shore!

So the most obvious thing I noticed at Apoteket was that it was the first true offshore as well as near shore project for me. We not only had Indians and Polish people in my team we also had a lot of them in other teams which we worked with as well.

For me this was a mixed bag I have met some really really good people with very impressive skills as well as some really terrible ones. But on the whole I think offshoring really is difficult if you are lucky (or perhaps skilled) you will have great success and if you are not: catastrophe.

So which music shall represent India and Offshore then? Well obviously it has to be end.user and Switch a break-core (ish) version of a song from the Bollywood movie Arzoo.

end.user – Switch

We also had some near shoring but I cant for the life of me find something Polish so I have to skip that one. But speaking of countries apart from India and Poland we also had quite a few other ones represented. A bunch of people from the USA, Some from the Netherlands, yet more from Malaysia and lets not forget our neighbors in Finaland, Norway and Denmark. But perhaps more importantly one Italian guy with whom I worked quite a lot and given that I essentially only know one group from Italy (ish) I guess I am stuck with Klangstabil and one of the few tracks which are (I think) in Italian I will go with perdere per vincere (and no I have no idea at all what the song is about).

Klangstabil–Perdere per vincere

Alien Monster (AM) Team

Sometimes working with various parts of the various teams and projects at Apoteket I felt pretty much like screaming like the protagonist of this next song: “What do you want from me?” This song will also step up the pace a bit with some solid Norwegian EBM. As for which project and which organization I leave that as an exercise to the reader.

So here we go Pantzer AG (WARNING MySpace) Filth God.

Pantzer AG – Filth God

Another word I guess I would use for this whilst keeping the pace is I guess S.K.E.T. since they have an amazing song called Center of Evil but alas that is not to be found on neither Youtube nor Spotify. So instead of the planned social commentary on American foreign policy in the post-911 era we wend up with some political commentary in regards to the American foreign policy in the gulf war called Is the Baby Normal. It is still rock solid power noise though so a feast for the ears!

S.K.E.T. – Is the Baby Normal

Longest project like…ever?

This could in fact be the longest project ever for me. I have not kept count and possibly SvK and KAII was longer but with close to two year it is by far one of the longest projects for me. And given the some-times rather hectic schedule I felt that the best representation would be Faithless and Insomnia as the Monster Mix clock in at 8:42 but since I got bored with static images I figured I’d go for the music video version instead which is considerably shorter (and not nearly as good).

Faithless – Insomnia

Then of course we have the fact that we some times cheat when we do AIA and we often make things much much simpler so in that regard we are Rebels right? WHich brings us to the next video. This one is still steady in the electronica genre but perhaps a bit more danceable (though I would probably dance to all of them under the right circumstance). So representing our aprach to AIA: Alice in Videoland (WARNING: Myspace link) We are Rebels. I will not comment on the video suffice to say this is not how it looks when I am out riding the trails… it is how I wish it would look Ler

Alice in Videoland – We are Rebels (ish)

Software Piracy!

One of the common reoccurring themes of conversation was software piracy (as there seems to be quite a few shady people working for certain companies). Myself I have a rather strict policy in software piracy (especially since I spent a fortune on both movies, music, books and games. I sort of have to defend the expense in some way right?

And the natural song to represent this is of course Weird Al Yankovic and Don’t Download this song. If nothing else both the song and video are epic… EPIC!

Weird Al Yankovic – Don’t Download This Song

So the other aspect of the Apoteket project is I guess my role at Oracle and why I quit. Once I started to ponder this I first went with Grendel and Chemical + Circuitry mainly due to the intro which is from A Scanner Darkly. But then a though struck me, almost an old classic, Banished from Hocico. Or perhaps even Wounds. But for me Banished has always been one of the best song from Hocico. It really has the pure darkness, rage and desperation which only Hocico can manifest.

Hocico – Banished

Departure Plan!

This section is dedicated to Oracles amazing Departure Plan (or lack there of) after I handed in my resignation (I actually notified them more then a month prior to handing in my resignation). Their general policy was pretty much that of a ostrich: “Lets not tell Apoteket and hope they will not notice it”. This was never really a big problem for me (and I wont inflate my self worth my saying that it was a big deal for Apoteket either) but I would have felt happier with a slightly better handover to “Henrik Brokvist” whom they found in the last few days. But I feel pretty comfortable leaving them in his very capable hands. So which song do we get? Well felt it was time to switch genre a bit and thus I will go with Soilwork and Departure Plan.

Soilwork – Departure Plan

And so now we are need something to finish it all of right? Something epic something of a statement… maybe even leaving on a positive note?

I would like to finish then by saying this has been one of the most fun projects – period! And I can but hope that I will again work with such crafty and ingenious people with such a good team spirit and such grand goals, such … perfection lacking a better description. So that’s pretty much leaves us with some nice trance from one of the best DJs: DJ Tiësto (WARNING Annoying background music). I was aiming for Love Comes again but I felt that was a bit too much so instead we end up with Forver Today.

Tiësto – Forver Today

Locust!

Now I was really angling for having Machine Head and Locust in here but alas I failed to find a reason to motivate it… so you get it as a bonus song… amazing tune all in all… I have listened to it a lot lately at Apoteket…

Machine Head – Locust