Lessons learned in developing Festival Star (with links so non tech types can keep up)
When we started out with Festival Star (originaly GlastoStar) we had plans to support only one festival, with only one map and only one (though massive) schedule. This meant we were expecting to support a limited dataset in the app. Keeping true to the KISS software design principle and also YAGNI, I decided to _not_ implement an Sqlite data storage strategy and instead use simple POCO -> XML serialization.
This meant that there was no need to utilise external software in order to store and access our in-app data. There was no need for any clever (and sometimes buggy) LINQ providers as we're accessing IList
Or so I thought.
We pretty quickly realised that we wanted to provide a more generic app - better for us in terms of maintenance and better for our customers in terms of value.
And so Festival Star was born.
With not long to go before the festival season kicked in we began with the upgrades: new site, new web services, new database on the server, new archetecture for the app plus some. Working on a set of test data the initial proof of concept runs were promising and work progressed fast, I was putting in crazy hours and working evenings and weekends to get the app back to a working state with multi festival support. We got there in the end and launched the app and so people started downloading it! Happy days.
And then the trouble started.
Looking back, there's no way that I would have used serialization as a data storage strategy by design. We "found ourselves" there and now, with the lessons learned I am here to tell you that XML serialization as a storage strategy sucks! It's Ok for small scale apps, with small amounts of data. But we quickly found our data ramping through the roof:
All of this needs to be synchronised with the server and kept up to date. It's no use having all this data if it's all stale and out of date. So I created a set of WCF services to orchestrate a synchronisation strategy based on deltas to minimize data on the wire. Pretty soon I we started getting word that the app was failing for people. Initially it was due to the fact that the app would serialise its data as the app was closing down. You get a set timeframe to do some clearing up as the app shuts down. The serialisation process was taking longer than that set time frame and so the operating system was just booting us out before we'd saved all the data. The fix? Simples, just save data which does not change throughout the users session as it is synchronised, anything likely to change gets saved as the app shuts down.
That worked for a bit - but the strategy sucks and it came back to bite!
This fixed the issues for a short while but I knew the straetegy sucked and so I started thinking about what it would take to fix it. By this point we had an app that was capable of quite a lot (considering it's an iPhone app) and the code associated with communicating and synchronising the way we needed had gotten complex. The plan was to remove all of the old synchronisation code, remove all the old serialisation code, make sure all data access was centralised and factor in Sqlite, using Faks Sqlite-net code: http://code.google.com/p/sqlite-net/ which works like a charm btw :)
The good, the bad...
This has meant that the app no longer needs to do any crazy deserialisation on start up, no need to store large amounts of objects in memory for the entire lifetime of the users session, no saving on shutdown. All good! The only downside I see at the moment - and these are minimal are:
But otherwise the app seems to be faster in general, faster at start up, lots more stable and clears the way to handle LOADS of data with ease. All of this now means I am free to concentrate on adding new and great features such as:
V3 is currently in review with Apple, this is the foundation, a very solid foundation for a lot of great and new stuff. A lot of data and a lot of ways for you guys to plan, interact, and also reminisce. I expect V3 will be out within the week (provided I haven't borked anything!) - I hope you guys like the way it's progressing. As always we're really happy to hear from you guys, if you have any queries, questions, suggestions or fancy a banter of the idle chit-chat variety then we're your guys.
In the mean time Jon & I have started a company to make the whole Festival Star thing official, we called it project:attack (or prj:atk for short) we're not entirely sure why. Sounds good though. Jon's been biiiiizzzzeeeee racking his head for a snazzy look: http://yfrog.com/ju2cavj
He's hanging out for some feedback on that if anyone can provide some :)
w://
When we started out with Festival Star (originaly GlastoStar) we had plans to support only one festival, with only one map and only one (though massive) schedule. This meant we were expecting to support a limited dataset in the app. Keeping true to the KISS software design principle and also YAGNI, I decided to _not_ implement an Sqlite data storage strategy and instead use simple POCO -> XML serialization.
This meant that there was no need to utilise external software in order to store and access our in-app data. There was no need for any clever (and sometimes buggy) LINQ providers as we're accessing IList
Or so I thought.
We pretty quickly realised that we wanted to provide a more generic app - better for us in terms of maintenance and better for our customers in terms of value.
And so Festival Star was born.
With not long to go before the festival season kicked in we began with the upgrades: new site, new web services, new database on the server, new archetecture for the app plus some. Working on a set of test data the initial proof of concept runs were promising and work progressed fast, I was putting in crazy hours and working evenings and weekends to get the app back to a working state with multi festival support. We got there in the end and launched the app and so people started downloading it! Happy days.
And then the trouble started.
Looking back, there's no way that I would have used serialization as a data storage strategy by design. We "found ourselves" there and now, with the lessons learned I am here to tell you that XML serialization as a storage strategy sucks! It's Ok for small scale apps, with small amounts of data. But we quickly found our data ramping through the roof:
- >1500 mapped locations most with textual descriptions and meta data
- >2000 artists profiles each with a summary, biog and loads of meta data
- ~30,000 song previews each with a list of links to photos, reviews and even more meta data
- internal links to wire acts to sets to stages to events plus a whole load more
All of this needs to be synchronised with the server and kept up to date. It's no use having all this data if it's all stale and out of date. So I created a set of WCF services to orchestrate a synchronisation strategy based on deltas to minimize data on the wire. Pretty soon I we started getting word that the app was failing for people. Initially it was due to the fact that the app would serialise its data as the app was closing down. You get a set timeframe to do some clearing up as the app shuts down. The serialisation process was taking longer than that set time frame and so the operating system was just booting us out before we'd saved all the data. The fix? Simples, just save data which does not change throughout the users session as it is synchronised, anything likely to change gets saved as the app shuts down.
That worked for a bit - but the strategy sucks and it came back to bite!
This fixed the issues for a short while but I knew the straetegy sucked and so I started thinking about what it would take to fix it. By this point we had an app that was capable of quite a lot (considering it's an iPhone app) and the code associated with communicating and synchronising the way we needed had gotten complex. The plan was to remove all of the old synchronisation code, remove all the old serialisation code, make sure all data access was centralised and factor in Sqlite, using Faks Sqlite-net code: http://code.google.com/p/sqlite-net/ which works like a charm btw :)
The good, the bad...
This has meant that the app no longer needs to do any crazy deserialisation on start up, no need to store large amounts of objects in memory for the entire lifetime of the users session, no saving on shutdown. All good! The only downside I see at the moment - and these are minimal are:
- Added complexity, dependancies on external code, DllImports, a basic Linq provider.
- Slightly slower sync due to executing loads of inserts/updates/deletes rather than just accessing a List
But otherwise the app seems to be faster in general, faster at start up, lots more stable and clears the way to handle LOADS of data with ease. All of this now means I am free to concentrate on adding new and great features such as:
- Highlighted map regions (including the 3d maps I spoke about before)
- Improved UI - this is apparent in V3 but there's loads more on the way
- A free, cut down version of the app for BlackBerry, Android, Nokia users (and iPhone owners who don't want/need the full blown version) written in HTML5 and open sourced! This should be ready for initial viewing in less than 2 weeks.
- Online - a website for registered users to manage their app related things such as upload photos, browse events, add/remove events from their app, connect with mates and do stuff.
V3 is currently in review with Apple, this is the foundation, a very solid foundation for a lot of great and new stuff. A lot of data and a lot of ways for you guys to plan, interact, and also reminisce. I expect V3 will be out within the week (provided I haven't borked anything!) - I hope you guys like the way it's progressing. As always we're really happy to hear from you guys, if you have any queries, questions, suggestions or fancy a banter of the idle chit-chat variety then we're your guys.
In the mean time Jon & I have started a company to make the whole Festival Star thing official, we called it project:attack (or prj:atk for short) we're not entirely sure why. Sounds good though. Jon's been biiiiizzzzeeeee racking his head for a snazzy look: http://yfrog.com/ju2cavj
He's hanging out for some feedback on that if anyone can provide some :)
w://
Comments
Post a Comment