Subject: "PhotoStory - Devlog #3"
So, it has been around 3 weeks since the last major update to PhotoStory. I've been a bit more busy with life and exams in general so I've barely had time to work on the app. I've been working on some other projects so PhotoStory has just been stuck in a bit of a limbo but I've mostly been tying up some loose ends.
I feel satisfied enough with the app for me to put its development on hold. I'll hopefully make a full project release post here once I'm free but for now, a devlog will have to suffice. I fixed the last few issues I had with the app and I think I'm happy with where it is now. I'll talk a bit more about this near the end.
Database Issues Fixed
I finally fixed the two outstanding database issues since the first devlog. It was an N+1 query and a TOCTOU bug. I fixed the N+1 query by replacing a for loop with a full outer join that allows me to get all the data I need within a single query.
Secondly, the TOCTOU bug is a significant anti-pattern. It wouldn't matter too much in SQLite since it only has a single concurrent writer but still, the function would first execute a SELECT statement and then use the data from that to update a value. This is not atomic.
The easy way to mitigate this in a traditional RDBMS would be to use row-level locks and transactions but in SQLite, there are no row-locks since it's a file. A simpler alternative would be to use a non-standard SQL query known as an upsert. This was a pretty simple and idiomatic solution to a problem that had my head scratching. I'll leave it for anyone having the same issue in the future.
INSERT INTO ... ON CONFLICT ( ... ) DO NOTHING;
Orphaned Images Removal
So this was quite possibly the most annoying problem that I had to solve throughout most of PhotoStory. I didn't really account for this during the main program development so I mostly had to come up with a pretty terrible solution to it that is only saved because of SQLite's lax query system.
When a user was deleted, all their database entries would delete just fine. The images in the corresponding tables were easy to remove because they were configured to cascade delete. However, the storage layer doesn't know that the images were removed so the actual files would be orphaned on disk, eating up storage space.
The solution was to query the database before requesting all the images owned by the same user that is only unique to them. The SQL query for that is a complete mess and possibly number one on my list for a future refactor. It has code smells on so many levels but for now it works and it does the job so I'm not complaining.
File Validation Routine
This might just be the biggest blunder in the history of PhotoStory. The most naive assumption I've probably made. I forgot to validate the actual image files that were uploaded by the user. This was the deadliest combination of bad security and programming practices and could potentially cripple the security of any PhotoStory instance.
PhotoStory originally handled file format issues by running the byte data through the image processing library itself (in our case, libvips) and checks if it raises an exception. It is essentially like running a package through a metal detector that is off and praying it doesn't explode on the other side. This poses so many problems I don't even know where to begin.
The first thought that comes to mind would be to just check the file extensions. However, this can be spoofed easily. Files can just have their names alone changed while the payload will still be the same. Therefore, I decided to set-up a small function that checks if a file matches the few well-known headers of popular file extensions. This way, a simple exception can be raised and stop further execution before it can even be processed.
Setup and Configuration
This is the last leg of work left to fully release PhotoStory. I've been testing the program out on a small Fedora VM on my laptop. I've been having fun setting it up and I'm trying to get an easy-to-install and configure shell script running as well as a thorough install guide that covers most of the various tweaks that can be made that can affect how PhotoStory works during run-time.
I plan on using Nginx as a reverse proxy so that it will be easier to set-up certificates for encryption of data during transit, completing the final piece of the E2EE dream. Before I get too far ahead though, there are some SysAdmin work that needs to be done. I need to create a systemd service file that can automatically boot up and activate the environment for the app.
What's Next?
After everything is said and done, this might be the last few updates I would push to PhotoStory. The long-term project roadmap will still be there. Others can always fork the project and add their own features to the mix and I'm more than happy to check out any incoming pull requests.
With pretty critical exams looming quite nearby, I find it very difficult to divide my attention between this project and studies. I also just don't find it as interesting anymore sadly. The app pretty much has everything I would ever use personally baked in and I don't really feel like I need anything more.
After spending about 4-5 months working on this project, I believe the true beauty of PhotoStory doesn't lie in the UI or code itself. A lot of it looked ugly, sure, but I always found the modularity of the code to be the best part about it. I've never once had to move heaven and Earth to introduce a new feature into PhotoStory so a lot of the abstractions ended up working to my favour.
I'll push these last few commits out and release the full 'Making of' and wrap up development by the end of this week hopefully. That's really about it. I can't wait to see what captures my interest next.