I've started to work for Chtd in October 2011 and worked till February 2015. It was a good time and I want to say thank you to everyone who worked with.
At the moment of writing this post the company is in a difficult sutiation and it had to shrink (four of the six developers were forced to leave the company).
This is going to be a long post and probably I'll keep it as is.
I'm going to sum up my experience working as a developer for Chtd talking about the some interesting points.
The company has developed its own implementation of graph-oriented database (think of EAV) built on top of PostgreSQL using Django called NetDB (authors of the core are Vjacheslav Fyodorov and Konstantin Lopuhin).
The database provides a way of storing historical data meaning we have a full history of changes in it (no physical delete operations). The library for this is open sourced and is called documents.
A typical project on the basis of NetDB consists of several libraries providing needed functionality.
Working with NetDB means we can build reports almost unlimited in size and have automatic calculations according to calc rules specified by analysts.
We have a specialized report builder and some other useful tools.
This is rather important point to begin with it. Going back to 2011 the company used selfhosted subversion as a version control system and actually there was no established workflow for development. The team was small and for some time the situation was acceptable. But the more we wrote the code more difficult it becomes to control the situation.
I don't remember exact dates but we changed it.
As a first step it was switch to mercurial repositories hosted by BitBucket. We've splitted monolitic svn-repository to several hg repos.
As a second step we've adopted gitflow as a main workflow to work with features, fixes and releases (it was my investigation about possible workflows which led us to this excellent solution). Thank you, Vincent Driessen, for sharing your experience!
As a third step we've started to use pull requests to review the code to be merged to default branches.
From the very beginning we were using Redmine to track issues and manage projects and it's just great.
Using svn we had a post-commit hook configured to send email with diff to developers mailing list.
After switching to BitBucket we wanted to keep these emails with diff so our internal project called bbhook was born and released by me.
It is backed by PostgreSQL and Django and has two modes of operation - as a service to be notified by BitBucket about push and as a django-command configured to run by crontab every 5 minutes.
Bbhook project can handle multiple hg repositories and under the hood it does the following:
Bbhook works pretty well and we had just a few issues with it.
Going back to 2011 and 2012 we were using makefile to prepare configuration files to deploy a web service (project built on NetDB).
It was a working solution but I wasn't satisfied with it. We still had a lot of things to do by hand.
It should be noted that we had one development server running a lot of web services to be tested and reviewed by our testers and analysts. The time was critical here - we were spending too much time deploying a service.
So, I've remembered of Fabric and some time later we were using our specific fabfile.
Its initial version had several most important commands for us:
Later the fabfile was extended to build a project distribution, to attach newly deployed service to apache instance, to clean up service etc.
The core command was deploy and combined with others we could deploy a service by issuing just one command in bash.
The key win here was a way we specify service settings (server name, db parameters etc.) - in a command line instead of manually editing configuration file.
Update. Sometimes I'm still fighting with gulp spending too much time
configuring it. The article by Keith Cirkel
helped a lot to start using npm as a build tool.
This is a very long story on its own that is why I'll not go into details.
We had a Django-based project with a lot of libraries. Some of them had static files - javascript, css, handlebars templates for client-side, less and sass.
We passed this way:
I was managing switch to grunt and to gulp and I am the only author of bundler.
Reasons for this journey were speed and stability. We need speed to compile static files and we need stability deploying new services on different servers running different Linux flavours.
Using grunt we were suffering from speed and using gulp we were suffering from stability (stability of gulp plugins to be installed on different servers).
I am sure these are not the issues any more for grunt and gulp (and I am successfully using gulp in my pet projects) but at some moment I've decided to drop this dependency.
Bundler is a one javascript file (nodejs module) which is rather complex inside but works pretty well.
There is a chance I'll write a post focusing on bundler.
That's all for automation section.
This can be a huge section but I'll focus only on some examples (without any specific order).
It was my first library written for Chtd and we called it Object editor (oditor).
Due to NetDB specific internals there was a need for an administration tool (Django embedded admin site didn't suit us).
I think it was great for me to create oditor because I had a way to better understand NetDB internals.
Oditor had two implementations.
The first one was made using simple a-la REST backend and Dojo-based frontend.
To tell you the truth - I don't like Dojo. It is great and awesome but it's not for me. I am sure there is something with me, not with Dojo. I'll keep it as is.
The second implementation was made using Backbone and Bootstrap (details on them follow) and it was a big step forward to better python and javascript code and to usability.
Going back to 2011/2012 my javascript code was ugly. I was the beginner and I was trying to have code organized but I'm not sure I've managed it.
As it usually happens there is a moment at which you understand - you have to do something. And by chance there was a project in which I was the only person responsible for frontend.
I took the chance and created the basis for some of our client-side libraries.
It was based on modification of djangbone for backend (an excellent library I must say).
And it is based on backbone, marionette and bootstrap - for frontend.
I think it was one of the most important steps during the history of Chtd because QuBeQu (NetDB SaaS solution) was built on top of this library.
This is a story about refactoring python code.
We had rather complicated permissions system developed back to 2012. Really complicated. The author of the system has left the company a year later.
Supporting one of our projects we've got a complain from the end user about speed. It took too much time loading one specific page from our web service (more than a minute - awful).
Profiling showed that it's user permissions system to blame.
Quick tests showed a possible way of fixing it and at first I was thinking it will took from seven to ten days. Actually it took almost two months if I'm not mistaken.
I should say there was my bad refactoring tests and I lost a week or so but still - there is a huge codebase and my estimates were naive.
Refactoring the code was a great lesson.
For the first time I've tried to use Python descriptors and I really liked it.
Python descriptors are a great way of structuring your codebase.
The result of the refactoring was awesome - page load time decreased to 1 second (from more than a minute!).
We had a couple of projects with the need for digital signature support.
In general this is about integration with CryptoPro software (Russia-only solution).
One of the tasks here was to provide full functionality for users having certificates signed by accredited certified centers.
There is a specialized portal distributing the XML list of accredited centers but that is not enough. We need to load CRLs too and there is no single point to get all CRLs from.
It took some time to develop an optimal solution (this is about fetching data from Internet).
It was one more battle for speed to load certificates and CRLs within a reasonable time.
Mostly client-side library acting as an OLAP builder built on top of our backbone-based library.
Most interesting part here was drag'n'drop to provide better user experience.
I've created the first version of this library and later it was handed over to another developer. It is one of the most valuable libraries at QuBeQu.
One more client-side library created by me.
I'm expecting it to be published by Chtd under the MIT license
(we had such an agreement). It is published
by Chtd under the MIT license.
The library is used to create custom layouts of reports and charts - for dashboards.
Drag'n'drop here too.
There are more libraries and tasks I've done working for Chtd, a lot more.
This is one more subject I'd like to talk about.
We had an import/export functionality providing support for Excel files (among other formats).
There were several python-only solutions but we were not satisfied by them. We needed a way to work with any xls/xlsx file keeping its original formatting untouched.
One day I've stumbled upon POI.
At first I was not sure we would be able to use it. I thought we need Java developer.
But some time later I've remembered about the Jython project. It was an experiment and it was a success.
I've splitted the functionality into several parts.
The first one was Jython-powered http service to work with Excel documents.
The second one was python library to communicate with this service and send and receive data.
And a bit later there was the third one - client-side import/export configuration library providing a way to set parameters (we can read/write data from/to different sheets and cells within Excel documents).
POI-based solution worked great and we had pretty good performance for import/export operations.
It was a good time. There were moments I was not satisfied with myself or someone from colleagues but we are all human beings.
We can and we must make mistakes - to be able to create something better.
I have no opportunity but would love to work using Lean Software Development principles.
While developing sometimes it's better to think about optimisation ahead. You must find a balance inside your mind between the desire to experiment and the need to provide a solution in time.
It's better to stay focused.
It's important to form the team. Very important for the project.