Words and Thoughts

Initial commit

2026-07-26T12:40:00+00:00

Initial commit ¶

Note

Update 1. This blog is powered by self-made library wt.

Note

Update 2. wt library has been moved to archive, no plans to develop it further. ~~I've switched to zola to manage the blog~~.

Note

Update 3. wt library was unarchived, rebuilt from scratch and released as 3.0.0 version.

This is just a very first record in the blog which will be powered by ~~Pelican~~.

Module level abstraction in python

2015-01-30T17:40:00+00:00

Module-level abstraction in Python ¶

Sample code first.

# -*- coding: utf-8 -*-
#
# file: jukoro/pickle.py

import cPickle


load = cPickle.load
loads = cPickle.loads


def dump(obj, f):
    return cPickle.dump(obj, f, protocol=cPickle.HIGHEST_PROTOCOL)


def dumps(obj):
    return cPickle.dumps(obj, protocol=cPickle.HIGHEST_PROTOCOL)

No magic inside - in our own project we've created pickle.py module, the one to be used in place of cPickle from standart library.

As a result we've got an abstraction, which helps us to keep code DRY and improves code manageability.

I like this approach.

Grief

2015-03-01T06:15:00+00:00

Grief ¶

Boris Nemtsov.

Terrible pain. Emptiness inside. Impossible to comprehend.

Chtd

2015-03-28T04:25:00+00:00

Chtd ¶

Preface ¶

I've started to work for Chtd in October 2011 and worked till February 2015. It was a good time and I want to say thank you to everyone who worked with.

At the moment of writing this post the company is in a difficult sutiation and it had to shrink (four of the six developers were forced to leave the company).

This is going to be a long post and probably I'll keep it as is.

Context ¶

I'm going to sum up my experience working as a developer for Chtd talking about the some interesting points.

The company has developed its own implementation of graph-oriented database (think of EAV) built on top of PostgreSQL using Django called NetDB (authors of the core are Vjacheslav Fyodorov and Konstantin Lopuhin).

The database provides a way of storing historical data meaning we have a full history of changes in it (no physical delete operations). The library for this is open sourced and is called documents.

A typical project on the basis of NetDB consists of several libraries providing needed functionality.

Working with NetDB means we can build reports almost unlimited in size and have automatic calculations according to calc rules specified by analysts.

We have a specialized report builder and some other useful tools.

Workflow ¶

This is rather important point to begin with it. Going back to 2011 the company used selfhosted subversion as a version control system and actually there was no established workflow for development. The team was small and for some time the situation was acceptable. But the more we wrote the code more difficult it becomes to control the situation.

I don't remember exact dates but we changed it.

As a first step it was switch to mercurial repositories hosted by BitBucket. We've splitted monolitic svn-repository to several hg repos.

As a second step we've adopted gitflow as a main workflow to work with features, fixes and releases (it was my investigation about possible workflows which led us to this excellent solution). Thank you, Vincent Driessen, for sharing your experience!

As a third step we've started to use pull requests to review the code to be merged to default branches.

From the very beginning we were using Redmine to track issues and manage projects and it's just great.

Automation ¶

BitBucket and Redmine ¶

Using svn we had a post-commit hook configured to send email with diff to developers mailing list.

After switching to BitBucket we wanted to keep these emails with diff so our internal project called bbhook was born and released by me.

It is backed by PostgreSQL and Django and has two modes of operation - as a service to be notified by BitBucket about push and as a django-command configured to run by crontab every 5 minutes.

Bbhook project can handle multiple hg repositories and under the hood it does the following:

update repositories mirrors (repositories must be cloned on the same server running bbhook)
process each new commit arrived from BitBucket:
- get diff
- prepare email text using pygments
- send email
- parse commit message to get Redmine issue number
- post a message to corresponding Redmine issue

Bbhook works pretty well and we had just a few issues with it.

Fabric ¶

Going back to 2011 and 2012 we were using makefile to prepare configuration files to deploy a web service (project built on NetDB).

It was a working solution but I wasn't satisfied with it. We still had a lot of things to do by hand.

It should be noted that we had one development server running a lot of web services to be tested and reviewed by our testers and analysts. The time was critical here - we were spending too much time deploying a service.

So, I've remembered of Fabric and some time later we were using our specific fabfile.

Its initial version had several most important commands for us:

deploy a new service from a specified hg project branch
update previously deployed service
create postgresql database according to settings
load postgresql dump to database or
load database with initial data
and some others

Later the fabfile was extended to build a project distribution, to attach newly deployed service to apache instance, to clean up service etc.

The core command was deploy and combined with others we could deploy a service by issuing just one command in bash.

The key win here was a way we specify service settings (server name, db parameters etc.) - in a command line instead of manually editing configuration file.

Static files ¶

Update. Sometimes I'm still fighting with gulp spending too much time configuring it. The article by Keith Cirkel helped a lot to start using npm as a build tool.

This is a very long story on its own that is why I'll not go into details.

We had a Django-based project with a lot of libraries. Some of them had static files - javascript, css, handlebars templates for client-side, less and sass.

We passed this way:

webassets library (python)
grunt (nodejs)
gulp (nodejs)
selfmade bundler (nodejs)

I was managing switch to grunt and to gulp and I am the only author of bundler.

Reasons for this journey were speed and stability. We need speed to compile static files and we need stability deploying new services on different servers running different Linux flavours.

Using grunt we were suffering from speed and using gulp we were suffering from stability (stability of gulp plugins to be installed on different servers).

I am sure these are not the issues any more for grunt and gulp (and I am successfully using gulp in my pet projects) but at some moment I've decided to drop this dependency.

Bundler is a one javascript file (nodejs module) which is rather complex inside but works pretty well.

There is a chance I'll write a post focusing on bundler.

That's all for automation section.

Python, Javascript ¶

This can be a huge section but I'll focus only on some examples (without any specific order).

Low-level NetDB administration ¶

It was my first library written for Chtd and we called it Object editor (oditor).

Due to NetDB specific internals there was a need for an administration tool (Django embedded admin site didn't suit us).

I think it was great for me to create oditor because I had a way to better understand NetDB internals.

Oditor had two implementations.

The first one was made using simple a-la REST backend and Dojo-based frontend.

To tell you the truth - I don't like Dojo. It is great and awesome but it's not for me. I am sure there is something with me, not with Dojo. I'll keep it as is.

The second implementation was made using Backbone and Bootstrap (details on them follow) and it was a big step forward to better python and javascript code and to usability.

Backbone and Bootstrap ¶

Going back to 2011/2012 my javascript code was ugly. I was the beginner and I was trying to have code organized but I'm not sure I've managed it.

As it usually happens there is a moment at which you understand - you have to do something. And by chance there was a project in which I was the only person responsible for frontend.

I took the chance and created the basis for some of our client-side libraries.

It was based on modification of djangbone for backend (an excellent library I must say).

And it is based on backbone, marionette and bootstrap - for frontend.

I think it was one of the most important steps during the history of Chtd because QuBeQu (NetDB SaaS solution) was built on top of this library.

User permissions ¶

This is a story about refactoring python code.

We had rather complicated permissions system developed back to 2012. Really complicated. The author of the system has left the company a year later.

Supporting one of our projects we've got a complain from the end user about speed. It took too much time loading one specific page from our web service (more than a minute - awful).

Profiling showed that it's user permissions system to blame.

Quick tests showed a possible way of fixing it and at first I was thinking it will took from seven to ten days. Actually it took almost two months if I'm not mistaken.

I should say there was my bad refactoring tests and I lost a week or so but still - there is a huge codebase and my estimates were naive.

Refactoring the code was a great lesson.

For the first time I've tried to use Python descriptors and I really liked it.

Python descriptors are a great way of structuring your codebase.

The result of the refactoring was awesome - page load time decreased to 1 second (from more than a minute!).

Digital signature ¶

We had a couple of projects with the need for digital signature support.

In general this is about integration with CryptoPro software (Russia-only solution).

One of the tasks here was to provide full functionality for users having certificates signed by accredited certified centers.

There is a specialized portal distributing the XML list of accredited centers but that is not enough. We need to load CRLs too and there is no single point to get all CRLs from.

It took some time to develop an optimal solution (this is about fetching data from Internet).

It was one more battle for speed to load certificates and CRLs within a reasonable time.

OLAP ¶

Mostly client-side library acting as an OLAP builder built on top of our backbone-based library.

Most interesting part here was drag'n'drop to provide better user experience.

I've created the first version of this library and later it was handed over to another developer. It is one of the most valuable libraries at QuBeQu.

jQuery dashboard ¶

One more client-side library created by me.

~~I'm expecting it to be published by Chtd under the MIT license (we had such an agreement)~~. It is published by Chtd under the MIT license.

The library is used to create custom layouts of reports and charts - for dashboards.

Drag'n'drop here too.

There are more libraries and tasks I've done working for Chtd, a lot more.

Jython ¶

This is one more subject I'd like to talk about.

We had an import/export functionality providing support for Excel files (among other formats).

There were several python-only solutions but we were not satisfied by them. We needed a way to work with any xls/xlsx file keeping its original formatting untouched.

One day I've stumbled upon POI.

At first I was not sure we would be able to use it. I thought we need Java developer.

But some time later I've remembered about the Jython project. It was an experiment and it was a success.

I've splitted the functionality into several parts.

The first one was Jython-powered http service to work with Excel documents.

The second one was python library to communicate with this service and send and receive data.

And a bit later there was the third one - client-side import/export configuration library providing a way to set parameters (we can read/write data from/to different sheets and cells within Excel documents).

POI-based solution worked great and we had pretty good performance for import/export operations.

Summary ¶

It was a good time. There were moments I was not satisfied with myself or someone from colleagues but we are all human beings.

We can and we must make mistakes - to be able to create something better.

I have no opportunity but would love to work using Lean Software Development principles.

While developing sometimes it's better to think about optimisation ahead. You must find a balance inside your mind between the desire to experiment and the need to provide a solution in time.

It's better to stay focused.

It's important to form the team. Very important for the project.

Excellent article

2015-04-06T16:58:00+00:00

Excellent article ¶

Really excellent article (original in Russian).

Yesterday I've summed up my experience working for Chtd and I have one more thought to share.

It's just one more reason to minimize frameworks usage.

From my experience frameworks are a trap. You can safely use them for some time but in case your project is evolving and you want to follow the evolution of used frameworks you will find yourself spending time refactoring your code to work with that exact framework.

Instead of developing your project's business logic you have to support some framework code evolution.
It sounds strange for me.
I've faced this problem working for Chtd but I had no luck explaining the problem to colleagues.

There is a way to use framework - just freeze it on a specific version but that's another trap.

You will have to support framework at that specific version on your own fixing bugs or adding missed functionality or backporting or whatever. Do you really need it?

I think it's better to spend time focusing on your business logic.

To be a good developer you have to find your own way. Build something on your own. Make mistakes. Fix them. Explore and share your knowledge.

Become a creator, not a consumer.

Micro datetime parsing benchmark

2015-03-26T07:05:00+00:00

Micro datetime parsing benchmark ¶

Just a note for myself about datetime parsing in python.

Let's assume we know ahead that we have a datetime string in ISO 8601 format and timezone is UTC.

Here is a benchmark.

In [1]: import arrow, re, datetime as dt

In [2]: iso = arrow.utcnow().isoformat()

In [3]: iso
Out[3]: '2015-03-26T07:59:44.210642+00:00'

In [4]: r = re.compile(r'[\-T:\.]')

In [5]: arrow.get(iso)
Out[5]: <Arrow [2015-03-26T07:59:44.210642+00:00]>

In [6]: arrow.get(dt.datetime.strptime(iso.rpartition('+')[0], '%Y-%m-%dT%H:%M:%S.%f'), 'UTC')
Out[6]: <Arrow [2015-03-26T07:59:44.210642+00:00]>

In [7]: arrow.get(dt.datetime(*map(int, r.split(iso.rpartition('+')[0]))), 'UTC')
Out[7]: <Arrow [2015-03-26T07:59:44.210642+00:00]>

In [8]: %timeit arrow.get(iso)
10000 loops, best of 3: 54.7 µs per loop

In [9]: %timeit arrow.get(dt.datetime.strptime(iso.rpartition('+')[0], '%Y-%m-%dT%H:%M:%S.%f'), 'UTC')
100000 loops, best of 3: 17.5 µs per loop

In [10]: %timeit arrow.get(dt.datetime(*map(int, r.split(iso.rpartition('+')[0]))), 'UTC')
100000 loops, best of 3: 8.83 µs per loop

Unfortunatelly, the third way of parsing is wrong because it doesn't take into account microseconds zero-padded to the left.

Arrow library is just great and is certainly worth trying out.

Markdown preview

2015-03-31T12:02:00+00:00

Markdown preview ¶

Update 2. I've created mdpreview project - self-made solution to preview markdown files in browser. The code is written in Rust. Short introduction is available here.

Update 1. For reStructuredText there is an excellent restview library which works out-of-the-box.

It took half-a-workday time to find suitable solution to be able to somehow preview rendered markdown files.

GitHub uses GitHub Flavored Markdown so before committing anything to the Jukoro reporsitory it would be great to take a quick look at rendered markdown files.

My editor of choice is vim and I have rather strong intension to keep it as is.

I prefer simple solutions meaning right after save I can see changes rendered in browser (I don't need really live preview to see changes while typing). Browser-based editors don't suit me due to the fact you have to edit a file in browser.

I looked at Atom and almost decided to use it but a small annoyance forced me to look better for other ways (the annoyance is about caret disappearing in editor after switching from and back to it). For me there is one more problem with it. I'd like to turn off bold font in editor (the way I get used to working in terminal) but I have no luck searching for this feature in preferences.

There are several nodejs-based solutions but with most of them I had a problem with live update (for some reason it was not working as expected by me).

markdown-live is the solution that I chose. It works great and have a side panel to quickly switch between markdown-files. Just great.

Jukoro documentation using Sphinx

2015-04-01T10:40:00+00:00

Jukoro documentation using Sphinx ¶

Update. jukoro library has been moved to archive, no plans to develop it further.

Summary about creating python library documentation using Sphinx and publishing it to GitHub Pages (based on this article).

$ git clone git@github.com:ysegorov/jukoro.git
$ cd jukoro

$ mkdir docs
$ pip install sphinx
$ sphinx-apidoc -A "Yuri Egorov" -F -o docs jukoro

$ git add docs
$ git commit -m "sphinx-apidoc generated docs folder contents"

$ mkdir gh-pages
$ vim docs/Makefile  ## change BUILDDIR to ../gh-pages

$ /usr/share/git/workdir/git-new-workdir . gh-pages/html
$ cd gh-pages/html

$ git checkout --orphan gh-pages
$ git rm -rf .

$ cd ../../docs
$ make html
$ cd -

$ git add .
$ git commit -m "sphinx generated docs"
$ git push -u origin gh-pages

$ touch .nojekyll  ## to skip Jekyll
$ git add .nojekyll
$ git commit -m ".nojekyll"
$ git push

~~And now there is jukoro's documentation available online~~ (still requires some attention).

For some reason I've started to use markdown for readme and changelog files but now it seems it would be better to use reStructuredText.

This way it will be possible to include readme and changelog to documentation (which is not finished yet).

Jukoro 0.1.2 release

2015-04-06T17:01:00+00:00

Jukoro 0.1.2 release ¶

Update. This library has been moved to archive, no plans to develop it further.

It happened - first public release of Jukoro (not on PyPi but still).

It is an experiment and I'll do my best working on it.

Features highlight (PostgreSQL-focused) ¶

unified tables structure in PostgreSQL storing data in a single per table jsonb field
out-of-the-box way to keep historical data in tables
out-of-the-box support for psycopg2 named cursors
single per-database sequence generator for objects identifiers
out-of-the-box support for created/updated/deleted timestamps
indices and constraints support for data stored in jsonb field
connection to PostgreSQL always operate with timestamps in UTC
PostgreSQL schema support
way to introspect current database state
way to prepare sql queries to synchronize database state and entities (models) declarations (queries to create missed tables/indices/constraints and to drop orphaned tables/indices/constraints)
separation of concerns to work with PostgreSQL (explicit ways to work with pool/connection/transaction/results, to describe entities (models in terms of Django's ORM), to introspect database, to create customized sql queries)

Packages and modules ¶

jukoro.pg - package to work with PostgreSQL built on top of psycopg2
jukoro.redis - package to work with Redis having several useful helpers
jukoro.arrow - package built on top of arrow library
jukoro.decorators - module containing some useful decorators
jukoro.json - abstraction built on top of Python's stdlib json
jukoro.pickle - abstraction built on top of Python's stdlib cPickle
jukoro.base32 - abstraction built on top of base32_crockford library
jukoro.email - email helpers
jukoro.structures - some useful structures
jukoro.utils - utilities

~~I am going to use it in project I'm currently working on~~ (it was never used).

Meet Juvi

2015-06-29T08:44:00+00:00

Meet Juvi ¶

Update. This library has been moved to archive, no plans to develop it further.

While working on a pet project I've created a small client-side library responsible (mostly) for a View part in MV* patterns and have named it juvi (without any specific reason, just to have short name being easy to remember).

It is young and unstable but still looked pretty good for me.

Juvi depends on another library - proto-js which is a modified fork of Axel Rauschmayer's code. The idea behind of proto-js is very interesting and it's not clear to me why it's not as popular as it should be. Currently I'm thinking about embedding proto-js code to juvi due to the fact it's small and it will be easier to manage juvi.

So, coming back to juvi.

Originally plan was to create something simple and extendable and to have the following features included out-of-the-box:

extendability (must be a way to extend view to keep code DRY and organized)
unified way to render view
unified way to attach event handlers to DOM
unified way to declare, render and attach to DOM children views
simple and reliable way to close view on its own and children views and free resources

There is almost no magic inside view.js, just check the code (code can speak, certainly).

Probably, the most interesting part is handleEvent method. Here is an article describing the technique.

And one more note about children views.

Children views must be described in view instance as:

// code can be placed into init method

this.children = {
    $title: TitleView(somePropsObject),
    $items: this.items.map(function (it) { return ItemView.new(it); })
};

and DOM nodes as:

// code should be placed into populateNodes method

var el = this.el,
    $ = this.$;         // NB. not a jQuery!

this.nodes = {
    $title: $('[data-title]', el),
    $items: $('[data-items]', el)
};

Having such a definitions view will automatically render all children views and attach them to the specified DOM nodes:

this.children.$title view will be attached to this.nodes.$title node
this.children.$items views will be attached to this.nodes.$items node (keeping order)

There is no way to emit or listen for events per view because I'm trying to keep events flow organized in a Flux way. Here is the dispatcher.js - instance of pubsub.js.

And I almost forgot about this dependency - Promise. It's up to you to pick the one and include it in your project. Currently I'm using some parts of core.js library. Here is es6-promise polyfill suitable for pubsub too.

That's it.

The mantra stays the same.

Become a creator, not a consumer.

Django project debugging

2016-01-17T12:32:00+00:00

Django project debugging ¶

Quite often there is a need to debug some code, to see what's going on and why there is so much time spent generating pretty simple web page.

There are tools making this task easier, django-debug-toolbar looks like must have debug swiss knife but I'd like to use a self-made solution.

I used to use logging as a way to get the information needed so here are two classes to debug code execution time and SQL queries and timings. They can be used as decorators or as context managers (see below).

To initialize these classes you will have to pass prefix as the only argument (needed to grep logging output).

Here are some abstract usage examples:


# -*- coding: utf-8 -*-

from django.shortcuts import render

from project.articles import get_articles, get_stats
from project.debug import timer, debug_sql


@debug_sql('index')
@timer('index')
def index(request):
    return render(request, 'index.html')


def articles(request):
    with timer('articles'):
        articles = get_articles()
        with debug_sql('articles'):
            stats = get_stats(articles)
    return render(request, 'articles.html', context={'articles': articles,
                                                     'stats': stats})

It's pretty easy to use the helpers, there is no need for a project-wide configuration and you can focus on the code you are debugging to avoid logging noise.

Webpack loader to use svg icons as snabbdom VNode

2016-07-04T08:45:00+00:00

Webpack loader to use svg icons as snabbdom VNode ¶

Update. This library has been moved to archive, no plans to develop it further.

I'm exploring FRP for frontend and it looks very interesting to me, especially for dataflow using streams.

Unidirectional data flow, introduced as Flux by Facebook engineers, is an excellent paradigm and I was thinking about finding a way to use it without React (due to the fact I don't like it - just the same way I don't like Angular as both of them look overengineered to me).

It took some time to understand the subject of FRP and there are a lot of useful resources available (just a couple of links here - Mostly Adequate Guide to Functional Programming, React-less Virtual DOM with Snabbdom, The introduction to Reactive Programming).

My first attempt was to understand snabbdom / flyd / union-type / ramda approach and the most challenging part was union types.

Studying The ELM Architecture helped a lot in this.

To dive deeper on the subject I've created a very simple implementation of union types which in it's core just tags values using specified tag name:

// type.js

var curryN = require('ramda/src/curryN');

function Type(spec) {
    var o = {},
        keys = Object.keys(spec);

    keys.forEach(function (key) {
        var fn = spec[key];
        if (spec[key].length > 2) {
            o[key] = curryN(fn.length - 1, function () {
                return {
                    value: [].slice.apply(arguments),
                    tag: key,
                    handler: fn
                };
            });
        } else {
            o[key] = function (smth) {
                return {
                    value: smth,
                    tag: key,
                    handler: fn
                };
            };
        }
    });
    function update(tagged, model) {
        var tag = tagged.tag,
            value = tagged.value,
            fn = tagged.handler;
        value = Array.isArray(value) ? value
                                        : (value != null ? [value] : []);
        value.push(model);
        return fn.apply(null, value);
    }
    o.update = curryN(2, update);
    return o;
}

Example of union type definition:

// component.js

var Type = require('./type');
var R = require('ramda');

var Msg = Type({
    Click: function (evt, model) { return R.assoc('clicks', model.clicks + 1, model); },
    Remove: function (evt, model) { return R.assoc('_removed', true, model); },
    Tick: function (model) {
        return R.assoc('count', model.count > 359 ? 0 : model.count + 1, model);
    }
});

It is a bit simplier than union-type, doesn't have type checking and probably missing something in this implementation.

Anyway, it helped me to understand the data flow.

My experiments are available in this project (work in progress).

I like to use svg icons for ui components and I used to create separate html file with all icons embedded in it using single svg and multiple symbol definitions and referencing them using xlink:href attribute.

I was planning to use icons to enrich the ui of the experiments and thought it would be great to have svg icons available as snabbdom VNodes.

So, here is the webpack loader for this (this is my first published npm module).

Usage is pretty simple just like any webpack loader:

// icons.js

var home = require('./svg/home.svg'),
    alert = require('./svg/alert.svg'),
    ...

module.exports = {
    home: home,
    alert: alert,
    ...
};

// alerts.js

var icons = require('./icons'),
    h = require('snabbdom/h');

function view(model) {
    return h('div.info', {}, [icons.alert, 'Some alert here']);
}

The loader can be chained with svgo-loader, please check the docs for this.

That's it for now.

Alcor Micro AU6601 PCI-E Flash card reader controller

2016-09-30T05:16:00+00:00

Alcor Micro AU6601 PCI-E Flash card reader controller on ArchLinux ¶

Update. Please check this post for installation instructions for Linux kernel 4.16+.

I'm running ArchLinux on this perfect ASUSPRO BU401LG notebook by ASUS and the only problem I faced recently was sd-card reader not working at all.

lspci shows the following:


$ lspci
00:00.0 Host bridge: Intel Corporation Haswell-ULT DRAM Controller (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated Graphics Controller (rev 09)
00:03.0 Audio device: Intel Corporation Haswell-ULT HD Audio Controller (rev 09)
00:04.0 Signal processing controller: Intel Corporation Device 0a03 (rev 09)
00:16.0 Communication controller: Intel Corporation 8 Series HECI #0 (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I218-V (rev 04)
00:1b.0 Audio device: Intel Corporation 8 Series HD Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 8 Series PCI Express Root Port 1 (rev e4)
00:1c.1 PCI bridge: Intel Corporation 8 Series PCI Express Root Port 2 (rev e4)
00:1c.3 PCI bridge: Intel Corporation 8 Series PCI Express Root Port 4 (rev e4)
00:1c.4 PCI bridge: Intel Corporation 8 Series PCI Express Root Port 5 (rev e4)
00:1d.0 USB controller: Intel Corporation 8 Series USB EHCI #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation 8 Series LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 8 Series SATA Controller 1 [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 8 Series SMBus Controller (rev 04)
00:1f.6 Signal processing controller: Intel Corporation 8 Series Thermal (rev 04)
02:00.0 Unassigned class [ff00]: Alcor Micro AU6601 PCI-E Flash card reader controller
03:00.0 Network controller: Intel Corporation Wireless 7260 (rev 6b)
04:00.0 3D controller: NVIDIA Corporation GK208M [GeForce GT 730M] (rev a1)

There are no events in logs after inserting sd-card into slot. So, after a bit of searching here is the solution.

Thanks to Sabin Iacob there is a driver for the card reader and the only thing to do is to install it.

Driver is packaged for Ubuntu and these are the steps I've done to install it on Arch:

download appropriate deb-package (i386 or amd64) from packages page
install debtap package from AUR if not already installed
initialize debtap database
convert deb package to archlinux package using debtap
install generated package using yaourt -U
install kernel driver
use card reader :)


$ wget https://launchpad.net/~iacobs/+archive/ubuntu/au6601/+files/au6601-dkms_0.3-0~ppa7_amd64.deb
$ yaourt -S debtap
$ sudo debtap -u
$ debtap au6601-dkms_0.3-0-ppa7_amd64.deb
$ yaourt -U au6601-dkms-0.3-1-x86_64.pkg.tar.xz
$ sudo modprobe au6601-pci
$ udevil mount /dev/mmcblk0p1

# profit!

DO

2016-10-11T07:55:00+00:00

DO ¶

Not long ago reading some news I've read about Letters Live project and looking through the letters I've came to a brilliant one.

Certainly, this letter is very impressive and powerfull and I want to keep it here.

Text is taken from correspondent Letters Live page and you definitely should go there to watch the video.

Letter from Sol LeWitt to Eva Hesse, April 14, 1965 ¶

Dear Eva,

It will be almost a month since you wrote to me and you have possibly forgotten your state of mind (I doubt it though). You seem the same as always, and being you, hate every minute of it. Don’t! Learn to say “Fuck You” to the world once in a while. You have every right to. Just stop thinking, worrying, looking over your shoulder, wondering, doubting, fearing, hurting, hoping for some easy way out, struggling, grasping, confusing, itching, scratching, mumbling, bumbling, grumbling, humbling, stumbling, numbling, rambling, gambling, tumbling, scumbling, scrambling, hitching, hatching, bitching, moaning, groaning, honing, boning, horse-shitting, hair-splitting, nit-picking, piss-trickling, nose sticking, ass-gouging, eyeball-poking, finger-pointing, alleyway-sneaking, long waiting, small stepping, evil-eyeing, back-scratching, searching, perching, besmirching, grinding, grinding, grinding away at yourself. Stop it and just

DO

From your description, and from what I know of your previous work and your ability; the work you are doing sounds very good “Drawing – clean – clear but crazy like machines, larger and bolder… real nonsense.” That sounds fine, wonderful — real nonsense. Do more. More nonsensical, more crazy, more machines, more breasts, penises, cunts, whatever — make them abound with nonsense. Try and tickle something inside you, your “weird humor.” You belong in the most secret part of you. Don’t worry about cool, make your own uncool. Make your own, your own world. If you fear, make it work for you — draw & paint your fear & anxiety. And stop worrying about big, deep things such as “to decide on a purpose and way of life, a consistant approach to even some impossible end or even an imagined end.” You must practice being stupid, dumb, unthinking, empty. Then you will be able to

DO

I have much confidence in you and even though you are tormenting yourself, the work you do is very good. Try to do some BAD work — the worst you can think of and see what happens but mainly relax and let everything go to hell — you are not responsible for the world — you are only responsible for your work — so DO IT. And don’t think that your work has to conform to any preconceived form, idea or flavor. It can be anything you want it to be. But if life would be easier for you if you stopped working — then stop. Don’t punish yourself. However, I think that it is so deeply engrained in you that it would be easier to

DO

It seems I do understand your attitude somewhat, anyway, because I go through a similar process every so often. I have an “Agonizing Reappraisal” of my work and change everything as much as possible — and hate everything I’ve done, and try to do something entirely different and better. Maybe that kind of process is necessary to me, pushing me on and on. The feeling that I can do better than that shit I just did. Maybe you need your agony to accomplish what you do. And maybe it goads you on to do better. But it is very painful I know. It would be better if you had the confidence just to do the stuff and not even think about it. Can’t you leave the “world” and “ART” alone and also quit fondling your ego. I know that you (or anyone) can only work so much and the rest of the time you are left with your thoughts. But when you work or before your work you have to empty your mind and concentrate on what you are doing. After you do something it is done and that’s that. After a while you can see some are better than others but also you can see what direction you are going. I’m sure you know all that. You also must know that you don’t have to justify your work — not even to yourself. Well, you know I admire your work greatly and can’t understand why you are so bothered by it. But you can see the next ones & I can’t. You also must believe in your ability. I think you do. So try the most outrageous things you can — shock yourself. You have at your power the ability to do anything.

I would like to see your work and will have to be content to wait until Aug or Sept. I have seen photos of some of Tom’s new things at Lucy’s. They are very impressive — especially the ones with the more rigorous form; the simpler ones. I guess he’ll send some more later on. Let me know how the shows are going and that kind of stuff.

My work has changed since you left and it is much better. I will be having a show May 4–29 at the Daniels Gallery 17 E 64th St (where Emmerich was), I wish you could be there. Much love to you both.

Sol

Pipeline As a Code With Jenkins

2016-12-27T10:45:00+00:00

Pipeline as a code with Jenkins ¶

Jenkins has a pretty nice Pipeline plugin to manage build, test and deploy steps in a single file named Jenkinsfile. This file should be stored alongside with the project code just like python, javascript, html or whatever files.

The only problem with Jenkinsfile is that it must be written in Groovy. To me this is the same problem as with Vagrantfile which must be written in Ruby. There is something strange forcing users to study language in order to have proper configuration file.

It took some time to prepare workable Jenkinsfile which works for Multibranch Workflows so I decided to create a gist with it.

Here it is:

It works pretty good for me providing slack notifications to production or staging channels and having links to Bitbucket diff page to see changes commited between builds.

This Jenkinsfile is suitable for python or nodejs projects and requires just minimal tuning - to provide proper repository name and url and build, test and deploy commands for the project. Deployment happens only for develop (staging server) and master (production server) branches.

Hope somebody will find this gist usefull.

Self-made Virtualenv Wrapper

2016-12-28T08:27:00+00:00

Self-made Virtualenv Wrapper ¶

Not long ago I've switched from virtualenvwrapper to self-made solution based on this idea by datagrok. It works pretty well for me and I'd like to share it here.

The idea is to have a root directory ($HOME/_dev in my case) for all projects and to have shell helper to easily jump into any project and to automatically activate it's virtualenv if it exists.

Here is how it looks in bash:


[local] [egorov@asus 11:33:19]:~  $ dev blog
Virtual environment found in /home/egorov/_dev/blog/env, activated.
(blog) [local] [egorov@asus 11:34:31]:~/_dev/blog git:master= $

Besides just jumping into directory I wanted to have tab autocompletion and easy way to create new projects home directory:


# autocompletion
[local] [egorov@asus 11:43:01]:~  $ dev service-  # tab hitted twice
service-account   service-feedback  service-products  service-workers

# new project home
[local] [egorov@asus 11:52:08]:~  $ dev rogue
mkdir: created directory '/home/egorov/_dev/rogue'
No virtual environment found, skipping activate.
[local] [egorov@asus 11:52:25]:~/_dev/rogue  $

So, here are bash helpers which should be placed somewhere inside ~/.bashrc or ~/.bash_profile (mine lives there). _dev function depends on inve shell script which must be available in PATH (I have $HOME/bin in PATH for this and inve lives there).


# dev
DEV=${HOME}/_dev  # home for all projects

_dev() {
    local p=${DEV}/$1
    [ ! -d $p ] && mkdir $p
    cd $p
    inve  # binary file to be accessible in PATH
    return 0
}
_dev_complete()
{
    local cur prev opts
    local p=${DEV}
    [ ! -d $p ] && `which mkdir` $p
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts=`cd $p && find -L . -maxdepth 1 -type d | sed 's|[\./]||g'`

    COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
}
alias dev=_dev
complete -F _dev_complete dev

inve executable content:


#!/usr/bin/env bash

# tuned idea from: https://gist.github.com/datagrok/2199506
#
# inve
#
# For use with Ian Bicking's virtualenv tool. Attempts to find the root of
# a virtual environment and activate it in subshell.

# First, locate the root of the current virtualenv
cwd=$PWD
export DEV_ROOT=$cwd

while [ "$PWD" != "/" ]; do
    # Stop here if this the root of a virtualenv
    # will look for virtualenv in `current dir` or in `env` subdirectory
    if [ \
        -x bin/python \
        -a -e lib/python*/site.py \
        -a -e include/python*/Python.h ]
    then
        export VIRTUAL_ENV="$PWD"
        break
    elif [ \
        -x env/bin/python \
        -a -e env/lib/python*/site.py \
        -a -e env/include/python*/Python.h ]
    then
        export VIRTUAL_ENV="$PWD/env"
        break
    fi
    cd ..
done
if [ "$PWD" = "/" ]; then
    export PATH="$cwd/node_modules/.bin:$PATH"
    echo "No virtual environment found, skipping activate." >&2
    cd $cwd
else
    # Activate
    export PATH="$VIRTUAL_ENV/bin:$cwd/node_modules/.bin:$PATH"
    unset PYTHONHOME
    echo "Virtual environment found in ${VIRTUAL_ENV}, activated." >&2
fi
exec "$SHELL"

inve script looks for virtualenv in env subdirectory or attempts to find it somewhere upper in the filesystem tree. It modifies PATH prepending $VIRTUALENV/bin and node_modules/.bin directories and starts new shell. I like the idea of starting a new subshell in order to activate virtualenv as this way we can be sure there will be no problems with PATH within shell and it's pretty safe to exit this subshell and have parent shell clean as it should be.

To easily navigate to some directories within activated virtualenv I have a couple of bash aliases defined:


alias cdsrc='cd ${VIRTUAL_ENV}/src'
alias cdsitepackages='cd ${VIRTUAL_ENV}/lib/python*/site-packages/'
alias cdproject='cd ${DEV_ROOT:-VIRTUAL_ENV}; if [ `basename ${PWD}` = "env" ]; then cd ..; fi'

This approach allows me to have virtual environments using python2 or python3 under single root directory and to have single tool to jump into project be it python2, python3 or even nodejs project.

Everything works pretty good for me.

Static blog generator

2026-07-26T11:41:00+00:00

Static Blog Generator ¶

Note

Update 1. This library has been moved to archive, no plans to develop it further.

Note

Update 2. This library was unarchived, rebuilt from scratch and released as 3.0.0 version.

This is a short introduction to wt - library I've made for my personal use.

It is pretty young and it lacks some features I'd like it to have but it works pretty good for me.

So, this is yet another static site/blog generator. Yes. One more.

Some time ago I thought it would be great to build something for my own use. I have several ideas and this is the first one named wt. I know that wt is a name for executable from WiredTiger distribution but I think it's ok. This library is not expected to be installed system-wide, it should be used within virtual environment. And luckily the name was free on pypi.

The library requires python 3 and have the following features (citing from readme):

markdown for content
yaml for configuration
jinja2 for templating
atom for feed
aiohttp for development server
only two types of content - page and post
content metadata lives in configuration file
have sensible defaults for content sources
no python coding needed to work with

At the moment I've used it in a couple of projects.

The first one is this blog (blog source code) and another one is the tolsha.ru site (in Russian) I've created for my elder brother who makes pottery and they are amazing (site source code).

These two sites are pretty different in nature but wt works very well for both.

I'll keep this library moving forward adding more features and refactoring some internals for a better code.

~~The next project I have plans for is rss reader and there is a big chance I'll build it using Elixir/Phoenix~~ (not happened).

Logger service using epoll - Part 1

2017-03-24T06:45:00+00:00

Logger service using epoll - Part 1 ¶

Update. There is a continuation of this post.

Suppose we have a number of microservices and we need some way to easily handle logging from all of them.

Graylog might be a good choice but in case you don't need such a full-featured solution self-made logging service can serve you well.

Python's logging facility has almost everything we need for this providing SocketHandler to send log records to a network socket. The missed part is actual logging service which will be able to receive and save those records. Logging Cookbook contains basic implementation of such a service which is good for a quick start.

I was experimenting with curio as an engine for such a service and it worked pretty well and certainly might be a good choice for this task. The only thing forced me to look for another solution was it's Python 3 dependency. All microservices I need logging service for are using Python 2 and I'd like it to use the very same python.

So, after a bit more experiments I came up with edge-triggered epoll-based solution which is compatible with Python 2 and Python 3. The code for the service is available here.

It took some time to understand epoll mechanics and this article by Scot Doyle helped a lot. My local test confirms service is pretty stable and properly handle incoming logging records.

Code in gist is rather generic and I'm going to add some features, but they won't be published there, sorry for that:

use some magic number in message header as a simple measure to prevent unknown clients from spamming (even though the service will listen on localhost)
enforce message size limits
store logging records in database

Another part of this story is SocketHandler which has a feature I'd like to remove - createSocket method implemets exponential back-off algorithm which I just don't need. During my tests intentionally broken (just for tests) connection to logging service leads to logging records being missed.

To be clear with the problem. Suppose we have a server and a microservice. Everything works as expected and is in good shape. Some time later we need to update the code for logging service and restart it. Using SocketHandler from python's standard library will force us to restart our microservice because without this step we will be missing at least one or two logging records from it (due to the way this handler works).

I wanted a way to automatically handle such a situation without user intervention. So, below is a modified version of SocketHandler. Every time it needs to handle a record it will try to connect to logging service to guarantee message delivery. It uses small trick to detect socket state and to re-establish connection when needed.

# -*- coding: utf-8 -*-

from __future__ import absolute_import

import logging
import socket
import struct
import traceback
try:
    import cPickle as pickle
except ImportError:  # pragma: no cover
    import pickle


class SocketHandler(logging.Handler):
    """
    [!] modified version of python's logging.handlers.SocketHandler

    """

    def __init__(self, host, port):
        logging.Handler.__init__(self)
        self.host = host
        self.port = port
        self._sock = None
        self.closeOnError = 1
        self.retry_count = 3

    def makeSocket(self):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        try:
            s.connect((self.host, self.port))
        except socket.error:
            # TODO sentry
            traceback.print_exc()
            s = None
        else:
            s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
        return s

    def _get_sock(self):
        if self._sock is None:
            self._sock = self.makeSocket()
        return self._sock

    def _set_sock(self, value):
        if value is None and self._sock:
            self._sock.close()
        self._sock = value

    sock = property(_get_sock, _set_sock)

    def get_sock_state(self, s):
        # see http://stackoverflow.com/a/18189190
        fmt = 'B' * 7 + 'I' * 24
        sz = struct.calcsize(fmt)
        x = struct.unpack(
            fmt, s.getsockopt(socket.IPPROTO_TCP, socket.TCP_INFO, sz))
        return x

    def send(self, s):
        retries = self.retry_count
        while retries > 0:
            sock = self.sock
            retries -= 1
            if sock:
                state = self.get_sock_state(sock)
                # check tcp_info.tcpi_state value is TCP_ESTABLISHED
                if state[0] != 1:
                    self.sock = None
                    continue
                try:
                    if hasattr(sock, 'sendall'):
                        sock.sendall(s)
                    else:
                        sentsofar = 0
                        left = len(s)
                        while left > 0:
                            sent = sock.send(s[sentsofar:])
                            sentsofar = sentsofar + sent
                            left = left - sent
                    break
                except socket.error:
                    self.sock = None

    def makePickle(self, record):
        ei = record.exc_info
        if ei:
            dummy = self.format(record)  # noqa
            record.exc_info = None  # to avoid Unpickleable error
        d = dict(record.__dict__)
        d['msg'] = record.getMessage()
        d['args'] = None
        s = pickle.dumps(d, pickle.HIGHEST_PROTOCOL)
        if ei:
            record.exc_info = ei  # for next handler
        slen = struct.pack('>L', len(s))
        return slen + s

    def handleError(self, record):
        if self.closeOnError and self.sock:
            self.sock = None
        else:
            logging.Handler.handleError(self, record)

    def emit(self, record):
        try:
            s = self.makePickle(record)
            self.send(s)
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            # TODO sentry
            traceback.print_exc()
            self.handleError(record)

    def close(self):
        self.acquire()
        try:
            if self.sock:
                self.sock = None
        finally:
            self.release()
        logging.Handler.close(self)

I'll put logging service and this handler in production next month. Stay tuned.

The finger pointing at the moon is not the moon

2017-03-26T12:51:00+00:00

The finger pointing at the moon is not the moon ¶

14 precepts by Thich Nhat Hanh.

Do not be idolatrous about or bound to any doctrine, theory, or ideology, even Buddhist ones. All systems of thought are guiding means; they are not absolute truth.

Do not think that the knowledge you presently possess is changeless, absolute truth. Avoid being narrow-minded and bound to present views. Learn and practice non-attachment from views in order to be open to receive others' viewpoints. Truth is found in life and not merely in conceptual knowledge. Be ready to learn throughout our entire life and to observe reality in yourself and in the world at all times.

Do not force others, including children, by any means whatsoever, to adopt your views, whether by authority, threat, money, propaganda, or even education. However, through compassionate dialogue, help others renounce fanaticism and narrowness.

Do not avoid contact with suffering or close your eyes before suffering. Do not lose awareness of the existence of suffering in the life of the world. Find ways to be with those who are suffering by all means, including personal contact and visits, images, sound. By such means, awaken yourself and others to the reality of suffering in the world.

Do not accumulate wealth while millions are hungry. Do not take as the aim of you life fame, profit, wealth, or sensual pleasure. Live simply and share time, energy, and material resources with those who are in need.

Do not maintain anger or hatred. As soon as anger and hatred arise, practice the meditation on compassion in order to deeply understand the persons who have caused anger and hatred. Learn to look at other beings with the eyes of compassion.

Do not lose yourself in dispersion and in your surroundings. Learn to practice breathing in order to regain composure of body and mind, to practice mindfulness, and to develop concentration and understanding.

Do not utter words that can create discord and cause the community to break. Make every effort to reconcile and resolve all conflicts, however small.

Do not say untruthful things for the sake of personal interest or to impress people. Do not utter words that cause diversion and hatred. Do not spread news that you do not know to be certain. Do not criticize or condemn things you are not sure of. Always speak truthfully and constructively. Have the courage to speak out about situations of injustice, even when doing so may threaten your own safety.

Do not use the Buddhist community for personal gain or profit, or transform your community into a political party. A religious community should, however, take a clear stand against oppression and injustice, and should strive to change the situation without engaging in partisan conflicts.

Do not live with a vocation that is harmful to humans and nature. Do not invest in companies that deprive others of their chance to life. Select a vocation which helps realize your ideal compassion.

Do not kill. Do not let others kill. Find whatever means possible to protect life and to prevent war.

Possess nothing that should belong to others. Respect the property of others but prevent others from enriching themselves from human suffering or the suffering of other beings.

Do not mistreat your body. Learn to handle it with respect. Do not look on your body as only and instrument. Preserve vital energies (sexual, breath, spirit) for the realization of the Way. Sexual expression should not happen without love and commitment. In sexual relationships be aware of future suffering that may be caused. To preserve the happiness of others, respect the rights and commitments of others. Be fully aware of the responsibility of bringing new lives into the world. Meditate on the world into which you are bringing new beings.

No words can replace practice, only practice can make the words.

Logger service using epoll - Part 2

2017-03-30T16:45:00+00:00

Logger service using epoll - Part 2 ¶

Update. This is a follow-up for part 1 post about logger service.

So, the task is to have logger service capable of handling, let's say, 1k logging records per second. I am expecting about 100 client connections to this service and It's clear that it must be stable enough to handle incoming requests.

Please do remember that all figures in this post I've got running the code on my laptop - YMMV.

To test the service I have this script:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os
import time
import random
import logging
import logging.config
import multiprocessing as mp


def setup_logging():
    conf = {
        'version': 1,
        'disable_existing_loggers': False,
        'formatters': {
            'default': {
                'format': (
                    '%(levelname)1.1s %(asctime)s %(name)s:%(lineno)3s '
                    '%(message)s'),
                'datefmt': '%Y.%m.%d %H:%M:%S',
            }
        },
        'handlers': {
            'service_logger': {
                'level': 'DEBUG',
                # must be importable
                'class': 'svctools.handlers.SocketHandler',
                'host': '127.0.0.1',
                'port': 5000,
            },
        },
        'loggers': {
            '': {
                'level': 'DEBUG',
                'handlers': ['service_logger'],
                'propagate': False,
            },
        },
    }
    logging.config.dictConfig(conf)


def run(idx):
    msg = '[%06d] log message %08d'
    pid = os.getpid()
    logger = logging.getLogger(__name__)
    cnt = count = random.randint(300, 800)
    while True:
        logger.debug(msg, pid, cnt, extra={'trace_id': idx})
        cnt -= 1
        time.sleep(random.choice((0.01, 0.02, 0.03, 0.04, 0.03, 0.02, 0.01)))
        if cnt == 0:
            break

    logger.debug('[%06d] --------', pid, extra={'trace_id': idx})

    return count


def main():
    pool = mp.Pool(200, setup_logging)
    r = pool.map_async(run, range(1000))

    data = r.get()
    print('sent %s records' % sum(data))


if __name__ == "__main__":
    main()

We have pool of 200 processes performing the same action - just calling logger.debug. Single action to be performed by a process is the following:

randomly select number of logging messages to be send to service (somewhere between 300 and 800 messages)
send messages one by one with a tiny delay in between (to simulate some kind of real load)

We will be running 1000 of such actions which means we will send about 500k messages to our service (this is just an approximation).

Step 1 - logging to a file (only) ¶

It was not interesting at all as I've got about 8700 records per second - very impressive. But we need all records to be stored in database - to do a query later. Let's see.

Step 2 - logging to a file and to PostgreSQL using logging handler ¶

The code for PostgreSQL logging handler is available here. This time results were disappointing - about 200 records per second (at the same time there was no data loss which is good).

Step 3 - logging to a file and to PostgreSQL using self-made handler ¶

It was clear that our service everytime runs the same query inserting just one record to database. So, the trick is to employ buffering and to insert records in bulk using, for example, psycopg2.extras.execute_values. This time I've got about 3600 records per second using 1k-blocks - much better. For a batches of 100 records I've got about 3000 records per second which is a bit slower but still good enough.

This solution has one trick though. As we are prebuffering records there will be times buffer is not full and there is no incoming records for some time. This can lead to data loss which is bad. To solve this problem we can use signal.setitimer to setup simple timer to send SIGALRM on a regular basis and to flush receiving buffer on it.

Here are the features of the logger service:

no threads or child processes
edge-triggered epoll for i/o
PostgreSQL backend
the code looks ugly

Last feature will be fixed soon. Stay tuned.

WT documentation

2026-07-26T12:22:00+00:00

WT documentation ¶

Note

Update. wt library has been moved to archive, no plans to develop it further.

Note

Update 2. wt library was unarchived, rebuilt from scratch and released as 3.0.0 version.

This is just a few words about wt documentation.

It's ~~available online~~ and covers almost all wt features.

And it was built using wt. Documentation sources are available in the wt repository under docs/ folder.

There was minor issue with deployment though.

~~The documentation is hosted using GitHub pages for projects and it's served under baseurl (/wt-docs/ in my case)~~. I was not aware of this requirement while developing wt so internally it's not supported. This means there is a difference between local links in development mode and in build mode which can lead to broken local links in build mode.

I've solved the problem but I'm not sure I like the solution. There must be a better way to handle baseurl (this issue was solved in wt later).

Reloadable decorator

2026-07-26T12:24:00+00:00

Reloadable decorator ¶

Note

Update 1. wt library has been moved to archive, no plans to develop it further.

Note

Update 2. wt library was unarchived, rebuilt from scratch and released as 3.0.0 version.

wt has a feature I'd like to share here. It was one of the first features of wt I think.

Problem ¶

Suppose I have an idea for some post, just like this one.

I've opened terminal, started wt in development mode using wt develop command and started vim for authoring.

I'm typing for some time...

Ok, now I'd like to check it in browser and I'm adding some metadata for this post to configuration - wt.yaml file.

After this step I'm expecting refreshed page in the browser to contain updated data from configuration.

Solution ¶

Internally wt has a so called engine to handle incoming requests. It depends on wt.yaml configuration and reads it on initialization.

It is quite possible for the engine to contain logic to reload configuration when needed but I thought this task shouldn't be implemented there.

It would be better to have the possibility to easily turn this feature off when needed (not while authoring but while developing the library) or even just throw away.

So, here comes reloadable decorator:

# -*- coding: utf-8 -*-

import logging
import os


class reloadable(object):

    def __init__(self, message):
        self._message = message
        self._values = {}
        self._lastmods = {}
        self.logger = logging.getLogger('wt.reloadable')

    def __call__(self, fn):

        def inner(filename):
            v = self._values.get(filename)
            m = self._lastmods.get(filename)
            try:
                modified = os.stat(filename).st_mtime
            except FileNotFoundError:
                self.logger.warn(
                    'File not found "%s", skipping stat', filename)
                pass
            else:
                if m is None or modified > m:
                    self.logger.debug(
                        'File "%s" modified, %s', filename, self._message)
                    self._lastmods[filename] = modified
                    v = None

            if v is None:
                self._values[filename] = v = fn(filename)
            return v
        return inner

It keeps last modified times for files and caches results of function calls. In case file was modified cached value will be automatically updated. Pretty simple yet very useful.

Real life usage example looks like this:

# -*- coding: utf-8 -*-

from .decorators import reloadable
from .engine import WT


@reloadable('(re)loading configuration...')
def engine(fn):
    return WT(fn)

Currently wt uses aiohttp as http backend.

It's only configured handler calls engine function to get rendering engine instance (you can check server module sources for details).

In case configuration file was updated this call will create new fresh instance of engine ready to render pages and posts. This instance will be cached within reloadable decorator cache so any subsequent calls to engine function will just have the very same instance in return (till configuration was updated again).

That's it for reloadable decorator. Stay tuned.

Dotenv

2017-04-13T16:40:00+00:00

Dotenv ¶

I've got used to python-dotenv library to customize some project's settings on per-deployment basis. But there is one thing which I can't say I like.

Problem ¶

The problem is the way this library makes values from .env available to project - by using os.environ.setdefault which actually pollutes project's environment.

By occasion this can lead to some sensitive information like api keys to leak. Let's fix it.

Solution ¶

My first attempt was to check the library's API and find a way to load values without polluting os.environ. This is possible and here is the code:

# -*- coding: utf-8 -*-

import os

from dotenv.main import dotenv_values

BASE_DIR = os.path.dirname(os.path.abspath(__file__))
DOTFILE = os.environ.get('DOTFILE', '.env')

dotenv = {}
dotenv_path = os.path.join(BASE_DIR, DOTFILE)
if os.path.isfile(dotenv_path):
    dotenv = dotenv_values(dotenv_path)

def env(key, default=None):
    return os.environ.get(key, dotenv.get(key, default))

API_KEY = env('API_KEY', 'test-key')

This solution works and looks pretty, well, reasonable. At the same time dotenv_values function is not documented and there is a chance API can be changed.

Besides python-dotenv has features I don't need, like command-line interface or automatic .env file location up the directory tree (I always know where .env file should be).

So, I've rolled my own solution which has fewer lines of code and is very easy to reason about:

# -*- coding: utf-8 -*-

from __future__ import absolute_import

import os
import codecs
import logging


logger = logging.getLogger(__name__)


def load_dotenv(filename, encoding='utf-8'):

    decoder = codecs.getdecoder('unicode_escape')

    def decode(v):
        if v and v[0] == v[-1] and v[0] in ('"', '\''):
            v = decoder(v[1:-1])[0]
        return v

    with codecs.open(filename, 'rb', encoding=encoding) as f:
        lines = (x.strip() for x in f if x.strip())
        lines = (x for x in lines if not x.startswith('#') and '=' in x)
        lines = (x.split('=', 1) for x in lines)
        lines = ((x.strip(), y.strip()) for x, y in lines)
        lines = ((x.upper(), decode(y)) for x, y in lines)
        for line in lines:
            yield line


class dotenv(object):

    def __init__(self, filename='.env', encoding='utf-8'):
        fn = filename
        if not os.path.isabs(filename):
            filename = os.path.abspath(os.path.join(os.curdir, filename))
        if not os.path.exists(filename):
            logger.warning('dotenv file "%s" not found', fn)
            filename = None
        self.filename = filename
        self.encoding = encoding
        self._data = None

    @property
    def data(self):
        if self._data is None and self.filename is not None:
            self._data = dict(
                load_dotenv(self.filename, encoding=self.encoding))
        return self._data

    def get(self, key, default=None):
        data = self.data or {}
        return os.environ.get(key, data.get(key, default))

It can be used like this:

# -*- coding: utf-8 -*-

import os

from .dotenv import dotenv

BASE_DIR = os.path.dirname(os.path.abspath(__file__))
DOTFILE = os.environ.get('DOTFILE', '.env')

env = dotenv(filename=os.path.join(BASE_DIR, DOTFILE))

API_KEY = env.get('API_KEY', 'test-key')

It is very easy to extend dotenv class with methods like get_bool, get_int, get_list or whatever you need and provide some logic to verify settings values as needed.

Value coming from os.environ takes precedence over .env value when specified (this is useful while running jenkins jobs).

By implementing self-made solution we can simplify our project's dependency tree and be sure everything works as we need.

That's it for dotenv. Stay tuned.

AU6601 PCI-E Flash card reader and Linux kernel 4.16

2018-05-05T07:12:00+00:00

Alcor Micro AU6601 PCI-E Flash card reader controller and Linux kernel 4.16 ¶

I've got email from Justin Faber today saying the procedure, described here, doesn't work for Linux kernel 4.16.6-1 in ArchLinux - build fails with the following log:

make -C /lib/modules/4.16.6-1-ARCH/build/ SUBDIRS=/var/lib/dkms/au6601/0.3/build modules
make[1]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
make[1]: Entering directory '/usr/lib/modules/4.16.6-1-ARCH/build'
  CC [M]  /var/lib/dkms/au6601/0.3/build/au6601.o
/var/lib/dkms/au6601/0.3/build/au6601.c: In function ‘au6601_pci_probe’:
/var/lib/dkms/au6601/0.3/build/au6601.c:1187:2: error: implicit declaration of function ‘setup_timer’; did you mean ‘setup_irq’? [-Werror=implicit-function-declaration]
  setup_timer(&host->timer, au6601_timeout_timer, (unsigned long)host);
  ^~~~~~~~~~~
  setup_irq
/var/lib/dkms/au6601/0.3/build/au6601.c: In function ‘au6601_send_cmd’:
/var/lib/dkms/au6601/0.3/build/au6601.c:619:5: warning: ‘ctrl’ may be used uninitialized in this function [-Wmaybe-uninitialized]
  u8 ctrl; /* some mysterious flags and control */
     ^~~~
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:325: /var/lib/dkms/au6601/0.3/build/au6601.o] Error 1
make[1]: *** [Makefile:1562: _module_/var/lib/dkms/au6601/0.3/build] Error 2
make[1]: Leaving directory '/usr/lib/modules/4.16.6-1-ARCH/build'
make: *** [Makefile:33: default] Error 2

DuckDuckGo (my default search engine, by the way) helped to localize the issue - changes in Linux kernel timers API.

I've followed instructions from the commit comments and prepared the patch and PKGBUILD to simplify installation:

Please use this procedure and generated package at your own risk as I'm not a Linux kernel driver developer.

Here is an updated version of steps to install the driver for amd64 architecture and Linux kernel 4.16+:

create temporary directory somewhere and cd into it
download patch and PKGBUILD (ensure PKGBUILD and au6601-timers.diff are located in the root of previously created temporary directory)
build and install package using makepkg
install kernel driver
use card reader :)


$ mkdir au6601-dkms && cd au6601-dkms
$ wget -O au6601-pkgbuild.zip https://gist.github.com/ysegorov/951ae99426f5bd9092c3b4138e207af2/archive/master.zip
$ unzip -j au6601-pkgbuild.zip
$ makepkg -si
$ sudo modprobe au6601-pci
$ udevil mount /dev/mmcblk0p1

# profit!

Wireguard and Linux network namespaces - perfect fit

2020-04-22T17:35:00+00:00

Wireguard and Linux network namespaces - perfect fit ¶

I'm using two browsers usually - Firefox (switched to IceCat recently) and Chromium.

Firefox/IceCat is my workhorse, which is configured to route all traffic through SOCKS5 proxy (Linode instance actually).

I would definitely stay solely with it but some sites don't like requests coming from VPS servers so I have to use Chromium for such cases (and it always run in private mode to keep it clean).

SOCKS5 proxy works pretty good for me but I was curious if it's possible to use wireguard for my usecase - to route Firefox/IceCat traffic through VPN keeping all other traffic as is.

For domain name resolution I'm using locally installed Unbound server which is bound to localhost. My /etc/resolv.conf file looks like this:

nameserver 127.0.0.1

It is configured to forward requests to Quad9 resolver (no Google, no Cloudflare, right) but I'm planning to turn forwarding off and just use Unbound on its own.

There are numerous posts in the Internet covering wireguard configuration so I'll just show my server/client configuration with some notes.

For some reason I've missed the fact that wg-quick helper uses advanced version of wg configuration file syntax which means wg-quick configuration file can't be used with wg. I've lost some time while battling this issue.

Summary of the settings:

10.10.10.0/24 - VPN network
10.10.10.1/32 - VPN server address
10.10.10.10/32 - VPN client address
55000 - VPN server port wireguard will be listening on
VPN server will route all traffic coming from client to the outside world

Server ¶

My server configuration file looks like this:

# /etc/wireguard/wg0.conf on the server

[Interface]
PrivateKey = <private key of the server>
ListenPort = 55000

[Peer]
PublicKey = <public key of the client>
PresharedKey = <preshared key>
AllowedIPs = 10.10.10.10/32

I use Alpine Linux on the server for my experiments with wireguard and my wg0 interface is defined like this:

# excerpt from /etc/network/interfaces

auto wg0
iface wg0 inet static
  address 10.10.10.1
  netmask 255.255.255.0
  pre-up ip link add dev wg0 type wireguard
  pre-up wg setconf wg0 /etc/wireguard/wg0.conf
  pre-down ip link delete dev wg0

One more player here is firewall (iptables in my case). It must be configured to allow traffic coming from wg0 interface to be forwarded to outside world.

Something like this should help you to get started:

$ sudo iptables -A FORWARD -i wg0 -o eth0 -j ACCEPT
$ sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
$ sudo iptables -t nat -A POSTROUTING -o eth0 -s 10.10.10.0/24 -j MASQUERADE

Don't forget to make these rules permanent and allow forwarding on the server using sysctl:

net.ipv4.ip_forward=1

That's it for the server.

Client ¶

Client configuration file looks like this:

# /etc/wireguard/wg0.conf on the client

[Interface]
PrivateKey = <private key of the client>

[Peer]
PublicKey = <public key of the server>
PresharedKey = <preshared key>
Endpoint = <server public ip>:55000
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25

I've faced another issue while configuring the client - something absolutely unexpected when pinging any host, even VPN server endpoint:

$ ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
From 10.10.10.10 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Required key not available
From 10.10.10.10 icmp_seq=2 Destination Host Unreachable
ping: sendmsg: Required key not available
From 10.10.10.10 icmp_seq=3 Destination Host Unreachable
ping: sendmsg: Required key not available
^C
--- 10.10.10.1 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2028ms

The same for pinging Google DNS:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 10.10.10.10 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Required key not available
From 10.10.10.10 icmp_seq=2 Destination Host Unreachable
ping: sendmsg: Required key not available
From 10.10.10.10 icmp_seq=3 Destination Host Unreachable
ping: sendmsg: Required key not available
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2037ms

The reason for this was AllowedIPs misconfiguration:

# This is wrong setting
AllowedIPs = 0.0.0.0

I've somehow lost CIDR part from the address which means address will look like 0.0.0.0/32 which is absolutely wrong.

Pay attention to wg output, to allowed ips value especially:

$ sudo wg
interface: wg0
  public key: <client public key>
  private key: (hidden)
  listening port: 52312

peer: <server public key>
  preshared key: (hidden)
  endpoint: <server ip address>:55000
  allowed ips: 0.0.0.0/32
  latest handshake: 1 minute, 12 seconds ago
  transfer: 644 B received, 2.23 KiB sent
  persistent keepalive: every 25 seconds

So, correct value for AllowedIPs should look like this:

# This is correct setting
AllowedIPs = 0.0.0.0/0

And now we've come to the most interesting part of the show - to client-side network configuration.

The solution for my usecase was here - simple script by (probably) wireguard author. Thank you, Jason!

After some experiments I've taken an opposite approach from the original - prepared wg0 interface is moved to vpn namespace and is configured to be the default route within this namespace.

We can check what interfaces our new vpn namespace has and how routing table looks like:

$ vpn up
...
$ vpn exec ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
10: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none
...
$ vpn exec ip route
default dev wg0 scope link
10.10.10.0/24 dev wg0 proto kernel scope link src 10.10.10.10

Pretty good, right?

One more minor issue was the lack of domain name resolution for programs running within vpn namespace. I was not expecting this so it took some time to understand the reason.

Citing ip-netns manual:

A network namespace is logically another copy of the network stack, with its own routes, firewall rules, and network devices.

This means nobody is listening on 127.0.0.1:53 within vpn namespace. We can easily solve this issue by starting another copy of unbound within vpn namespace (I've decided to use separate copy of unbound.conf for this named unbound.vpn.conf, YMMV).

Here is the script I'm using to manage VPN:

#!/usr/bin/env bash

set -ex

[[ $UID != 0 ]] && exec sudo -E "$(readlink -f "$0")" "$@"

NS="vpn"
WGIF="wg0"
WGCONF="/etc/wireguard/$WGIF.conf"
WGADDRIPV4="10.10.10.10/24"
UNBOUNDCONF="/etc/unbound/unbound.$NS.conf"
UNBOUNDPID="/run/unbound.$NS.pid"

up() {
    ip netns add $NS
    ip link add $WGIF type wireguard
    wg setconf $WGIF $WGCONF
    ip link set $WGIF netns $NS
    ip -n $NS addr add $WGADDRIPV4 dev $WGIF
    ip -n $NS link set lo up
    ip -n $NS link set $WGIF up
    ip -n $NS route add default dev $WGIF
    ip netns exec $NS unbound -c $UNBOUNDCONF
}

down() {
    UPID=`cat $UNBOUNDPID`
    kill $UPID || true
    ip -n $NS link set $WGIF down
    ip -n $NS link del $WGIF
    ip netns del $NS
}

status() {
    exec ip netns exec $NS wg
}

execi() {
    exec ip netns exec $NS sudo -E -u \#${SUDO_UID:-$(id -u)} -g \#${SUDO_GID:-$(id -g)} -- "$@"
}

command="$1"
shift

case "$command" in
    up) up "$@" ;;
    down) down "$@" ;;
    exec) execi "$@" ;;
    status) status "$@" ;;
    *) echo "Usage: $0 up|down|exec" >&2; exit 1 ;;
esac

Now I can start Firefox/IceCat within vpn namespace and be sure all traffic is routed through VPN:

$ vpn exec icecat

I'm using netctl to manage my network interfaces and the plan is to use netctl hooks to bring VPN interface up or down as needed.

wireguard looks quite good (one more time - thank you, Jason A. Donenfeld!) and I'm planning to use it for some time to decide if I can switch from SOCKS5 proxy to VPN.

Zola

2026-07-26T12:23:00+00:00

Zola ¶

Note

Update. I've decided to switch back to wt. It was rebuilt from scratch and released as 3.0.0 version.

I've decided to switch from wt (which is archived now) to zola to manage this blog.

wt worked quite well for me but it requires time to manage it and to develop it further (and I was not happy with yaml).

There are a lot of alternatives in this area and by occasion I've met zola. Luckily, it has most of the features I was using with wt (actually, it was the reason to switch to zola and to archive wt).

It took some time to understand the way zola works - mostly to convert templates from jinja to tera and to update front-matter from yaml to toml format within markdown files.

I was hit by just one issue while converting this blog from wt to zola.

It was not clear to me that there is no need to apply markdown filter to content in templates as it was applied implicitly. Applying it for a second time breaks syntax highlighting formatting which led me to a thought that I can't use zola.

Happily, I've looked through the zola's issues and found this one. There is no need to apply markdown filter to content in templates. Yay. Perfect.

Let's see how it will work for me.

Stay tuned and take care of you and your family.

Markdown preview (Rust version)

2020-05-05T07:38:00+00:00

Markdown preview written in Rust ¶

About a year or so ago I've switched to self-made python script to preview markdown file in browser (this commit introduced it in my own dotfiles repository).

It was using pandoc under the hood to convert markdown to html. Well, it was not optimal, probably, but it worked.

I've started to read Rust documentation recently and decided to (re)implement markdown preview in Rust.

The most valuable thing I saw in this was the fact that it will be single binary to use - no third-party dependencies would be needed to install.

So, here is the code - markdown preview written in Rust.

This is my first project written in Rust and it is definitely not as idiomatic as it should be and has no tests at the moment but it works as expected. Yay.

Under the hood mdpreview uses following Rust libraries:

clap - to process command-line arguments,
pulldown-cmark - to convert markdown to html,
hyper and tokio - web server to serve generated html.

Some ideas for the code I've got looking through sources of zola.

Ownership is a new concept to me and I'm still not sure I'm getting it right but Rust is definitely worth trying. Here is the list of resources I'm using to study Rust (just for the record):

That's it for the mdpreview intro, let's move on and (re)implement in Rust something else. Stay tuned.