Tag Archives: Coding

Adding Unique Constraints After the Fact in SQLAlchemy [Copy]

This post is originally from https://skien.cc/blog/2014/01/31/adding-unique-contraints-after-the-fact-in-sqlalchemy/. But the URL is throwing a 404 and I could access the page only from the Google cache. I am copying it here in case it goes missing in the future.

Update:

QGIS – Creating new column from existing using Python

Yesterday, I was working on the ward level parks map of Chennai I had to join a CSV data layer with the boundary polygon layer, but there was one issue while my CSV file has the ward numbers as integers (1,2,3..etc), the polygon layer had them as strings (Ward 1, Ward 2, Ward 3 …etc.,) So I was thinking, wouldn’t it be nice just to strip the word Ward and put it in a new column, so that I can make a join by matching the ward numbers. Turns out Python integration in QGIS is so good that, I did it without even searching the internet. Here is how.

  1. Open the Attribute table
  2. Open Field Calculator.
  3. Enter the “Output field name”
  4. Switch to “Function Editor”
  5. Click the [+] button to create a new function file.
  6. Changed the function name, parameter and return the value after stripping “Ward ” from the string. Read the docs given below the function editor to understand what’s going on the file.
QGIS Field Calculator
QGIS Field Calculator
from qgis.core import *
from qgis.gui import *

@qgsfunction(args='auto', group='Custom')
def strip_ward(name, feature, parent ):
    return name.split(" ")[-1]

Now switch back to the Expression tab and call the function to calculate the new field

strip_ward.png

Click OK. Now the new field with the computed value would be created.

I had a simple use case, by one can use the power of Python to calculate anything from existing data and generate a new field based on it. I was really blown away by the level of Python integration in QGIS.

Python Pitfalls

I was woken up today with the following question:
[python]
def foo(x=[]):
x.append(1)
return x

>>> foo()
>>> foo()
[/python]

What could be the output? The answer is

[1]
[1, 1]

I was stupefied for a minute before I started DuckDuckGo-ing Python default arguments, Python garbage collection, Python pitfalls..etc.,

These links helped me understand mutable objects’ memory management.
Deadly Bloody Serious – Default Argument Blunders
Udacity Wiki – Common Python Pitfalls
Digi Wiki – Python Garbage Collection

Thattachu – Open Source Typing Tutor

Typing tutor is a known ancient domain to work on. There are a number of places online/offline, tangible/intangible places to learn typing. But Srikanth (@logic) stumbled on a peculiar problem when worked for the Wikimedia Language Engineering team. The new age Indic input methods involved in computers seem to have no place to learn how to type on them. The only way seems to be – have a visual reference for the layout and begin typing one key at a time. This might be the most inefficient method of learning to input information. So what do we do?

Enter Thattachu

Thattachu is an open source typing tutor. It is built using the tool that Wikimedia Language Engineering Team have developed called jQuery IME. jquery.ime currently supports 62 languages and 150+ input methods. This is a JavaScript library which can be used on any web page. So we (I & Srikanth) set out to build a generic typing tutor which could employ any of the 62 languages or 150+ input methods. The project was conceived in May 2014 and was worked on only by May 2015 as I was busy with my Teach For India Fellowship. Thattachu borrows its tutor style from GNU Typist or gTypist which I used to learn touch typing in English.

Interface

Thattachu has three pages:

  1. Home page – A welcome page for those visiting the site and explaining what it is about.Thattachu_page1
  2. Course Selector – A place where you choose the course to learn. You select the language and the input method you want to learn and it lists the available courses.Thattachu_page2
  3. Workbench – A place where you practice typing. When you select a course in the Course Selector, the workbench loads with the course you selected and you can begin typing with the input method you chose. It remembers your most recent course and lesson so you can continue from where left it the previous session.Thattachu_page3

Course Structure

Each language has a set of input methods – each input method has a set of courses. The courses are classified based on their difficulty as “Beginner”, “Intermediate” and “Expert”. Each course has a set of lessons to complete and and each lesson is a collection of lines that have to be typed.

thattachu_courses

Thattachu Asiriyar

Creating the tool is the easier part of a content dependent system. The real work is generating the content that the tool can be used with. That way we faced the challenge of creating course.JSON files required for the tool. Hence a user friendly tool Thattachu Asiriyar was born.

Thattachu Asiriyar lets anyone author a course and generate a course file. If you want to author courses, go to Thattachu Asiriyar create the course file and mail it to
arun [at] arunmozhi [dot] in -mentioning “Thattachu course” in the subject.

Github savvy authors

Or if you have a Github account and know about pull requests. Kindly

  1. Fork the Thattachu repohttps://ghbtns.com/github-btn.html?user=tecoholic&repo=thattachu&type=fork&count=true
  2. Put the course file into the data/language_code folder
  3. Update the courselist.json in your folder with the metadata and the filename
  4. Send me a pull request.
  5. Feel awesome for helping the humanity learn typing

Developers

Here are a few points for those interested in the code or those who think they can improve Thattachu.

  • Thattachu is a web application written in HTML and JavaScript (AngularJS).
  • It is a completely static site with all the information stored as JSON files and served by XHR requests when requested by the Angular $http.
  • For input jQuery.ime is used.
  • It is a completely static site and can be hosted in any web server.
  • It uses localStorage of the user to track last worked on course and load it when the user opens the page next time.

Zimbalaka – Zim file creator for Offline Wikipedia

OpenZim is a Wikimedia developed format for offline reading of Wikipedia. Read more here. But the project was sadly sidelined and the support from MediaWiki, the software that runs Wikipedia sites, was also removed.

I came to know about all this from Bala Jeyaraman of Vasippu. He is planning to introduce tablets in a classroom of 6th standard students, with exceptional comprehension levels compared to average Indian classrooms, and wanted a way to load select material into the tablets. The OpenZim files have an excellent reading app called Kiwix, which also offers complete Wiki sites as downloads. Tablets can’t afford to have a huge amount of data, like full Wikipedia. There is no way to create a zim file with select topics. One has to request the OpenZim team to do it for him/her.

Enter Zimbalaka

Zimbalaka is a project which tries to solve just that. It creates offline wikipedia content files in zim file format. A person can input a list of pages that need to be created as a zim, or at least a Wikipedia category. Then Zimbalaka downloads those pages, removes all the clutter like sidebar, toolbox, edit links …etc., and gives a cleaned version as a zim file for download. It can be opened in Kiwix.

The zim is created with a simple welcome page with all the pages as a list of links. The openzim format also has an inbuilt search index and Kiwix uses this really well. So you can create zims of 100 articles and still navigate to them easily either way.

Zimbalaka has multi-lingual and multi-site support. That is, you can create a zim file from pages of any language of the 280+ existing Wikipedias, and also from sites like Wikibooks, Wiktionary, Wikiversity and such. You can even input any custom URL like (http://sub.domain.com/), Zimblaka would add (/wiki/Page_title) to it and download the pages.

It is currently hosted by my good friend Srikanth (@logic) at http://srik.me/zimbalaka

Screenshots

Here is how the content looks in Kiwix for Android.

navigate

multi

Pain points

  • A small pain point is that Zimbalaka also strips the external references that occur at the end of the Wikipedia articles, as I didn’t find it useful in an offline setup.
  • You cannot add a custom Welcome page in the zim file. Not a very big priority. The current file does its work of listing all the pages
  • You cannot include pages from multiple sites as a single zim file. The workaround is to create multiple files or use a tool called zimwriterfs, which has to be compiled from source (this is used by zimbalaka behind the scenes).

Developers

This tool is written using Flask – A simple Python web framework for the backend, Bootstrap as the frontend and uses the zimwriterfs compiled binary as the workhorse. The zimming tasks are run by Celery, which has been automated by supervisord. All the coordination and message passing happen via Redis.

Do you want to peek in how it is all done? Here is the source code [https://github.com/tecoholic/Zimbalaka]. Feel free to fork, modify and host your own instance.

Update

The OpenZim team has appreciated the effort I had put in and offered to host the tool on their server at http://zimbalaka.openzim.org. They have also pointed me to the desired backend called ‘mwoffliner’ that they have developed to download and clean the HTML. I will be working on it in my free time.

Apparix – Bookmarking in terminal

When working in a large code base like Quantum GIS or when dealing with a lot of repositories in the machine, it is always tedious to cd all the way to the folder we require to move to. Enter apparix, an excellent linux tool I found by googling “bookmarking in the terminal”. This blog post has the complete details of how to use it.

Yay, no longer cd goto/project/src/core/of/module1 and again cd ../../../test/number/three. I can simply do

$ bm projectsrc
$ bm test3
$ bm fancypants4

to bookmark my locations and simply

$ to projectsrc
$ to test3
$ to fancypants4

One more tool added in the arsenal to improve productivity.

Unit Testing with CasperJS

Today I sat down to create a JavaScript library. I wanted to do it the way I have long dreamed of – TDD (Test Driven Development). There is no dearth of Unit testing libraries and frameworks for JavaScript, so after some reading on the internet settled on CasperJS and PhantomJS combination. CasperJS is just awesome for functional testing, but Unit testing? Even though it supports Unit Testing, it as such is not a dedicated unit testing framework like Karma or Protractor. Read this for more information on TTD frameworks for JS libraries.

Loading plain JavaScript files in CasperJS for unit testing seems to be completely undocumented. I tend to think it is because it wasn’t meant to be webpage-less. But the note on docs of tester module says:

The best way to learn how to use the Tester API and see it in action is probably to have a look at CasperJS’ own test suites.

Thanks for this quote, I found that using the fs module one can load local filesystem files as modules to be used. Using that now I could write unit tests while developing the library and later on write functional tests while using the library.

Here is the file structure

library
--test
  |--unit
  |--test.js
  |--index.html
--src
  |--livetransit.js

And here is are the two tests – i) uses a webpage based approach; ii) uses module approach
https://gist.github.com/tecoholic/937f51b6889448836db8