Oh, metaclass!

Looking at a friend’s python code which have tens of lines of metaclass. I thought I know __metaclass__ which is how Django implemented its database model. But then I rethink maybe I don’t know it very well.

Google showed me What are metaclasses in Python?

Oh, it’s a great answer! It’s using the “refactoring to patterns” style, telling all the magic behind it with a reasoning process. It tells you how to fish!

To me the Python3 metaclass is new, but makes a lot of sense. I’m glad I found it.

Silly memory modules issue on my iMac

I have an iMac 27″ (2017), which shipped with 8GB memory (2x DDR 4 2400 8GB). I upgraded it to 32 GB (4x DDR4 2400 8GB modules, Patriot Viper) when I got the Mac, and used it for 2 years without any problems.

2 years ago, I saw a deal of Crucial 32GB (2x DDR4 2666 16GB modules). I thought that I can install them into this iMac and then repurpose the Patriot Viper memory modules for my Intel NUC mini-PC. I remembered removing the 4 modules form the iMac is pretty hard, there’s no way to grab them firmly. But other than that I remember the replacement is success and the iMac is stable for a while (greater than 6 months).

And then for almost a year the iMac starts to develop stability issues. It will restart abruptly (showing different kind of kernal panic), showing four languages what not. I was thinking maybe the fans are fault or maybe the mainboard, graphic cards developed age issue. I also suspect that the Mac OS have too much garbage in the libraries folder.

And then couple of months ago the Montery upgrade breaks the APFS partition, leaves a huge “black whole” of disk space. There was 512GB SSD but visible capacity is about 200GB. It turns out that’s some weird APFS hidden volume bug during OS upgrade, which may related to iPhoto’s photo caches. That’s OK, I can do time machine and partition from scratch to address that issue. But that addressed the disk issue, but the system is still restarting every other day.

One day, I thought I should try to replace the memory modules, because that might be in fault with all different kind of kernal panics. Maybe the older iMac is not suppose to run the memory at 2666Mhz, that might be a issue.

Then I pulled the 4x 2400 modules, pushed them to the iMac, restart, no boot. Then I pull 2x of them out, leave 1 & 3 modules in. And then the iMac boot flawlessly (note, you need to wait about 20 seconds before the iMac can boot when you press the power button). Then I run couple of stress tests (stressng), it’s pretty stable. I run the system for couple of days and it stays stable! So maybe my hypnosis is right and I should use 2400 modules.

Then I tried to push the 32GB 2666 modules in, mixed with 16GB of 2400 modules. That won’t boot. Try 2 & 4 modules with the 16GB modules, won’t boot. Try 4 modules of 8GB modules again, won’t boot. And then to make sure I captured the ghost I installed the 32GB 2666 modules at 1 & 3 again, oops, it booted. And then for couple of days, it’s stable.

OK, I think the problem is resolved. But I have no idea why! I seems a year of stability issue might be some electric conduction issue? Anyway, it’s hard to “reproduce” this kind of issue sometimes. So the IT solution of “kick it and restart” might works.

Simple design is zen

Dan North (a.k.a. the BDD guy) tweets:

SOLID is well known Object-Oriented Design principles, Uncle Bob introduced this acronym in his book. If someone ask you about design principles in a technical interview, this probably is the most sound answer. But I totally get Dan’s point, those acronym is cool but hard to apply in practice. I mean everyone who is good at code know what is good design, and those principles does apply well. But in most of time, this is like a review of stock market. It make sense, but you can’t use the review to predict the market. I feel they are confirmation bias.

Please take a look on Dan’s deck “Why Every Element of SOLID is Wrong”, it’s humorous. His point is “Write Simple Code”, which I dig. This is a way to say don’t trust prediction of stock market, use your common sense. It’s not because any simple code is good code, it is just saying that you should “Chop Wood Carry Water”.

The goal of Zen, his master taught him, was to “archive a void … noiseless, colorless, heatless void” – to get to that state of emptiness, whether it was on the mound or in the batter’s box or at practice.

Before that, Zhuang Zhou, the Chinese philosopher, said, “Tao is in the emptiness. Emptiness is the fast of the mind.”

Quote from the book Stillness Is the Key

I feel the “Write Simple Code” tag line is in the spirit of zen. Always want to apply best practice, design principles feels like ego. We have an impression those pattern matching process make perfect code. But the order is wrong. We can refactor code to use patterns, but pay upfront cost to use pattern may be a waste. If we can put down our ego, we can just go back to our emptiness and write simple code. Make It Work Make It Right Make It Fast. Go back to basic, Chop Wood, Carry Water, Chop Wood, Chary Water.

rMBP kernal_task cpu spikes when connecting more than 1 external monitors

rMBP kernal_task cpu spikes when connecting more than 1 external monitors

This “bug” bother me frequently, and I don’t know why? I didn’t thought that’s related to connecting more than 1 external monitors, so I always search for “kernel_task high cpu”, which leads to similar fix than what I found lately.

TL;DR;

  • If this problem is highly correlated with connecting to more than 1 monitor and you don’t really need that much monitor. Then the easy fix is throw 1 of your monitor for other task, and working with only 1 external monitor.
  • There’s a scientific way to figure out which kernel extension cause the issue. If you like to fix it in this please read following text.

So there’s some falty kernel extension which trick the kernel_task, and kernel_task try to steal CPU time to cool CPU down. Which make the system very un-responsive. This issue normally took couple of hours of my time, I can’t get back to productivity.

I thought those 3 things may fix it:

But those method is like Voodoo 🙂 I even tweeted SMC reset fix my problem, but it turns out not true.

Until I finally found the issue is correlated of connecting to more than 1 monitors. Yes, do not connect to external monitor or connecting to only 1 monitor fix the problem. And my daily setup is connecting the rMBP (MacBookPro11,3, MacBook Pro Retina, 15-inch, Late 2013) to 2 monitors: 24 Inches Dell and 30 Inches Dell. I found unplug the monitor fix the kernal_task CPU usage issue in couple of minutes.

So I change my google keyword, and find the better answer immediately.

Those article actually is mostly overlap with How to fix kernel_task CPU usage on El Capitan. The difference is it tells you that you need to “poke each of the kext to figure out which one is the lemon”. This make it more like a science 🙂

Mac have serious software quality issue. But their intention is good, you can seamlessly upgrade from major OS versions. It doesn’t bother about “Drivers update” like windows (although windows 10 addressed the driver issues pretty much already). But it doesn’t mean you will load wrong thing into your kernel extension which ultimately cause weird issue. Apple also have bad knowledge base, you can’t find anything useful on their “wiki”. But noise in user forum, and Apple so very bad at indexing them.

OK, rant finish. To me this is what I learned:

  • Be careful to listen to the problem. Try to understand the symptom better.
  • Avoid voodoo fix.
  • When you find low quality result, change your search keyword. Even sometime it’s hard to craft better keyword. But the keyword is the key in search engine era.

And how to fix this issue on your macbook?

  • See if it’s related to connecting to more than 1 monitors. Because if unplug solve the problem, then you don’t need to fix it when there’s other high priority tasks.
  • Perform SMC + PRAM reset. This is almost free.
  • Disable SIP, and re-enable it when you fix it.
  • Disable the kext 1 by 1 as [Technology] kernel_task consumes almost 100% of CPU on Mac OS X suggests. Don’t follow other instruction to delete your exact mac’s plist. Just disable that kext as whole, which is easier to recover and save your google time too.
  • Make sure you Time Machine your Mac.

Finally, I need to say OS X breaks so often recently. And maybe I should say it never works correctly, it almost always accidentally barely works. Those components breaks for me frequently:

  • kernel_task, that what I complains above.
  • Keychain is broken frequently. And the keychain first aid is gone.
  • Disk integrity and permission breaks frequently, and you need to restart your computer to recovery mode to check/fix.
  • Spotlight breaks frequently.
  • The network (and VPN) is not as stable as before. But this might be a false feeling due to lack of trust.
  • Updates no longer always make the OS better.

Dan Abramov’s Redux lessons are great

Screen Shot 2015-12-29 at 4.56.29 PM

My awesome colleague Dustan Kasten recommend Dan Abramov‘s Getting Started with Redux on egghead.io.

It includes 30 short video lessons, which are great example of Refactoring to Patterns. Although there’s not a goal to form any patterns. The goal is just making sense in coding. The process of refactoring make me feel exciting. Nowdays the javascript community is full a micro-libraries, like the react-redux. This video actually shows the intention of refactoring and extract reuseful code as tiny library.

Thanks Dan and egghead.io!

Why I prefer using new/prototype/this to ‘createClass’?

Here is my thoughts on “factory” vs. “new/class”. First I agree that class is not necessary when we use prototype system in javascript. And prototype is superior than class, that’s why we love javascript.

But when we design a system, we need some tool to help use minify the side effects between your internal API calls. Making zero side effects system is possible, but that doesn’t make too much sense. Closure is a important feature in javascript, the closure is mutable, we use that commonly to reserve state in function. This is not pure functions anymore, but that’s an sweet spot in the middle of the spectrum from pure functional programming to none-pure functional programming. So the rules of thumb here is localize the side effects, and make the side effects physically close to functions. That’s one style we use to describe how Object Oriented Design marry with functional programming happily.

The previous chapter help us explain why the prototype is good. We can use the object to localize their local states (side effects), and the prototype is the functions which apply side effects on them. This style help us get rid of the dirts from classical OO’s class system. So here is a new question, do you think the class and new harmful in javascript? My answer is NO.

So the intention of the functional and localized side effects is a goodwill, and we should think about what’s the right tool/pattern for us to achieve that? And there’re 2 common patterns in javascript to achieve that.

  1. Factory:
var User = function() {
    var privateState = {}

    var setPrivateState = function(value1) {
        privateState.state1 = value1;
    }

    return {
        publicMethod: function(value1) {
            setPrivateState(value1);
            this.otherPublicMethod();
        },
        otherPublicMethod: function() {}
    }
}
var user = User();
  1. Use function, prototype and new:
var User = function constructor() {
    this.privateState = {}
}

User.prototype = Object.create({
    _setPrivateState: function(value1) {
        this.privateState.state1 = value1;
    },
    otherPublicMethod: function() {}
});

User.prototype.publicMethod = function(value1) {
    this._setPrivateState(value1);
    this.otherPublicMethod();
}

var user = new User();

The Factory use closure simulate private methods and variable, the reference of this is inexplicit reference the returned object literal itself. The magical stuff is that the Factory way don’t need the new keyword, which is the reason why someone love it.

The prototype way lost the private method/variable (you can do that by define some private method inside the constructor, but let’s put that aside). But the benefit is explicit this binding in new, and having a prototype chain in new (that’s how the prototype chain works). Although prototype chain is commonly used for simulating inherit (which is bad), but we can also doing mixin by prototype (which shares all methods on prototype; in comparison, mixing is normally done as method copy between objects in Factory way).

Because the prototype is a killer feature IMHO, so I lean towards using the new/prototype/class pattern. I don’t have strong opinion on the keyword new (because I have some friends don’t like it), but let’s review what’s new is doing?

  1. A new object is created, inheriting from foo.prototype.
  2. The constructor function foo is called with the specified arguments and this bound to the newly created object. new foo is equivalent to new foo(), i.e. if no argument list is specified, foo is called without arguments.
  3. The object returned by the constructor function becomes the result of the whole new expression. If the constructor function doesn’t explicitly return an object, the object created in step 1 is used instead. (Normally constructors don’t return a value, but they can choose to do so if they want to override the normal object creation process.)

So I think new is still quite useful 🙂

Going back to the implementation side. Using the new/prototype/class is my choice, but there’s some drawback. The prototype is very flexible, and we may misuse it. I’m especially against using that for inheritance reuse. Because most people believe composition over inheritance (for reuse).

So what I need?

  • Make clear that we need a constructor for a object factory (I don’t call it class)
  • We define a set of own methods for that object, and they should be assign into constructor’s prototype
  • We’d like to use mixin (on prototype) to reuse

And probably you know that React is hot in our community. And react have a method React.createClass, which is doing exactly that 3 thing I describe above. It’s like a factory of object factory, put some restriction to you, but showing you a schema of a object factory. I like it. But you don’t need React to use createClass, you can do it with couple of lines of code.

var _ = require(‘underscore’);

var createClass = function(options) {
    options = options || {};

    var constructor = options.hasOwnProperty(‘constructor’) ? options.constructor : (function() {});
    delete options.constructor;
    var mixins = options.mixins;
    delete options.mixins;

    options.mixin = _.mixin;
    constructor.prototype = options;

    if (mixins) {
        mixins.forEach(function(mixin) {
            _.extend(constructor.prototype, mixin);
        });
    }

    return constructor;
}

I used methods in underscore, you can copy them out and have a standalone createClass.

How about class support in coffeescript and ES6? I’m a believer of both. I use coffeescript for years, and I love it’s class implementation. The class support doesn’t give you mixin out of box, because mixin is a personal choice. So the class is just a factory of object factory, which is same as what we introduce above.

The how is not very important in this post, because you can write your own (better) implementation easily. The more important is why we doing this, the core is adopt functional and localized side effects design style.

SQLAlchemy gevent Mysql python drivers comparison

sqlalchemy-gevent-mysql-drivers-comparison

Compare different mysql drivers working with SQLAlchemy and gevent, see if they support cooperative multitasking using coroutines.

The main purpose of this test is to find which mysql driver has best concurrency performance when we use SQLAlchemy and gevent together.
So we won’t test umysql, which isn’t compatible with DBAPI 2.0.

Code is here. Thank you CMGS.

Result example

100 sessions, 20 concurrency, each session take 0.5 seconds (by select sleep)

mysql://root:@localhost:3306/mysql_drivers_test total 100 (20) 50.5239 seconds
mysql+pymysql://root:@localhost:3306/mysql_drivers_test total 100 (20) 2.6847 seconds
mysql+oursql://root:@localhost:3306/mysql_drivers_test total 100 (20) 50.4289 seconds
mysql+mysqlconnector://root:@localhost:3306/mysql_drivers_test total 100 (20) 2.6682 seconds

With greenify ptached MySQLdb (mysql-python).

mysql://root:@localhost:3306/mysql_drivers_test total 100 (20) 2.5790 seconds
mysql+pymysql://root:@localhost:3306/mysql_drivers_test total 100 (20) 2.6618 seconds
mysql+oursql://root:@localhost:3306/mysql_drivers_test total 100 (20) 50.4437 seconds
mysql+mysqlconnector://root:@localhost:3306/mysql_drivers_test total 100 (20) 2.6340 seconds

Pure python driver support gevent’s monkey patch, so they support cooperative multitasking using coroutines.
That means the main thread won’t be block by MySQL calls when you use PyMySQL or mysql-connector-python.

1000 sessions, 100 concurrency, each session only have 1 insert and 1 select

mysql://root:@localhost:3306/mysql_drivers_test total 1000 (100) 10.1098 seconds
mysql+pymysql://root:@localhost:3306/mysql_drivers_test total 1000 (100) 26.8285 seconds
mysql+oursql://root:@localhost:3306/mysql_drivers_test total 1000 (100) 6.4626 seconds
mysql+mysqlconnector://root:@localhost:3306/mysql_drivers_test total 1000 (100) 22.4569 seconds

Oursql is faster than MySQLdb in this case.
In pure python driver, mysql-connector-python is a bit faster than PyMySQL.
use greenify or not won’t affect the testing result in this user scenario.

Setup

mkvirtualenv mysql_drivers_test # workon mysql_drivers_test
pip install --upgrade setuptools pip cython
pip install -r requirements.txt
python mysql_drivers_comparison.py

Test with greenify MySQL-python (on OSX with homebrew).

mkvirtualenv mysql_drivers_test # workon mysql_drivers_test
pip install --upgrade setuptools pip cython
pip install -r requirements.txt
git clone https://github.com/CMGS/greenify
cd greenify
cmake -G 'Unix Makefiles' -D CMAKE_INSTALL_PREFIX=$VIRTUAL_ENV CMakeLists.txt
make & make install
cd ..
export LIBGREENIFY_PREFIX=$VIRTUAL_ENV
pip install git+git://github.com/CMGS/greenify.git#egg=greenify # if you are using zsh, use \#egg=greenify
git clone https://github.com/CMGS/mysql-connector-c
cd mysql-connector-c
export DYLD_LIBRARY_PATH=$VIRTUAL_ENV/lib
cmake -G 'Unix Makefiles' -D GREENIFY_INCLUDE_DIR=$VIRTUAL_ENV/include -D GREENIFY_LIB_DIR=$VIRTUAL_ENV/lib -D WITH_GREENIFY=1 -D CMAKE_INSTALL_PREFIX=$VIRTUAL_ENV CMakeLists.txt
make & make install
cd ..
git clone https://github.com/CMGS/MySQL-python.git
cd MySQL-python
export DYLD_LIBRARY_PATH=$VIRTUAL_ENV/lib
export LIBRARY_DIRS=$VIRTUAL_ENV/lib
export INCLUDE_DIRS=$VIRTUAL_ENV/include
unlink /usr/local/lib/libmysqlclient.18.dylib
ln -s $VIRTUAL_ENV/lib/libmysql.16.dylib /usr/local/lib/libmysqlclient.18.dylib
python setup.py install
brew switch mysql [version] # brew switch mysql 5.6.15 on my env, brew info mysql to check which version is available on your env
cd ..
python mysql_drivers_comparison.py

If the greenify doesn’t work for you, you can use otool -L _mysql.so in your $VIRTUAL_ENV/lib/python2.7/site-packages MySQL-python folder. Can’t find otool even after you installed XCode’s command line tools? Follow this link.

I need say thank you to CMGS. He guided me how to install greenify and how it works, he also help me triage the issues I met (include the otool part). Make greenify and mysql work on OSX make no sense, you shold do it on your application server which probably will be a linux, hope you will figure out how to.

ThoughtWorks 将于 2014 年 3 月 8 日举行的第十一届 B’QConf (北京软件质量大会)

本 Post 是代朋友转发。

亲爱的软件业朋友们:

第十届B’QConf(北京软件质量大会)是否还在你的脑海回旋,爆棚的场面和激烈的讨论是否始终让你记忆犹新?

由ThoughtWorks中国和阿里巴巴集团共同主办,中国软件测试经理联盟协办的第十一届 B’QConf(北京软件质量大会)将于2014年03月08日如期举行。B’QConf 邀请IT测试及质量相关人员共聚一堂,一起分享经验,探讨话题,结识业界朋友。无论你是B’QConf的老伙伴,新伙伴,还是大伙伴,小伙伴,只要你热爱测试,关注质量,期待学习,喜欢分享,都欢迎加入我们的活动。

本期精彩话题:

  • 话题一:中小型项目的web性能测试模型

中小型项目的性能测试和大型项目不同。在时间和资源都有限的情况下,我们来聊聊如何选定性能测试指标,怎样设计性能测试用例,最终出一个什么样的结果,才对项目最有价值。

演讲嘉宾:孙弘 ThoughtWorks QA

  • 话题二:Android自动化测试实践

本分享将介绍针对Android手机app测试方面的一套自动化测试框架,该框架能解决一般Android系统上的自动化问题,原理以及使用方法。同时会根据结合示例讲解。

演讲嘉宾:赵婉萍(菁菁)阿里妈妈-无线

  • 话题三:虚拟机初探

日常工作中,对于多个不同的环境有需求,拥有多个物理机也不太现实。我们尝试利用虚拟化技术来解决这一问题。在一些调研之后,决定使用KVM来定义、管理我们的虚拟机,本话题会分享一些KVM上对于虚拟机的定义、配置和性能方面的一些经验。

演讲嘉宾:杨锐 ThoughtWorks QA

  • 话题四:大数据的质量监控和保障

分享将介绍大数据时代下分布式系统的监控和数据仓库,数据挖掘的测试方法和原理。

演讲嘉宾:李春元(春元)阿里妈妈-CNZZ

  • 报名:我要报名第十一届B’QConf
  • 时间:2014年 3 月 8 日 13:30 – 17:30 (现场提供茶点,水果,软饮。)
  • 地点:北京市东城区东直门南大街3号国华投资大厦11层1105室 (ThoughtWorks 北京办公室)

地铁路线: 2号线东直门站D口出,来福士商场旁边国华投资大厦

公交路线: 106,107,117,24,612,635等20多条公交车到东直门内/东直门外/东直门枢纽总站下车都可
自驾路线: 东直门桥西南角来福士商场旁边国华投资大厦

更多消息请大家关注 B’QConf的微博

Places to find essential new things

Had a discussion with friends this morning, about how to explore new things, learn new staff. Then I got a list for myself:

  • new.me digest email (daily)
  • zhihu.com and quora.com weekly digest email
  • HN (Hacker news), best realtime tech world aggregation
  • Github trends, place to find intresting ideas, or high quality projects
  • twitter geek list (not twitter timeline, you need pick your own geek list)
  • Techmeme, technology industry headlines
  • Theverge, for high quality reviews and news post with good quality of pictures
  • Reader (digg reader, feedly, or what-ever), baseline is you need to follow intresting people here

Some of my best coworkers and friends contribute those:

  • Lifehacker, for interesting howtos
  • ScienceDaily, Latest Science News
  • reddit, everything fun
  • google plus, depends on your circle quality and time you spend
  • Engadget

Compae redis and memcache as cache in python

I hate posting micro benchmark of library, it’s kinda stupid when they are not the bottle-neck. Now, I will hate my self 😀

The verdit is: using memcache and redis has similar performance. In our system, we use redis a fast data-structure store, so use it as cache save a moving parts.

  • cPickle 50000 rounds, used 1.61123394966
  • msgpack 50000 rounds, used 0.296160936356
  • memcached with pylibmc save/load 15000 rounds, used 1.68719887733
  • memcached python-memcached save/load 15000 rounds, used 3.92766404152
  • redis with hiredis save/load 15000 rounds, used 2.76974511147

Acctually we are using redis + msgpack now.

By the way, our new leaderboard service is based on gevent patched flask + redis + sqlalchemy tech stack. A aws 8 core instance give us 900rps, 1.5x throughput of a ruby version (with fiber and em) write by my ex-coworker @flyerhzm.