Random Walk of Life

It’s been some time since I last posted something. Had been busy gathering data for the OLS paper and once that was done, got some time to look into the couple of months old CPU-Hotplug issue. There are some more items to be done there, hopefully will get some free time this month.

Read this post earlier today:
http://contentconsumer.wordpress.com/2008/04/27/is-ubuntu-useable-enough-for-my-girlfriend/

How easily people can confuse monopoly with standards.

When someone says Linux is not user friendly, I ask “Compared to what?” and they say “Windows”. My question is, what if someone who has worked on Linux all his life was shown a windows box? How will he/she feel about it?

Given the fact that number of schools today, in the name of “Computer Education” bias the young and impressionable minds by teaching them the basics of Microsoft this-or-that product and tell them, “Children, Windows is an operating system. Operating system is windows”, how justified are we in blaming everything to be a problem with Linux? I agree, there are a number of issues that can be improved in Linux. Whether it is the kernel, or the applications, there’s always that scope. But accusing that “Linux is bad because it does not behave like windows” is a stupid thing to do.

To give you an example, I have a young nephew who is not that much exposed to computers. His only knowledge of a computer is restricted to “Computer has a monitor, a processing unit, a keyboard and mouse”.

The last time he came home, I was working on my home PC which has Fedora 8 on it. He wanted to see how the “Computer” works. I showed him a couple of things. Simple things such How to use the calculator with different base systems, use Firefox to browse the internet, what games are available. And yes, I showed him Compiz Fusion. He spent the rest of the time browsing through my pictures and videos. Didn’t call me for help. Not even once.

Next time I met him, he told me about this “computer with a My Computer on the desktop” that his friend had. He told me how stupid was this browser with no tabbed features on it. How everytime he wanted to do something, it asked him thousand times “Are you sure you wanna do this?” and how irritating the whole experience was.

So, that brings me back to the basic question, does it make any sense in comparing two different operating systems, when you are already have a biased opinion in favor of one of them?! How can you expect such a comparison to be objective? Is the way to make Linux operating system a more user friendly one by mimicking what windows does, or by educating people about Linux?

I guess, these are things one needs to understand before doing Experiments on the girlfriend or grandma.

PS: One of my friends wanted to connect her windows laptop to my Hathway modem. When things didn’t work, she insisted on rebooting the modem. On doing that, things started working! Hmm.. must be a Bug Feature. Because on Linux, I just need to set the static IP using my Network Manager! It doesn’t ask me to reboot the modem

While working on idle power management, I used vaidy’s klog based patches to profile an idle system to obtain stats such as:

* The time when a cpu enters into the tickless idle mode
* The various interrupts that bring the cpu out of idle state.
* The timers that expire in this interval and cause a wake up.
* The tasks that demand/ or are made to be wake up on the idle cpu.
* The time when the cpu comes out of the tickless idle mode and starts executing the tasks.

While observing the task wakeup instrumentation data, I noticed that the wakeup statistics for kondemand appeared pretty strange. For the uninitiated, kondemand is a kernel thread that belongs to the ondemand governor of cpufreq subsystem, which changes the p-states of the system, based on the utilization statistics. Thus it’s something that helps in power management.

root@llm43 tests]# ps aux | grep kondemand | head -4
root 1143 0.0 0.0 0 0 ? S< 09:21 0:00 [kondemand/0]
root 1145 0.0 0.0 0 0 ? S< 09:21 0:00 [kondemand/1]
root 1146 0.0 0.0 0 0 ? S< 09:21 0:00 [kondemand/2]
root 1147 0.0 0.0 0 0 ? S< 09:21 0:00 [kondemand/3]

From the file wakeups.txt, an output of my profiling experiment,

pid cpu nr_wakeups
————————–
1143 0 468
1145 1 279
1146 2 78
1147 3 68

Couple of things bothered me here.

1. The unusually high number of wakeups on CPU0 and CPU1. kondemand was wakeing up approximately at the rate of 4 time and 2 times respectively on these cpu’s over a observation idle period of 120 seconds.
2. The difference in the number of wakeups by kondemand on the different CPUs.

Bewildered, I fired a mail to Venki asking for possible explanations.
And I started looking at the code. Now, the number of times the kondemand thread is supposed to check for a change in the frequency is determined by this sysfs tunable called sampling_rate. It was set to 256000us on my system. Which accounted for the unusually high number of wakeups on the CPUs.

But I was still confused. The sampling_rate is a global tunable which maps to the variable dbs_tuners_ins.sampling rate, which is common to all the kondemand threads. Then why the different wakeup rates on different CPUs?

Venki replied to my original query reminding me that kondemand uses deferrable timers! That explained everything.

Deferrable timers, behave normally on a busy system. But on a idle system, when are about to decide when should we wake up next inorder to service the next timer in the list, we skip any deferrable timer we encounter.Thus, a deferrable timer on an idle cpu would expire when the next nearest *hard* timer would expire.

So, the reason why CPU0 and CPU1 were having such high number of wake ups on an idle system can be accounted to the fact the expiry of some other timer like the ehci_watchdog would trigger the expiry of kondemand timer, and along with it the wakeup of the kondemand thread! And depending on the number of timers that are queued on different CPUs, we have the corresponding number of wakeups of kondemand thread!

So, what I was thinking to be the major cause for wakeups in the kernel, turned out to be an effect of the expiring timers queued by a totally unrelated subsystem, thus confirming the old wisdom of mathematical logic: “If two events occur one after the other, it doesn’t necessarily imply that one is the cause for the other”

Okay, henceforth decided to give the opening and closing mood a break!

Anyway, I am currently reading the Linux scheduler load balancing code. And there’s this one function called find_busiest_group() which looks something like this:

/*
 * find_busiest_group finds and returns the busiest CPU group within the
 * domain. It calculates and returns the amount of weighted load which
 * should be moved to restore balance via the imbalance parameter.
 */
static struct sched_group *
find_busiest_group(struct sched_domain *sd, int this_cpu,
 	   unsigned long *imbalance, enum cpu_idle_type idle,
 	   int *sd_idle, cpumask_t *cpus, int *balance)
{
   struct sched_group *busiest = NULL, *this = NULL, *group = sd->groups;
   unsigned long max_load, avg_load, total_load, this_load, total_pwr;
   unsigned long max_pull;
   unsigned long busiest_load_per_task, busiest_nr_running;
   unsigned long this_load_per_task, this_nr_running;
   int load_idx, group_imb = 0;
  #if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
   int power_savings_balance = 1;
   unsigned long leader_nr_running = 0, min_load_per_task = 0;
   unsigned long min_nr_running = ULONG_MAX;
   struct sched_group *group_min = NULL, *group_leader = NULL;
  #endif

	max_load = this_load = total_load = total_pwr = 0;
        busiest_load_per_task = busiest_nr_running = 0;
 this_load_per_task = this_nr_running = 0;
 if (idle == CPU_NOT_IDLE)
 	load_idx = sd->busy_idx;
 else if (idle == CPU_NEWLY_IDLE)
 	load_idx = sd->newidle_idx;
 else
 	load_idx = sd->idle_idx;

	do {
 	unsigned long load, group_capacity, max_cpu_load, min_cpu_load;
 	int local_group;
 	int i;
 	int __group_imb = 0;
 	unsigned int balance_cpu = -1, first_idle_cpu = 0;
 	unsigned long sum_nr_running, sum_weighted_load;

Now by the time I reach this point, I am kinda confused asto what variable holds what value and represents what load exactly!
I guess, I need to increase the stack size of my mind to keep track of soo many local variables!

PS: I apologise for the formatting. The [sourcecode] macro of wordpress cannot do any better than this.
Sathya is planning to write a simple script to convert C code into color coded html instead.

Opening Mood: Hungry..
Opening song: Hamesha tum ko chaaya - Devdas

Got the official mail today morning. The “Power Aware Scheduler” Birds of Feathers Session Proposal that I had submitted for this year’s Ottawa Linux Symposium has been accepted. The rest of the BOFS can be found –> here

A couple of days back, Vaidy got confirmation that the paper “Energy-aware task and interrupt management in Linux” of which I am a co-author, has been selected! So it’s great news on the work front!!

The rest of the papers for this year’s OLS can be found –> here

OT, it has been a pretty silent day in my mailbox. I was wondering if everyone decided to take a mass-vacation or something! Only later I came to know that vger.kernel.org is down.

With only a few weeks left to turn in the paper, we should start writing soon! The data is pretty much there with the many experiments that Vaidy has been running for quite some time now.

And me, I am have started using qgit to track the CFS development from it’s inception to it’s present magnificent form! The latest qgit-4 which uses qt-4 looks pretty cool! Much better than the earlier 1.5.5 version

Until next time,
Ciao!

Closing Song: Laree Chooti - Xulfi, Ek-Chaalis ki last local
Closing Mood: Really  hungry!

Opening Mood: I’m back!
Opening Song: Romantic Piano - Chopin

Spent the last week at this place called Kudremukh. It used to be one of Karnataka’s famous mines before the operations were shut down in 2005. So now it is just a nice little place with lots of greenery and very few people to bother an occasional guest. Which was me

I started from karkala on 4th Feb at 8:00 AM to Kudremukh. I had booked a room for myself at the Sahyadri Bhavan lodge for five days through some family contacts. Reached the place at around 9:30 A.M. The ride was very good, and one could spot a lot of trees and forests on the way. Once you cross the Mala outpost, the weather becomes quite pleasant owing to the trees around.

The room where I was put up was really a big one. It had three beds, a balcony, and an attic! All for an extremely cheap then of Rs 200 per day! The guest house has a small restaurant which serves breakfast, lunch and dinner. Again, at very affordable rates. I liked the food there. But then, I also stayed for four years at NITK Surathkal and survived the mess food. So I really shouldn’t be talking about how good the food at any place is . The place also has a very nice town park where I used to go in the evenings for a stroll.

The purpose of the vacation was to do stuff which I had always wanted to do, but never got the time. Starting problem you can say . Over the past couple of years, I have been accumulating quite a bit of technical stuff to read, and the list was only growing. So I wanted to take some time off and get started on some of those items.

Thus I had five days at a place with very little disturbance and an awesome climate. Really, I couldn’t ask for more. The working style was simple. Start with one work item, when you get bored, move on to the next one. There were no other rules. No hard deadlines to finish this or that by the end of 5 days. Just have fun

So, in the five days, I read through parts of Linux Memory management. Two years of programming the kernel, and I was still not aware of how Linux manages the memory across the various platforms it supports. So I had to go back to the very basics, starting from the 386 memory management model and then how Linux represents it using a 4 level model. After that, I read up on the Zoned Buddy Allocator, the Slab Allocator and a part of Process address space. I was using “Understanding the Linux Kernel, 3ed” and 2.6.24 kernel code as reference.

Other than that, RB-trees was something I read about and implemented during this time. It reminded me a lot about AVL trees, which we had studied in our data-structures classes. RB-trees was actually easier to implement (atleast the deletion part).

Also, read the first three chapters from Ulrich Drepper’s “What every programmer must know about memory”. The paper is true to it’s name. Especially the chapter on caches. The paper gives a very clear picture of the implications of having different levels of caches, how the cache size matters, through various well designed experiments. It should be read by anyone who is interested in doing any kind of system programming.

Other than these, did a whole bunch of fun stuff, like wrote a ascii visualization tool for trees. One of the problems I had with the Data structures lab assignement was that we had to show the output through an inorder, or a breadth order traversal. And that was not always very easy to visualize. But then I was one of those who did the class assignments when there was only one or two days left for the deadline. So I guess I am doing all the supplemental things now

Oh yeah, spent a whole evening reverse engineering this obfuscated C Code which prints the poem about the various gifts given by the true love on each day of Christmas. Am planning on writing a small article on that. How recursion was used in place of loops and how the ‘,’ operator was used for statements. It’s a code when you see, you’ll go “This hurts my brain!” . Check it out here.

And when I had time to spare, I read some chapters from Jon Bentley’s “Programming Pearls”. Another book which I would recommend to anyone interested in serious programming. This book sure is fun!

So, now that I am back, the first thing to do would be to catchup with a week of email. Oh well

Closing Song: Marriage of Figaro - Mozart.
Closing Mood: Anxious to catch up with last week’s happenings.

Opening Mood: Good Morning.
Opening Song: Boondein - Silk Route

The last and final Day of FOSS.in had some really interesting talks.

I missed the talk by Andrew Cowie, since we reached the venue pretty late. However, I managed to catch the next talk by Ulrich Drepper, the person whose article on Memory management I thought was brilliant. The talk was on “Contributing to Linux Runtime”. One of the interesting things that Ulrich mentioned was that he is looking for a new malloc implementation, one which is NUMA aware. This would really be a nice thing to work on, but I hope even applications understand the concept of NUMA so that they don’t move the threads around from one node to another.

After the lunch, we had Rusty’s talk on Talloc. Talloc stands for Tridge-Allocator, or Tree-Allocator is a hierarchial memory allocator which can be a really useful thing to prevents memory leaks owing to the programmer’s forgetfulness. There were about 150 people in a hall whose capacity was 120. I sneaked into the projector room to catch that talk and got into the hall only for the Q & A session ( I won one of those caramello koalas!). You can find the Talloc details here –> http://talloc.samba.org/

I chose to attend Amit Shah’s KVM talk over Thomas’s Real-time talk. I was hoping to see some meaty details about the KVM way of doing virtualization. But unfortunately most people in the audience had problems visualizing virtualization in the first place. So it ended up as a demo session. I guess, since the code is available, I will have to just go over it.

The concluding ceremony had “Rusty and a Merry Band of FOSS hackers” (Doesn’t that remind you of some rebel group in some corner of the Sherwood forest?) talking to the audience and sharing their experiences with open source. He showed us how easy it was for anyone to go ahead and contribute to open source projects, by asking one of the volunteers to come over and send a Documentation patch for lguest. You can find that here –> http://lkml.org/lkml/2007/12/8/49

I had to leave early to help prepare for Sunday’s barbecue. That however is another story for some other time

Closing Song: Village Damsel - Indian Ocean.
Closing Mood: Power Aware!

Opening Mood: Sleepy.
Opening Song: Walk of Life - Dire Straits.

I couldn’t attend the morning nor the afternoon session owing to some work at office. However, I managed to catch the OpenMoko talk by Harald Welte. It’s a pretty neat stuff and has got an emulator which is based on qemu which does complete hardware emulation. So you can pretty much try out the whole moko-stack on the emulator without having the handset with you. One interesting thing I realized was that OpenMoko doesn’t use cpufreq support yet. I thought that ARM did have some kind of cpu-frequency scaling support. Something reminds me of powerops. But anyway, I am planning to look into it once I get some freetime.

Next, I met with the Mozilla developer Chris Hoffman to discuss a power management related issue on Linux, when we’re running on battery. I was fooling around with powertop the other day and observed that the one application which was consistently responsible for so many wakeups in the userspace was firefox. And that too when the firefox is minimized or is in a completely different workspace than the one which is active. It did seem stupid to do that, when the whole world is worried about the rising energy costs!!! More details on that bug can be found –> here

I also met Amit Shah one of the employees of Qumranet working on KVM. Looking forward to his talk tomorrow.

And at the end of the day a whole bunch of people that included Rusty, Thomas, Harald, Rasmus Lerdorf, Amit Shah, Srivatsan, Ravi my colleagues from office and folks from ABB went out to dinner at Bombay Post. A fitting end to the day I suppose.

Closing Song: Once upon a time in the west - Dire Straits.
Closing Mood: Really sleepy. Good night!

Opening Mood: Good Morning.
Opening Song: Afterglow - INXS

Indias premier FOSS event FOSS.in kicked off on Tuesday, 4th December at the IISc campus. This year’s speakers include

Rusty Russel, Thomas Gleixner, Ulrich Drepper, Harald Welte and a few of my Colleagues Bharata B Rao and Kamalesh Babulal. I am eagerly looking forward to these talks and meeting some of these hackers. Infact Thomas had come over to our workplace yesterday and we had a fun time discussing kernel projects the whole evening followed by a sumptuous dinner!

I am expected to be at the FOSS.in for the next three days. So see you there

Closing Song: Kandisa - Indian Ocean.
Closing Mood: Gotta rush if I need to reach IISc in time for registration.

Had a busy two weeks reviving the Cpu-Hotplug effort. Two versions have already been posted and the just finished porting the third one against the latest kernel version. And that’s when I finally got around to use Git.

Git is a source code management software project started by Linus Torvalds and now maintained by Junio Hamano. There are probably a million howtos out there,on how to use Git, but I followed the one which I thought was easier for me to follow, given the nature of my job.

Kernel Hackers’ Guide to Git

I cloned the git repository and created my own branches, ported the cpu-hotplug patches. For a person who had been using quilt all this while to create and maintain patches, git is very useful. The best part is that you could actually see the whole development process since it keeps track of each commit. So for example, if I wanted to know how a particular file, say kernel/cpu.c has changed over years, I just type

git whatchanged kernel/cpu.c

and you could see every single commit that went in since version 2.6.12 something. Pretty Amazing!

Next time I am wondering why a particular ‘if’ clause exists in the code, using git, I can find out, who added it, and what was the changelog look like when it was added.

Other than that, you could also checkout a particular kernel version say 2.6.18. So instead of maintaining a whole bunch of compressed kernel source files, all I need now is a git repo and maintain different branches for different development projects that I am working on.

The other cool utility was StGit. It’s a quilt type of a utility that allows you to create new patches, push and pop. So developers already familiar with quilt can use that with atmost ease.

Hope to discover more useful features with continued usage

And here’s to a Great Weekend ahead!!

Opening Mood: Sleepy. Night out did me in.
Opening Song: Flight of Fantasy - Yanni.

Just couldn’t help laughing at the thought of this one (at the risk of being accused of thinking about nothing else other than … you’ll know what!)

Some years ago, when cpu-hotplug was first introduced, the way to postpone a cpu-hotplug operation while operating in some critical section was by using the API lock_cpu_hotplug().

Some years ago, when Kaun Banega Crorepati (KBC) was first launched with Mr. Amitabh Bacchan as the host, the famous catchline was “lock kiya jaaye” ( Let’s lock it)

A few months ago, after being fed up with the ruckus created by the global cpu-hotplug lock, Andrew Morton suggested that we kill lock_cpu_hotplug() and use freeze_processes() instead. That’s what I have been working on for the past few months.

A few months ago, when KBC version 3 was launched with Shah Rukh Khan as the host, the catchline became - “Mr Computer, Freeze kiya jaaye” (Lets freeze it)

Am I the only one finding this correlation funny?!

Closing Song: Swept away - Yanni.
Closing Mood: The sun is up. Time for me to go down!