Tech Talkies: 2010

Tuesday, December 28, 2010

1) Backup your current Firefox 3 profiles
tar -cvzf $HOME/mozilla-firefox-profiles-backup.tar.gz $HOME/.mozilla/firefox/

2) Change to root user for below actions:

3) Add repositories for downloading FireFox4 for FC14:
yum --nogpgcheck install http://rpm.livna.org/livna-release.rpm http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm

## Remi Dependency on Fedora 14 ##
# These below repos didn't work out for me got error message saying system-release >=15 required for
# rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-rawhide.noarch.rpm
#rpm -Uvh http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-rawhide.noarch.rpm

## Fedora 14 ##
rpm -Uvh http://rpms.famillecollet.com/remi-release-14.rpm

Install Firefox4:
yum --enablerepo=remi list firefox4
yum --enablerepo=remi install firefox4

Create different Profile for Firefox4:
firefox --ProfileManager (Create 2 profiles "Default User" and "FF4")

Start Firefox with FF4 profile using command:
firefox4 -P FF4

Or start Firefox 3.6.x with command:
firefox -P 'Default User'

Reference:
http://www.if-not-true-then-false.com/2010/install-firefox-4-on-fedora/
http://www.fedorafaq.org/#installsoftware

Note: Found what I needed was the above steps. Got most of them from these above 2 sources. Either of them singly didn't work out for me so clubbing together the total install steps into this post

Thursday, December 23, 2010

Networking Protocol, RFC and Administration Links

Introduction to Network administration

Email
1) Email Bible: Internet Email Protocols: A Developer's Guide [Paperback] by Kevin Jhonson
2) Top-Down Overview: Wood, D (1999). Programming Internet Mail. O'Reilly
3) Uses VB Source: Rhoton, J (1999). Programmer's Guide to Internet Mail: SMTP, POP, IMAP, and LDAP
4) Totally NT Oriented: Hughes, L (1998). Internet e-mail Protocols, Standards and Implementation. Artech House Publishers

Network Configuration
Linux Network Configuration
Linux Security Administrator's Guide
The Linux Documentation Project Guides
VPN HOWTO
___The TCP-IP Guide - Read Online Free
___The TCP-IP Guide -- Table of Contents

RFC's:
Post Office Protocol - Wikipedia, the free encyclopedia
RFC 1081 - Post Office Protocol Version 3
RFC 2449 - POP3 Extension Mechanism
RFC 2595 - Using TLS with IMAP, POP3 and ACAP
RFC 2616 - Hypertext Transfer Protocol -- HTTP-1.1
RFC 2822 - Internet Message Format
RFC 3164 - The BSD Syslog Protocol
RFC 3207 - SMTP Service Extension for Secure SMTP over Transport Layer Security
RFC 3507 - Internet Content Adaptation Protocol (ICAP)
RFC 5034 - The Post Office Protocol (POP3) Simple Authentication and Security Layer (SASL) Authentication Mechanism
RFC 5321 - Simple Mail Transfer Protocol

Saturday, December 18, 2010

Study Notes RARP Protocol

Study Notes on Reverse Address Resolution Protocol:

(Reference Chapter 6: Internetworking with TCP/IP: Vol 1 Principles, Protocols and Architecture - 3rd Ed. by Douglas E. Comer)

1) RARP is used by diskless machines which can't "remember" their IP address after a boot.

They send a physical-address (ex: ethernet NIC card's unique id) to RARP server(s) which reply back with IP address for that particular physical address.

Note: A machine can query IP address for another machine by supplying that machine's physical address. ?? Used by subnet routers to setup routing tables for its children??

2) RARP uses only ethernet frames as it can't use IP/TCP protocols without an IP address.

On getting hold of the IP address it uses TFTP for loading the boot image from the remote image server.

3) Request Message broadcast by QueryingMachine (to RARP server) as an Ethernet Frame:

PREAMBLE,

SRC-ADDR=QueryingMachine-physical-address,

DEST-ADDR=broadcast-physical-address,

FRAME-TYPE=RARP,

FRAME-DATA=(REQUEST, QueryingMachine-physical-address, target-physical-address), //227 bits (28 Octets) long frame data

CRC

Note:

a) The Ethernet Frame Header may be processed only by NIC card, so the the QueryingMachine-physical-address is repeated in frame data so it's accessible to the O.S.

b) A QueryingMachine can also ask the RARP server to lookup a target-machine's IP address by supplying the target-physical-address.

In such a case the target-physical-address would be different from sender-physical-address. ??Could be used by subnet routers to setup routing tables for its children??

Reference:Fig 2.8 for diagram of Ethernet Frame in Comer)

TODO: Need to cross-check how 2 physical addresses of 48 bits (6 octets) each can fit into the 227bits (28 octet) frame data

4) Reply Message sent by RARP server to QueryingMachine as an Ethernet Frame:

PREAMBLE,

SRC-ADDR=RARPServer-physical-address,

DEST-ADDR=QueryingMachine-physical-address,

FRAME-TYPE=RARP,

FRAME-DATA=(REPLY,QueryingMachine-physical-address, target-physical-address, target-IP-address), // ?? Used to update a subnetting routers routing table ??

CRC

Note:

RARP server sends the target-IP-address to the QueryingMachine and NOT the target machine.

??Could be used by subnet routers to setup routing tables for its children??

5) RARP REQUESTs are retryed by QueryingMachine

a) Looping at periodic intervals to avoid server overload (used for critical remote boot situations)

b) fixed number of retries to avoid network flooding. (used in non-critical remote boot situations)

6) Flooding can occur in case of failed PrimaryServer. This is avoided as follows:

a) REQUEST flooding by QueryingMachine is avoided by Secondary Servers replying back only on the 2nd REQUEST.

b) REPLY flooding is avoided in case of failed Primary Server by Secondary Servers replying back with randomly differing delays of REPLY.

Friday, December 10, 2010

Compare features of different wiki software

Looking for the right wiki to build up your own hobby/group/professional wiki?

I was just looking for a small nifty wiki to use as a scratchpad to collect technical tid-bits for my work/learning. Didn't want to install/administer stuff but wanted to be able to link up the tid-bits as I found connections between them. Found good candidates Tiddlywiki/Twiki using WikiMatrix's comparison table.

Too many wikis with differing features, limitations and target audiences make it tough to decide. You can easily end up spending a lot of time/effort in installing, administering, learning to use a wiki.

WikiMatrix has a wiki comparison wizard which allows you to select a small set of options to figure out which wiki s/ws are closest to what you're looking for.
Then it creates a comparison table with filterable feature-set to view only the features you're looking for in your dream wiki.

The site also allows you to compare other things like blogs etc.

Monday, December 6, 2010

Network Debug Utilities

Networking:

tcpdump
ping
traceroute

netstat

Displaying Connections

Netstat - Wikipedia, the free encyclopedia

RFC's:
SMTP
HTTP
FTP

wireshark
wireshark bibliography, articles

SysAdmin:
Linux Newbie Admin Guide
Linux Network Admin Guide
Linux Sys Admin Guide

Wednesday, December 1, 2010

STUB: TechTalk: C++ : What happens during compilation?

A) Phases of compilation:
1) compiler,
Process of compilation ,
2) linker, linking, preprocessor, preprocessing, assembler, assembling,

B) Different types of Outputs

1) object-file (.o/.obj),
library(.a/.so (Linux), .lib/.dll - Windows),
static-library (.a, .lib),
dynamic-library(.so, .dll),
name-mangling

G++:

shared-object-file, --save-temps

VC++:
release/debug,
multi/single threaded,
dll,
importlibrary,
.defs file,
export/import symbols

Linux KVM Kernel Virtual Machine

Linux KVM: Linux Kernel Virtual Machine
apt-get install -y kvm libvirt-bin ubuntu-vm-builder qemu bridge-utils
apt-get install -y ubuntu-virt-server ubuntu-virt-mgmt #KVM server utils and management GUI

Reviews of Classic Technical Books

[Book Reviews]

Technical:
Apprenticeship Patterns - Guidance for the aspiring software craftsman
Code Reading : The Open Source Perspective
Why Programs Fail by Andreas Zeller

Mash-ups:
Hacking FireFox : More than 150 hacks, mods and customizations

Problem Solving:
How to Solve It by G. Polya
How to Solve It by Computer by Dromey
Lateral Thinking by Edward De Bono

Soft Skills:
How to Win Friends and Influence People by Dale Carnegie

BookReading:
How To Choose Books
Why Read Books

Sunday, November 7, 2010

Remote Debugging with GDB

Remote debugging:
Connecting to remote target
    a) (gdb) target remote /dev/ttyb   # serial-device
    b) (gdb) target remote host:port
    c) (gdb) target remote protocol:host:port
    d) (gdb) target remote | middleman-command   # gdb talks to a middleman command

Transfer files:
    remote put hostfile targetfile
    remote get targetfile hostfile
    remote delete targetfile

    (gdb) load filename    # this command is used instead of 'run' in case of remote debugging
    (gdb) break
    (gdb) step
    (gdb) continue
    (gdb) detach
    (gdb) disconnect

a) Remote gdbserver on target with gdb on host (Need O.S. for this):
     target> gdbserver /dev/com1 emacs foo.txt
    target> gdbserver host:2345 emacs foo.txt
    target> gdbserver --attach /dev/com1 pid

    # Run a wrapper for the debugee program on the target-side
    target> gdbserver --wrapper env LD_PRELOAD=libtest.so -- :2222 ./testprog

    #Connect to the target gdbserver from my host gdb
    $ gdb myprogram
    (gdb) target remote myhost:9999
            0x00007f215893ba60 in ?? () from /lib64/ld-linux-x86-64.so.2
    (gdb) b main
    (gdb) continue

   # Send commands to gdbserver using the monitor command
    monitor help      # List the available monitor commands.
    monitor set debug 0 | 1      # Disable or enable general debugging messages
    monitor set remote-debug 0|1    # Disable or enable debugging messages of the remote protocol
    monitor exit

   See Also: Remote Configuration
   More in-depth internal look at Remote Protocol

b) Remote Stub: Used on Embedded devices without an O.S.

Blogs:
1) http://blog.flameeyes.eu/2010/02/20/remote-debugging-with-gdb-part-2-gdb
2) http://www.gentoo.org/proj/en/qa/backtraces.xml

Debugging Multiple Processes/Threads with GDB

Debugging multiple inferiors and programs
Gdb allows you to debug multiple programs at once without exiting the process you're currently running.
Gdb uses an inferior as a handle for a process. An inferior is attachable/detachable from a process
info inferiors # display list of current inferiors.

     (gdb) info inferiors
       Num  Description       Executable
       2    process 2307      hello
     * 1    process 3401      goodbye

Switch to another inferior:
inferior infno # make inferior number infno the current inferior

Loading multiple inferiors:

1)    add-inferior [ -copies n ] [ -exec executable ] # n defaults to 1 and optional exec gives an empty inferior
2)    clone-inferior [ -copies n ] [ infno ]    # n defaults to 1 and infno defaults to current inferior number

Ending the inferiority complex:
remove-inferior infno        # Remove the empty inferior Note: Can't remove an inferior with a running assoc. process.
detach-inferior infno         # detaches from an inferior from assoc. process
kill-inferiror infno             # kills the process attached to the inferior. It can be removed or re-attached to some process.

Debugging Multi-threaded Programs:
    info threads    # displays thread info - thread number, thread-id, stack frame summary
    thread threadno     # make threadno as active current thread and shows stack frame summary

Starting and Stopping Multi-thread Programs
2 modes of debugging:
1) All stop mode: aka 3 musketeers mode i.e. "All-for-one, One-for-all"
2) Non-stop mode:
    set target-async on|off    # Switch asynchronous mode.
    show target-async         # Show the current target-async setting.

    interrupt    # suspend execution of the running program (whole process in all-stop, current thread in non-stop modes)
    interrupt -a    # stop the whole program in non-stop mode

Saturday, October 30, 2010

The Pyramid of Knowledge

Knowledge can be thought of as a Pyramid of Information, like the predator-prey biological pyramid or energy distribution in an ecosystem.

Building the Knowledge Pyramid is like a "Fox and Hounds Paper Chase" so that the Walk-The-Long-Road on the Information Trail/Path becomes much easier.

First need to categorize learning resources into 3 levels:
1) Top-Down: Solution Space - What problems does this technology address and solve?
2) Middle-Up/Down: Module level concepts like Manager code e.g. Virtual Filesystem Cache Manager, Storage Manager
3) Bottom-Up:

Search for Short-Intros materials like ppts, podcasts, pdfs, research papers, articles, blogs, videos, mindmaps, TOC
Find the Gurus in the trade and their JourneyMan followers
Subscribe to RSS Feeds to blogs by them
1) JourneyMen in the domain like Yeshawant Kanetkar who explain things well to Apprentices (WalkInThePark Pattern)
2) MasterCraftsMen in the domain like Stroustrup, Richter, Butenhof (WalkInTheForest Pattern)

0) Blogs, articles or PPTs doing a system walk-through relating how various things and flows through the system
     like "Gusatvo Duartes blogs on Operating Systems"
1) General books to get a feel for the domain and common constraints, solutions, patterns and principles
    like "Operating System Principles", "Distributed Systems: Principles and Paradigms"
2) Applied Books using APIs, Frameworks
    like"Linux System Programming", "Windows Application Programming by Richter"
3) System/Sub-System Level books describing internals of APIs, Frameworks
     like "Understanding the Linux Kernel", "MFC Internals"

Categorizing Method Applied to Information Trail:
    Follow most referred book/white-paper trail in collections like Bibliographies, Appendices, References etc.
    This method works on any of the above 3 levels,
    Example:
   Get generic Distributed domain book from amazon and follow Up/Down to higher/lower levels of Pyramid.
    "Distributed Systems: Principles and Paradigms" -> "C++ Network Programming" ->
     a) POSA2 (Basic Distributed Patterns)-> POSA3 (Resource Mgmt Patterns) - (Design Level Classics)
     b) Operating System -> "Operating System Principles" by Galvin -> "Understanding theLinux Kernel" by Bovet and Cesati
     c) Module Level Concepts -> Caching Virtual FileSystems, Remoting Patterns -> Research Papers -> Classic Books referring to the Research Papers
     d) Networking -> "Programming the Sockets API" by Stevens -> ... -> "Linux Network Internals" (Implementation Level Classics)

On a Related Note:
A Concept Map can very nicely simulate such a Tree of knowledge.
Hmmm.... kind of like the way WikiMindMap hierarchically orgranises a Wikipedia article into a tree structure. So we can do the following to convert a wiki/webpage into a concept map:
1) Use WikiMindMap to convert wikipedia article into a Freemind MindMap and export it as a .mm file
2) Use XMind to convert the .mm file into an XMind Concept Map which is shareable and editable with friends.

Howto Learn in Small Chunks - (Bite Sized Learning)

Many of the curious and interested people always end up saying at one time or another....
"I've really wanted to learn more BUT I can't seem to find the time for it...."

What happens to our time, after all as someone said "God gave each one of us 24 hours".
So how is that some people seem to make so much more use of them than others?

Apart from the solid block of time allocated for our salary-paying masters what's left of them?

Waiting in queues for doctors, meetings to start, buses to show up.... these are the places we lose our free-time 24 hours.

a) The main problem is these time-chunks are fragmented i.e. the free chunks occur in-between major tasks. We need some method to connect these small fragments together
b) Mostly these free-time chunks are too small to do any significant work.
c) Sometimes we cannot predict when we'll get these free minutes/hours. So we tend to browse stuff and use up these fragments of time.

But there is a way that we can make full use of these time-chunks if we learn how to
a) Learn in small chunks by finding bite-sized activities.
b) Actively Record-What-You-Learn so that you can continue where you left off last time. This way the task switching overhead is minimized as you save your work every few minutes - kinda like saving that important file so you don't lose all your work when the computer hangs.

1) Subscribe to RSS Feeds to blogs by Gurus in the trade and their followers (WalkInThePark, WalkInTheForest Patterns)
2) Browse constantly using a phone/iPad to browse the net on idle time-chunks like waiting for the bus, doctor, queue etc.
3) Keep a book handy for power-cuts, reading on bus etc. (Make sure that it's small enough and has big letters for easy reading). StudyTheClassics to avoid wasting time on useless stuff. Amazon ratings may not be the best guide at all times. A classic is normally referenced by many books even the time-waste ones. Utilize the bibliography at the end of most books.
4) Learn Research-Techniques like Skimming, SpeedReading, Mindmapping on a book for quick outline of the book to be added in the book review.
5) Capture the essence of what you understood by taking notes a book for quick reference later.

Use the book fully -
1) Annotate the TOC with educated guesses and already known-facts gathered from Short-Intros. Peek into topics which you can't explain well to yourself or to refresh/cross-check your memory.
2) Categorize the bibliography into ShortIntros, Mid-Stuff, Adv-Stuff to build up your Knowledge Pyramid
3) Keep incrementing your blog on the topic with all resources - links, notes, thoughts, questions.
4) Keep Reviewing your blog to reduce/add TODOs:
    correct grammatical mistakes,
    add further links for later annotation,
    cleanup and reorganize information,
     re-view the gathered information with a fresh eye,
     simplify, clarify, generalize, specify, elaborate, concise information,
     split of a huge post into different blog posts
     (each post should not be more than 2 PageDowns for the reader)
5) IMPORTANT: Blog should not be a place where information goes to die but a place where it comes alive.
With each review and update it should spawn a new generation of Rise-From-The-Ashes-Phoenixes.
6) You'll be surprised to see how much you can achieve in small bites spread across time.

See Also:
How to Read A Book And Why (from bookshelved wiki)

Useful Sections of a Syntopic Learning Post

0) Add a TODO section at the top for quickly scanning pending actions on the blog
1) Build the Knowledge Pyramid one link at a time.
2) Add Short-Intros: ppts, podcasts, pdfs, research papers, articles, blogs, videos, mindmaps, TOC, tutorials, sample-code, howtos,
3) Add Mid-Stuff: guides, idioms, best-practises, FAQs, Gotchas, traps-and-pitfalls
4) Add Adv-Stuff: Design-level stuff, Patterns etc.

Thursday, October 28, 2010

Howto Capture Your Ideas and Knowledge into a MindMap using Freemind

For some time now I’ve been using Freemind tool to make mindmaps to capture ideas and knowledge.
I was pretty happy with Freemind the opensource and free mind mapping tool.
It has some great features including exporting the mindmap into different formats - html, browsable java-applet, picture formats, flash file, pdf etc.

What I'm really trying to do is to use and leverage my blog by adding browsable mind-maps.
I've been somewhat successful in finding resources but still failing to get the last-mile work done.
Once I do this I'll be able to use mind-maps for sharing with others and allow them to edit my work too. Seems like the best bet right now is to move to wordpress plugin below:

1) There is a Free Mind Viewer plugin for Wordpress which allows read-only browsing of your mindmap file (not just an image). I’ve not been able to try it out yet though.
2) XMind converts a Freemind mindmap into a concept map. You can then upload and share it using a free XMind account. Paid accounts also allow collaboration.
3) WikiMindMap converts any Wikipedia article into a mindmap for reading it visually.(Thanks to dorai's log for this info)
4) DeliciousMind = Delicious (bookmarking) + Freemind
blainekendall.com - DeliciousMind = del.icio.us + FreeMind mindmap
https://api.del.icio.us/v1/posts/recent?count=10000

See Also:
Freemind Mindmap Embedded In Wordpress - Web Development Jobs - oDesk
WordPress › Freemind Viewer « WordPress Plugins

FreeMind Google Module/Gadget: (Requires you to have your own blog hosting)
Freemind Viewer Google Module – Rolando Garza
Dissecting the Freemind Viewer – Rolando Garza
yuriy.org: FreeMind Flash Browser Google Gadget

placingafreemindmindmapusingobjectelements - rolandog02
Embedding mind maps - FreeMind - free mind mapping software
Freemind | drupal.org
Extension:FreeMind - MediaWiki

Freemind Collaboration on Windows - FreeMind - free mind mapping software

Freemind:
Main Page - FreeMind - free mind mapping software
Download - FreeMind - free mind mapping software

Using FreeMind - FreeMind FAQ
Tutorial effort - FreeMind - free mind mapping software
Import and export - FreeMind - free mind mapping software
Accessories - FreeMind - free mind mapping software
FreeMind 0.9.0 The New Features
FreeMind 0.8.0 Features

FreeMind FAQ
After the Applet Start - FreeMind FAQ
Essays - FreeMind - free mind mapping software
Stuff - FreeMind - free mind mapping software

Mindmapping: Articles on general mindmapping concept and uses
internet mind reader - Improves your thinking with free mind map and brainstorming software
Mindmapping, concept mapping and information organisation software
Freemind combined with blogging

[Note: This is a Work-in-Progress so subscribe to RSS feed for updates]

Wednesday, October 27, 2010

Howto do Reverse Debugging with GDB

Webinar on Reverse Debugging with DSF-GDB
This webinar introduces the new DSF-GDB debugger integration that is part of CDT 6.0. It illustrates the recent GDB additions such as Reverse debugging, Multi-Process and Non-stop multi-thread debugging.

GDB Reverse Debugging Wiki
GDB and Reverse Debugging
Presentation (PDF) on Internals of GDB Reverse Debug and Process Record Target
GDB Reverse Debugging Tutorial

Tuesday, October 26, 2010

[Stub] Debugging Strategies and Techniques

Howto Debug with GDB

MediaPacker - Utility to help pack your DVDs and CDs to the brim

Media Packer is meant to solve the following problems.
1) Allow good-fit packing of DVDs, CDs with given set of directories/files.
2) Intention is to semi-automate (i.e. provide suggestions to users) or fully automate the process of packing backup media such as DVD, CDs.
3) To be developed in Python for portability and ease-of-extension

InProgress:
1) Display media space occupied by a folder structure (files and directories).
File Size code snippet,
>>> import os
>>> b= os.path.getsize("/path/isa_005.mp3")
>>> b
File I/O tutorial
>>> f = open( 'test.txt', 'at')
>>> f.writelines( str(b))
>>> f.close()
>>> f = open( 'test.txt', 'rt' )
>>> s = f.readlines()
>>> sDirectory Size Lister (Check out the comments at end of blog to get info on )

Pydirstat. pydirstat is a really handy cross-platform, command-line way to generate information about disk usage. http://developer.berlios.de/projects/pydirstat/

Future features:
1) Add catalog and search facility to uniquely identify each media, add tags for folders/files and user comments.
2) Allow collaboration and sharing of catalogs among friends to share useful files.
3) Possibly use bit-torrent protocol for the sharing??

To be released under GPL.

See Also:
MediaPacker sourceforge account

StarUml is dead (NOT)!! Long Live the Stars!!

Star UML a very nice stable open-source design tool was suspected to be dead.
The tool is downloaded quite heavily, but development seems to have been abandoned as of 2005 with a stable release version 5.0.

But hey!! Just-a-minute, StarUML seems to have been rebooted as of (2010-July-19).
Originally written in Delphi, looks like its being converted to an eclipse plug-in in java.
Don't know how long it'll take to do the conversion though.
Seems like there are two(?) developers working on it right now.
Best of Luck to you.
Anybody with experience in Eclipse plugins and Java wanna help?!!!

Refer the project news and an announcement below:

Kick-off with Java-based StarUML Project

http://sourceforge.net/news/?group_id=152825

Monday, October 25, 2010

Delta Debugging

References:
Download Delta from delta.tigris.org
Why Programs Fail has won a Software Development Jolt Productivity Award!
Eclipse Plug-Ins - Software Engineering Chair (Prof. Zeller) - Saarland University
DDchange - DDchangeWiki
Delta Debugging article on Wikipedia
Delta Debugging articles by Andreas Zeller

See Also:
Episode 101: Podcast with Andreas Zeller on Debugging | Software Engineering Radio

Video by Andreas Zeller on Debugging the Debugging activities
Video YouTube - Learning from Code History
Mining Software Archives by Andreas Zeller
About Andreas Zeller - S/W Engg. Chair at Saarland University

Publications by Prof. Andreas Zeller

Adv Debugging - My Program Worked Yesterday, but Not Today (WYNOT) by Andreas Zellar

Brief Notes of my understanding of on Delta Debugging as discussed in “Worked Yesterday, NOt Today" by Andreas Zeller
Note: To understand the basic premise and constraints/approaches of Delta-Debugging referred to the pdf link to:

Analyse the tables/graphs carefully. (80% of the understanding comes from the example tables/graphs)
Connect the WYNOT talk with Andreas Zellar's podcast on Software Engineering Radio.

Problem: Yesterday's code works but Today's code doesn't. How do we find the change(s) which induces the test-case failure.
Context: Looking through the CVS history we find that N changes have been added since Yesterday.

Alternative: Normally we would need run the debugger to reproduce the failure and then try to collect information which induces it. But this requires a programmer to interactively query the program state using a debugger.
Can the debugging be automated without needing the programmer?

Strategy: Delta Debugging using a test-case to reproduce the failure.
Data Method:
Using the Scientific Method we can first try to reduce the Input Data in a binary search mechanism to isolate the minimum data which causes the failure.
Code Method:

Simple Method: When failure is caused by a single change. We can try to isolate the change which causes the failure by doing a binary search through the change history. If there are N changes between Y(esterday) and T(oday) we test 'k/2'th change where 1 <= k <= N. We go on partitioning until we find a failure-free change. The latest failing change is the culprit. Complexity of this binary search is O (log N)
Complex Method: When failure is caused by a combination of changes. Unfortunately it's Not always possible to use the Simple Method in the case when combination of changes cause the failure. In such a situation we need to identify at least 2 (or more) changes which together cause the failure.

Divide the change history into N changes
Group N changes into say 4 units i.e. N = N/4 * 4
Generate combinations with different units.
Test combinations for the failure.
Prune any units which don't participate in the failure at all.
Recursive above steps by sub-dividing the units above (quarters into octets and so on) until we finally we end up with minimal changes that reproduce the failure.

Proposed Plans to Reduce to Reduce Computation and Avoid Inconsistent Configurations :
a) Group related changes - changes on common date/file/variable are more likely to be related. Group them into a single unit to avoid dependency across units.
b) Prune configurations when they're obvious dead-ends e.g. Where change dependency is chained all the way (Today) 10->9->8->7->6->5->4->3->2->1 (Yesterday)
c) Exclude changed code which is never executed.

See Also:
Pdf: "WYNOT - Worked Yesterday, NOt Today"
Andreas Zeller
Podcast on Software Engineering Radio
Delta Debugging

Saturday, October 23, 2010

Debugging with GDB's Cheat-Sheet of Commonly Used Options

Gdb:
Compiling a program for debugging:
    g++ -ggdb -O0 hello.cpp   #
    g++ -g3 -O0 hello.cpp       # -g3 compiles debugging info for preprocessor macros and no optimization

Running a program under the debugger without arguments
$> gdb myProgram

Setting the program arguments:
   a) $> gdb --args myProgram 10 20 30 40
   b) (gdb) start 10 20 30 40 # sets temporary breakpoint at main() and calls run.
   c) (gdb) file myProgram
        (gdb) run 10 20 30 40
   d)   (gdb) set args 10 20 30 40
   (gdb) show args
        (gdb) run
    e) $> gdb myProgram core
    f) $> gdb -tui myProgram core # start vim/emacs ide for debug session

Howto Debug a Core-Dump/Crash-Dump for a stripped binary from a production machine

Core dump Analysis:
core(5)– (Linux Programmer's Manual – File Formats)
Core Dump (Wikipedia)
Me mory dump (Wikipedia)
CallStack (Wikipedia),
Enabling core dumps in embedded systems,

Dr Dobbs - Postmortem Debugging
Dr Dobbs - Post-Mortem Debugging Revisited

Stackoverflow article (with comments) on core dump analysis on Solaris,
Backtrace commands in GDB documentation
Howto generate a Backtrace for Bug Reporting by Testers (from Gentoo doc)

Stack buffer overflow (Wikipedia)

Segmentation fault (SIG_SEGV, SIG_BUS)
A FAQ: User contributed answers regarding the definition of a segmentation fault,
A "null pointer" explained,
Answer to: NULL is guaranteed to be 0, but the null pointer is not?
Resolving crashes and segmentation faults,
What is the difference between a segmentation fault and a stack overflow?

Windows:
Crash dump analysis website

Books:
Debugging: The Nine Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems (by David J. Agans)
Software Exorcism: A Handbook for Debugging and Optimizing Legacy Code (by Bill Blunden)
Debugging by Thinking : A Multidisciplinary Approach (by Robert C. Metzger)
Memory Dump Analysis Anthology, Volume 1 (by Dmitry Vostokov)

Tuesday, October 12, 2010

Steps towards Technical Mastery

How do you know if you've become a Master in your field?

Is there any method to learn things quickly in any field?
How do you avoid stagnation and mediocrity after becoming competent in your field?
How do you become a Master?

Take software for example: How do you become a Master in it?

These are exactly the questions which have been pushing me to seek answers.
Well now that I've got some kind of grip on this quest I'm able to give some of the answers.
Or at the very least point the way.

Short Answer: You'll know it when you get there!!
Long Answer: You'll see the Sign-Posts as you move towards achieving Mastery. Read on for the gory details.

Basic Template with Ingredients for a Technical Blog

A good Technical Blog Post follows the Law Of Retention: repetition helps Retention

Some Questions on your Blog Post:

Is your blog a frequently used resource for yourself and your readers?
Does it grow as you learn more - addressing different levels of questions?
Is it short and informative while being complete and self-contained?

Is your blog addressing The Readers Perspective?

What's in it for a Reader?
Tell 'em what you're going to Tell 'em.
Tell 'em.
Tell 'em what you just told 'em.
Further info.

Understand Source Code by Reading Code and Refactoring for Maintaining Large Projects

This list consists of one-liner tips to be used for easy lookup and reference.

Rewriting Code for Better Understanding (Coding Horror)

What I cannot create, I do not understand. [Richard Feynman]
"What I do not create, I do not understand."[Chris]
"It's easier to understand 600 tables than 100,000 lines of code." [PaulC on comp.databases.theory/2005]
Create a new project with blocks of code copied from old code. Refactor to reduce "old code smell"
Start refactoring by adding small tests to better understand code and act as regression tests.
Just scan through huge amounts of code till it starts making sense - "understand by constant exposure".
Create a text file to capture your understanding - briefly describe how the component works, realtions to other components, what it persists, its states, and source browsing/debugging tips you found useful.
Manually reformat a copy of the code - white space, and indented as per your personal style.

References:


Avoid Bloggers-Block:

After the initial burst you can't think of what to say - so the blog languishes.; A blog is short form for web-log i.e. a kind of "A log entry in your Personal Diary".; It is not meant to be a book which you publish after many many rewrites and deep thought.; Some people even have one-liners or even other peoples blogs as blog-entries.

Find an Idea/Topic:

Your idea becomes self sustaining if it's useful or matters to you or your friends.

You can organise your self-learning or teaching notes as blogs. Collect questions from people around you or from forum newbies. This way you create more knowledge for yourself and write excellent blog-posts that are useful to others.

Publish or Perish:

Publish as soon as you have a line of text on your blogs.

Drafts are the best way to kill blogs.

Use them only when you require to do some basic tidying before posting.

Even this can be done later followed by a republish.

Blog Incrementally:

Paste your current state into the blog (like saving a session of your mind a.k.a. brain-dump).

Every time some find some new useful information you just paste that into your blog-post.

Use a multiple clips clipboard manager to copy all the info you need.

The post should be written in logical manner with least gaps in the reasoning or steps

Any good links that you open in Firefox tabs can be saved for:

a) Further investigation

b) Future use

Reduce Blogging Effort:

Ideally follow 80-20% rule.

Spend 80% time spent on gathering material.

Spend 20% time spent on filling in gaps and reordering/rewriting post.

See Tools section for more details

Tools:

Currently I'm trying to reduce editing/formatting effort/time by using the following tools:; Blog Upload: Evaluating desktop client s/w's for editing and posting to blog (like ScribeFire, Windows Live Writer, SeMagic)
I've used 2-3 good desktop blogging clients: 1) ScribeFire to handle blog-editing, upload and managing of multiple accounts/blogs.
Looks to be the best option for now. 2) Kompozer is good for a basic HTML whysiwyg editor.
3) Windows Live Writer - not really used it but looks good enough for basic blog maintainence.: Blog Composing: Kompozer, CSS Stylesheets (TODO); MindMaps: XMind with Freemind (for mindmaps and concept maps with collaboration respectively); Pictures: Paintbrush Digicam for Photos, MediaWiki Free Photos; Video: InstantDemo, Jing ScreenCapture software for Demonstration Videos.
Diagrams: Diagramming s/w (To-Do): Uploading: To-Do; See Also:Basic Format for a Technical Blog

Tuesday, October 5, 2010

Howto Improve your Blog's design

Accessibility: Does the title grab the attention of your intended audience?

Does the title have keywords used by the audience while searching google?
Does google index your blog regularly?
Push your blog updates to blog broadcastors like weblogs.com
Does your blog load up fast? Use the "Jump/Read More" feature and Optimize load time.

Retention: Is it easy for a casual browser to remember your blog and info you pain-stakingly put in?

mmap() is not the territory!! Part 2

Final code at end of debugging:

bool testFileCopyWithSharedMem(const string &srcFileName, const string &destFileName)
{
    bool isCopyOk = false;
    string diffCmd = "diff " + srcFileName + " " + destFileName;
    int retval = system(diffCmd.c_str());
    cout << "system(" << diffCmd << ") returned: " << retval << endl;
    if( retval == 0 )
    {
        isCopyOk = true;
    }
    return isCopyOk;
}

int doFileCopyWithSharedMem(const string &srcFileName, const string &destFileName, size_t sharedMemSize)
{
    //int retval = 0;

    //Map SourceFile
    errno = 0;
    int srcfd = open(srcFileName.c_str(), O_RDONLY);
    if (srcfd < 0) {
        const char * causeOfError = strerror(errno);
        cout << "open() returned:" << causeOfError << " at:" << __LINE__ << " in:" <<__FUNCTION__ << endl;
        cout << "open(" << srcFileName << ") returned: " << srcfd << endl;
        return -1;
    }
    struct stat sb;
    fstat(srcfd,&sb);
    long pageSize;
    //pageSize = sb.st_size;
    pageSize = sysconf(_SC_PAGESIZE);
    errno = 0;
    char *srcFilePtr = 0;
    srcFilePtr = (char *) mmap(0, pageSize, PROT_READ, MAP_SHARED, srcfd, 0);
    if (srcFilePtr == MAP_FAILED)
    {
        const char * causeOfError = strerror(errno);
        cout << "mmap() returned:" << causeOfError << " at:" << __LINE__ << " in:" <<__FUNCTION__ << endl;
        return -1;
    }

    //Map Dest File
    errno = 0;
    int destfd = open(destFileName.c_str(), O_RDWR|O_CREAT|O_TRUNC, 0600 );
    if (destfd < 0) {
        const char * causeOfError = strerror(errno);
        cout << "creat() returned:" << causeOfError << " at:" << __LINE__ << " in:" <<__FUNCTION__ << endl;
        cout << "creat(" << destFileName << ") returned: " << destfd << endl;
        return -1;
    }
    errno = 0;
    char *destFilePtr = 0;
    destFilePtr = (char *) mmap(0, pageSize, PROT_WRITE|PROT_READ, MAP_SHARED, destfd, 0);
    if (destFilePtr == MAP_FAILED) {
        const char * causeOfError = strerror(errno);
        cout << "mmap() returned:" << causeOfError << " at:" << __LINE__ << " in:" <<__FUNCTION__ << endl;
        return -1;
    }

    ftruncate(destfd, sb.st_size);
    cout<< "PID:" << getpid() <
    //cout << srcFilePtr << destFilePtr <
    system("cat /proc/self/maps");

    memcpy(destFilePtr, srcFilePtr, sb.st_size);
    //msync(destFilePtr, pageSize,MS_SYNC);

    munmap(srcFilePtr, pageSize);
    munmap(destFilePtr, pageSize);
    close(srcfd);
    close(destfd);
    return 0;
}

----------------------------------------------------------------------------

Time required:

Debugging 1 AM to 5:40 AM.

20 Minutes to note down these points in the blog.

2 hrs to massage it into shape.

Interesting Links:

DevShed : development tutorials: http://www.devshed.com/

Gentoo Bug Reporting Guide: http://www.gentoo.org/doc/en/bugzilla-howto.xml

C sample source on Gnu/Linux : http://www.c.happycodings.com/Gnu-Linux/index.html

Mmap() security bug with Null Pointers: http://blog.ksplice.com/2010/03/null-pointers-part-i/

Wiki on Chromium multi-process debugging with gdb: http://code.google.com/p/chromium/wiki/LinuxDebugging

BugReport: http://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg2333427.html

mmap is not the territory Part 1 : http://techtalkies.blogspot.com/2010/09/mmap-is-not-territory-or-mapfail-sigbus.html

Code Review Tools

Code Review Tools:
http://www.reviewboard.org/
http://code.google.com/p/rietveld/

Sample Review and Analysis of DVCS (Git/Mercurial) in Python Enhancement Proposals:
http://www.python.org/dev/peps/pep-0374/

Push your blog to google for indexing

Manually suggest your blog to google for indexing/crawling:
http://blogsearch.google.com/ping

Refer this site for further info on getting your blog indexed:
http://www.google.com/support/blogsearch/?hl=en#getlisted

mmap() is not the territory!! Part 1

(OR) how MAP_FAIL, SIGBUS,

beg to differ with mmap!!
Wrote a small program using mmap() to copy a fromFile.txt to toFile.txt using virtual memory

--------------------------------------------
Issue#1: mmap() returns MAP_FAIL
mmap() requires exactly the same flags as the filedescriptor/fd as when it was open()/creat()-ed
i.e. if you used O_RDONLY in open() then you can't mmap() it as PROT_WRITE.

chmod basics: http://www.linux.org/lessons/beginner/l14/lesson14b.html
creat/open man page: http://linux.about.com/od/commands/l/blcmdl2_open.htm
-------------------------------------------------
Issue#2:
Error Message: In gdb getting SIGBUS with error coming from inside memcpy!!

108 memcpy(destFilePtr, srcFilePtr, pageSize);
5: pageSize = 14
4: destFilePtr = 0xb7ffc000

3: srcFilePtr = 0xb7ffd000 "hello, world\n\n"
2: destfd = 6
1: srcfd = 5
(gdb) n

Program received signal SIGBUS, Bus error.
__memcpy_ia32 () at ../sysdeps/i386/i686/multiarch/../memcpy.S:75
75 ../sysdeps/i386/i686/multiarch/../memcpy.S: No such file or directory.
in ../sysdeps/i386/i686/multiarch/../memcpy.S

Searching on this gave no direct answers on the error message but gave some pointers on causes.

Tried all the below tools but still getting the same error message:

0) used apt-get update and rebooted "just-in-case"

https://help.ubuntu.com/8.04/serverguide/C/apt-get.html

1) used strace ./file-copy-vm to check the system calls were getting called properly.

getting SIGBUS on write() ostensibly called from inside the memcpy

2) used ldd ./file-copy-vm to check that the libstdc++ - and libc.so were existent.

They were present

3) Installed glibc-dbg and libstdc++-devel etc for debugging the library.

No change.

Finally found the problem in an off-by-N error in memcpy(dest,src, pageSize)

Since SIGBUS comes when a page worth is allocated but size of mapping is less than pagesize.

i.e. the accessed address is more than filesize but inside pagesize.

So realized my mistake and changed:

memcpy(destFilePtr, srcFilePtr, pageSize); //WRONG.

Corrected it to:

memcpy(destFilePtr, srcFilePtr, sb.st_size); //Correct

------------------------------------------------------------------------------------------------------------------
Issue#3: Still getting SIGBUS from memcpy() AND

out of bounds>

Breakpoint 3, doFileCopyWithSharedMem (srcFileName=..., destFileName=..., sharedMemSize=8192) at ../src/file-copy-vm.cpp:104
104 cout<< "PID:" << getpid() <
4: destfd = 6
3: srcfd = 5
2: srcFilePtr = 0xb7ffd000 "hello, world\n\n"
1: destFilePtr = 0xb7ffc000

0xb7ffc00 out of bounds>

(gdb) n
PID:7136
106 system("cat /proc/self/maps");
4: destfd = 6
3: srcfd = 5
2: srcFilePtr = 0xb7ffd000 "hello, world\n\n"
1: destFilePtr = 0xb7ffc00

0xb7ffc00 out of bounds>

(gdb) shell cat /proc/7136/maps
[SNIP]
08048000-0804a000 r-xp 00000000 08:01 933913     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/file-copy-vm
0804a000-0804b000 r--p 00001000 08:01 933913     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/file-copy-vm
0804b000-0804c000 rw-p 00002000 08:01 933913     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/file-copy-vm
0804c000-0806d000 rw-p 00000000 00:00 0          [heap]
b7fec000-b7fee000 rw-p 00000000 00:00 0
b7ffb000-b7ffc000 rw-p 00000000 00:00 0
b7ffc000-b7ffd000 rw-s 00000000 08:01 933725     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/toFile.txt
b7ffd000-b7ffe000 r--s 00000000 08:01 933672     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/fromFile.txt
b7ffe000-b8000000 rw-p 00000000 00:00 0
bffeb000-c0000000 rw-p 00000000 00:00 0          [stack]

Hmmm... here the destFilePtr seems to be pointing to the correct memory-mapped file i.e. toFile.txtThe destFilePtr

is pointing to out-of-bounds accesss.

Also the SIGBUG seems to suggest that I'm trying to write/access to an address in memory that's allocated but out of bounds of the empty dest file. Otherwise I'd have got a SIGSEGV if the memory had not been allocated.

I observed that toFile.txt is showing filesize as zero on the disk. Aha!!

Robert Love in his book "Linux System Programming" had mentioned files with slack-space/holes (in memory as well as disk respectively). Could this be the reason??!!

http://www.devshed.com/c/a/BrainDump/Using-mmap-for-Advanced-File-IO/

Finally I cross-checked my code with a sample implementation for mmap() file copy program.

Found out that they used lseek() to expand the memory mapping instead of ftruncate() to expand file size with a hole instead of ftruncate().

http://www.c.happycodings.com/Gnu-Linux/code6.html

So we just need to increase the size of the dest-file using ftruncate.

(gdb) help call
Call a function in the program.
The argument is the function name and arguments, in the notation of the
current working language. The result is printed and saved in the value
history, if it is not void.

(gdb) call ftruncate(destfd,sb.st_size)
$1 = 0

doFileCopyWithSharedMem() returned:0
system(diff ./fromFile.txt ./toFile.txt) returned: 0
testFileCopyWithSharedMem() returned:0

(gdb)

Hurray!! the unit testcase passes!!
It's 6 AM!! Tiring but worth it!! Gotta go and sleep now