Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Tuesday, October 26, 2010

Learning Python - Collection of Resources and Links



Beginner:
BeginnersGuide
BeginnersGuide/Download
BeginnersGuide/Overview
HowToEditPythonCode
BeginnersGuide/NonProgrammers
BeginnersGuide/Programmers
How do I Run a Program Under Windows
PythonEditors
BeginnersGuide/Programmers (Cpp2Python.pdf)
Good Presentation by a C guy on Python features


Blogs:
Dorai's blog post on Learning Python

CookBooks:


BeginnersGuide/Programmers/SimpleExamples
Python Snippets and Recipes
Python by Example
Common programming tasks in Python
Pythonic File Searches


Using Pythonic Idioms and Style:
Definition of "Pythonic" here
Presentations:
How to Write Pythonic" code by Christopher Arndt
Presentation on "Code Like a Pythonista: Idiomatic Python" by David Goodger
Presentation on "Code Like a Pythonista: Idiomatic Python" (Crunchy Remix) by Jeff Hinrich 
How to produce slides like above using Python tools

Style Guide for Python Code

My blog on Exploring the Python standard library source code

Books:
IntroductoryBooks
Python books
How to think like a computer scientist with Python
Bruce Eckel's community contributed book: Python 3 - Patterns & Idioms

Version Issues:
Python2orPython3
What's New in Python 3.0
What's New in Python 2.6
What's New in Python 2.7
Moving from Python 2 to Python3 (PDF)
Porting Code to Python 3 with 2to3
Case Study: Porting chardet to Python 3

Learning Resources:
Installing Python

MultiMedia:
Video on "Head First into Python 3"
5-minute videos on Python Capability
ShowMeDo Screencast of Python 2.5 Development on XP
Audio/Visual Talks on Python
Introduction to Computer Science and Programming (videos from MIT course)
Podcast Python411

Python3:
"What an IronPython user should know about Python 3"

Python 3 focused version of Dive Into Python
Teaching programming with Python 3

Python 3.1:
Setup
Documentation

Python2.x:

Python Tutorial
Dive Into Python
A Byte of Python
Instant Python
Download Thinking in Python (Old) by Bruce Eckel here
Python for Java programmers


Moving to Python From Other Languages
Python is not Java


MovingToPythonFromPerl
Practice Problems for Python


Reference:
Official Python website
Python Wiki
Python 2.7 FAQ
Python 2.7 Windows FAQ
Python Quick Reference
Python Language Reference manual 
Python's Standard Library Reference manual
Module Index of the Python docs
Python's reply to Perl's CPAN: Python Package Index (PYPI)
Extending and Embedding the Python Interpreter
Python/C API Reference Manual

See Also:
Eric S. Raymond article on moving to Python: 'Why Python?"

Tuesday, October 12, 2010

Steps towards Technical Mastery

How do you know if you've become a Master in your field?

Is there any method to learn things quickly in any field?
How do you avoid stagnation and mediocrity after becoming competent in your field?
How do you become a Master?

Take software for example: How do you become a Master in it?

These are exactly the questions which have been pushing me to seek answers.
Well now that I've got some kind of grip on this quest I'm able to give some of the answers.
Or at the very least point the way.

Short Answer: You'll know it when you get there!!
Long Answer: You'll see the Sign-Posts as you move towards achieving Mastery. Read on for the gory details.

Tuesday, September 28, 2010

mmap() is not the territory!! Part 2

Final code at end of debugging:

bool testFileCopyWithSharedMem(const string &srcFileName, const string &destFileName)
{
    bool isCopyOk = false;
    string diffCmd = "diff " + srcFileName + " " + destFileName;
    int retval = system(diffCmd.c_str());
    cout << "system(" << diffCmd << ") returned: " << retval << endl;
    if( retval == 0 )
    {
        isCopyOk = true;
    }
    return isCopyOk;
}

int doFileCopyWithSharedMem(const string &srcFileName, const string &destFileName, size_t sharedMemSize)
{
    //int retval = 0;

    //Map SourceFile
    errno = 0;
    int srcfd = open(srcFileName.c_str(), O_RDONLY);
    if (srcfd < 0) {
        const char * causeOfError = strerror(errno);
        cout << "open() returned:" << causeOfError << " at:" << __LINE__ << " in:" <<__FUNCTION__ << endl;
        cout << "open(" << srcFileName << ") returned: " << srcfd << endl;
        return -1;
    }
    struct stat sb;
    fstat(srcfd,&sb);
    long pageSize;
    //pageSize = sb.st_size;
    pageSize = sysconf(_SC_PAGESIZE);
    errno = 0;
    char *srcFilePtr = 0;
    srcFilePtr = (char *) mmap(0, pageSize, PROT_READ, MAP_SHARED, srcfd, 0);
    if (srcFilePtr == MAP_FAILED)
    {
        const char * causeOfError = strerror(errno);
        cout << "mmap() returned:" << causeOfError << " at:" << __LINE__ << " in:" <<__FUNCTION__ << endl;
        return -1;
    }

    //Map Dest File
    errno = 0;
    int destfd = open(destFileName.c_str(), O_RDWR|O_CREAT|O_TRUNC, 0600 );
    if (destfd < 0) {
        const char * causeOfError = strerror(errno);
        cout << "creat() returned:" << causeOfError << " at:" << __LINE__ << " in:" <<__FUNCTION__ << endl;
        cout << "creat(" << destFileName << ") returned: " << destfd << endl;
        return -1;
    }
    errno = 0;
    char *destFilePtr = 0;
    destFilePtr = (char *) mmap(0, pageSize, PROT_WRITE|PROT_READ, MAP_SHARED, destfd, 0);
    if (destFilePtr == MAP_FAILED) {
        const char * causeOfError = strerror(errno);
        cout << "mmap() returned:" << causeOfError << " at:" << __LINE__ << " in:" <<__FUNCTION__ << endl;
        return -1;
    }

    ftruncate(destfd, sb.st_size);
    cout<< "PID:" << getpid() <
    //cout << srcFilePtr << destFilePtr <
    system("cat /proc/self/maps");

    memcpy(destFilePtr, srcFilePtr, sb.st_size);
    //msync(destFilePtr, pageSize,MS_SYNC);

    munmap(srcFilePtr, pageSize);
    munmap(destFilePtr, pageSize);
    close(srcfd);
    close(destfd);
    return 0;
}
 
----------------------------------------------------------------------------
Time required:  
Debugging 1 AM to 5:40 AM. 
20 Minutes to note down these points in the blog. 
2 hrs to massage it into shape.

Interesting Links:
DevShed : development tutorials: http://www.devshed.com/
Gentoo Bug Reporting Guide: http://www.gentoo.org/doc/en/bugzilla-howto.xml 
C sample source on Gnu/Linux : http://www.c.happycodings.com/Gnu-Linux/index.html
Mmap() security bug with Null Pointers: http://blog.ksplice.com/2010/03/null-pointers-part-i/
Wiki on Chromium multi-process debugging with gdb: http://code.google.com/p/chromium/wiki/LinuxDebugging
BugReport: http://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg2333427.html 


mmap is not the territory Part 1 : http://techtalkies.blogspot.com/2010/09/mmap-is-not-territory-or-mapfail-sigbus.html

mmap() is not the territory!! Part 1

(OR) how MAP_FAIL, SIGBUS,
beg to differ with mmap!!
Wrote a small program using mmap() to copy a fromFile.txt to toFile.txt using virtual memory

--------------------------------------------
Issue#1: mmap() returns MAP_FAIL
mmap() requires exactly the same flags as the filedescriptor/fd as when it was open()/creat()-ed
i.e. if you used O_RDONLY in open() then you can't mmap() it as PROT_WRITE.

chmod basics: http://www.linux.org/lessons/beginner/l14/lesson14b.html
creat/open man page: http://linux.about.com/od/commands/l/blcmdl2_open.htm
-------------------------------------------------
Issue#2:
Error Message: In gdb getting SIGBUS with error coming from inside memcpy!!

108        memcpy(destFilePtr, srcFilePtr, pageSize);
5: pageSize = 14
4: destFilePtr = 0xb7ffc000

3: srcFilePtr = 0xb7ffd000 "hello, world\n\n"
2: destfd = 6
1: srcfd = 5
(gdb) n

Program received signal SIGBUS, Bus error.
__memcpy_ia32 () at ../sysdeps/i386/i686/multiarch/../memcpy.S:75
75    ../sysdeps/i386/i686/multiarch/../memcpy.S: No such file or directory.
    in ../sysdeps/i386/i686/multiarch/../memcpy.S
 
Searching on this gave no direct answers on the error message but gave some pointers on causes.
Tried all the below tools but still getting the same error message:
0) used apt-get update and rebooted "just-in-case"
    https://help.ubuntu.com/8.04/serverguide/C/apt-get.html
1) used strace ./file-copy-vm to check the system calls were getting called properly.
   getting SIGBUS on write() ostensibly called from inside the memcpy
2) used ldd ./file-copy-vm to check that the libstdc++ - and libc.so were existent.
   They were present
3) Installed glibc-dbg and libstdc++-devel etc for debugging the library.
    No change.
Finally found the problem in an off-by-N error in memcpy(dest,src, pageSize)

Since SIGBUS comes when a page worth is allocated but size of mapping is less than pagesize. 
i.e. the accessed address is more than filesize but inside pagesize.

 So realized my mistake and changed:
        memcpy(destFilePtr, srcFilePtr, pageSize); //WRONG.
Corrected it to:
        memcpy(destFilePtr, srcFilePtr, sb.st_size); //Correct
------------------------------------------------------------------------------------------------------------------
Issue#3: Still getting SIGBUS from memcpy() AND
out of bounds>

Breakpoint 3, doFileCopyWithSharedMem (srcFileName=..., destFileName=..., sharedMemSize=8192) at ../src/file-copy-vm.cpp:104
104        cout<< "PID:" << getpid() <
4: destfd = 6
3: srcfd = 5
2: srcFilePtr = 0xb7ffd000 "hello, world\n\n"
1: destFilePtr = 0xb7ffc000
0xb7ffc00 out of bounds>

(gdb) n
PID:7136
106        system("cat /proc/self/maps");
4: destfd = 6
3: srcfd = 5
2: srcFilePtr = 0xb7ffd000 "hello, world\n\n"
1: destFilePtr = 0xb7ffc00
0xb7ffc00 out of bounds>

(gdb) shell cat /proc/7136/maps
[SNIP]
08048000-0804a000 r-xp 00000000 08:01 933913     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/file-copy-vm
0804a000-0804b000 r--p 00001000 08:01 933913     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/file-copy-vm
0804b000-0804c000 rw-p 00002000 08:01 933913     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/file-copy-vm
0804c000-0806d000 rw-p 00000000 00:00 0          [heap]
b7fec000-b7fee000 rw-p 00000000 00:00 0
b7ffb000-b7ffc000 rw-p 00000000 00:00 0
b7ffc000-b7ffd000 rw-s 00000000 08:01 933725     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/toFile.txt
b7ffd000-b7ffe000 r--s 00000000 08:01 933672     /home/gurud/cdt-linux-tools-workspace/file-copy-vm/Debug/fromFile.txt
b7ffe000-b8000000 rw-p 00000000 00:00 0
bffeb000-c0000000 rw-p 00000000 00:00 0          [stack]

Hmmm... here the destFilePtr seems to be pointing to the correct memory-mapped file i.e. toFile.txt
The destFilePtr
is pointing to out-of-bounds accesss.
Also the SIGBUG seems to suggest that I'm trying to write/access to an address in memory that's allocated but out of bounds of the empty dest file. Otherwise I'd have got a SIGSEGV if the memory had not been allocated.
 
I observed that toFile.txt is showing filesize as zero on the disk. Aha!!
 
Robert Love in his book "Linux System Programming" had mentioned files with slack-space/holes (in memory as well as disk respectively). Could this be the reason??!! 
http://www.devshed.com/c/a/BrainDump/Using-mmap-for-Advanced-File-IO/
Finally I cross-checked my code with a sample implementation for mmap() file copy program.
Found out that they used lseek() to expand the memory mapping instead of ftruncate() to expand file size with a hole instead of ftruncate().
http://www.c.happycodings.com/Gnu-Linux/code6.html

So we just need to increase the size of the dest-file using ftruncate.

(gdb) help call
Call a function in the program.
The argument is the function name and arguments, in the notation of the
current working language.  The result is printed and saved in the value
history, if it is not void.

(gdb) call ftruncate(destfd,sb.st_size)
$1 = 0

doFileCopyWithSharedMem() returned:0
system(diff ./fromFile.txt ./toFile.txt) returned: 0
testFileCopyWithSharedMem() returned:0

(gdb)

Hurray!! the unit testcase passes!! 
It's 6 AM!! Tiring but worth it!! Gotta go and sleep now

----------------------------------------------------------------------------

See Also : 
mmap is not the territory Part 2 : http://techtalkies.blogspot.in/2010/09/mmap-is-not-territory-part-2.html

Friday, September 24, 2010

List of useful packages for C/C++ development on Ubuntu Linux Part 2

[SOURCE-ANALYZERS]
apt-get install -y exuberant-ctags
apt-get install -y doxygen doxygen-gui
apt-get install splint #Static analyzer Lint for C programs
apt-get install colorgcc #Colorizer for GCC errors/warnings
apt-get install cppcheck #C/C++ source static analyzer
apt-get install cscope #C/C++ source code browsing/searching
apt-get install cbrowser #browser for cscope
apt-get install cflow #Displays control flow graphs for C source
apt-get install cutils #Various C source code utils - cdecl, cobfusc
apt-get install cxref #Generate HTMl doc for C source code
apt-get install global #Global search/browse C++ source code
apt-get install id-utils #Identifier database used by global for search/browse C++ source code
apt-get install synopsis #C++/Python source code introspection tool
apt-get install gccxml #GCC source code described as xml
apt-get install gobject-introspections #Extract introspection data from libraries
apt-get install explain #Helps explain system call errors after the fact
apt-get install fhist #File history, compare and merge utility
apt-get install fastdep #Generates dependency info as makefile rules for C/C++
apt-get install eresi #Reverse Engineering, instrumentation, debugging, tracing framework
apt-get install evarista #Program transformer and data-flow analyzer for binaries using ERESI
apt-get install frama-c #GUI to combine multiple analyzers for C source code
apt-get install frama-c-base #Framework to combine multiple analyzers for C source code

apt-get install gnulib #Make programs portable using C macros/assertions/declarations/definitions
apt-get install gperf gperf-ace #Generate perfect hash given input strings
apt-get install gsoap #Web service stub/skeleton generator for C/C++ code
apt-get install kodos #GUI to debug, test and view regexps
apt-get install visual-regexp #GUI in TCL to debug, test and view regexps

[BEAUTIFIER]
apt-get install astyle # C++ source code beautifier
apt-get install indent #Beautifier for C Source code
apt-get install bcpp #C++ Source beautifier
apt-get install kwstyle #Ensure source code style of many people is same as one person
apt-get install uncrustify #C++ beautifier highly configurable
apt-get install unifdef #Remove #ifdef sections from source
apt-get install universalindentgui #GUI to configure and compare multiple beautifiers esp. for C++
apt-get install xmlindent #XML beautifier

[COMPILATION]
apt-get install distcc #distributed compiling
apt-get install distcc-pump #distributed preprocessing
apt-get install ccache #compiler cacher
apt-get install distccmon-gnome #GTK+ GUI to monitor distcc
apt-get install icecc #Distributed compiling
apt-get install icecc-monitor #GUI for monitoring Distributed compiling
apt-get install boost-build #Easy Cross-platform compilation
apt-get install gdc # D language compiler
apt-get install bison++ #C++ source generator enhancement to bison
apt-get install flex
apt-get install antlr3 #create compilers/interpreters using ANTLR

[LIBRARIES]
apt-get install libasio-dev libasio-doc #Cross Platform Boost library for Async IO (network programming)
apt-get install libclthreads-dev libclthreads-doc #POSIX threads C++ library
apt-get install libcorelinux libcorelinux-examples #Converting Linux core C libs to C++ libs
apt-get install libdar-dev #Disk archiver
apt-get install uc++ uc++-doc uclibc uclibc-source #Embedded C++ with multi-threading etc.
apt-get install witty-dev #AppServer and library for C++ web-deployment

[JAVA]
apt-get install gwis #C++ wrapper class generator to call Java objects/methods.
apt-get install jaranalyzer #Dependency management utility for jar files
apt-get install jclassinfo #Reads class files to get useful info from them
apt-get install jflex javacc #Flex and Bison for Java
apt-get install junit junit-doc #Unit testing for Java
apt-get install testng #NG unit testing with extra and best-of-breed features of JUnit and NUnit
apt-get install tijmp jmp #Java memory profiler
apt-get install visualvm #Tool for remote-admin, monitoring(dumping), profiling production/dev code, bug-reporting

[DATABASE]
apt-get install sqlite-database-browser #GUI for SQLite dbs.
apt-get install sqlrelay-dev #SQLite C/C++ APIs for proxying speeding up access to N DBMS
apt-get install unixodbc-dev #Unix port of ODBC

[XML]
apt-get install xmlcopyeditor #XML util for xsd, dtd, xslt, validation and syntax highlighting
apt-get install xsdcxx #Generate C++ classes from XSDs
apt-get install xml-rpc-api2cpp #Generate C++ wrapper classes for XML-RPC API

The Genius of Jerry Weinberg - The Psyche of Programming

Gerald (Jerry) Weinberg - The Programmer Psychologist, Writer, Systems Thinker:
          http://www.geraldmweinberg.com/Site/Home.html


This guys papers, articles and books on "Helping Smart People be Happy" are quoted on books by most of the Experts in any domain of programming.
Check out his articles/books: All books written by Gerald Weinberg on Amazon



See below for customer reviews on Amazon:

Amazon Customer Reviews/Comments on Jerry Weinberg's books:
The Classic :

Guide for Interviewers and Interviewees:

Problem Solving (useful for OnTheJob and Interviews too):

Guide for Technical Leaders:

Guide on becoming a Consultant:

Know Thyself:

Systems Thinking:

Tuesday, September 21, 2010

List of useful packages for C/C++ development on Ubuntu Linux Part 1

Linux in a Nutshell: A superb bible for looking up Linux commands and options. http://oreilly.com/catalog/9780596154493


Installing Development related packages on Ubuntu:
http://software.jessies.org/salma-hayek/ubuntu-setup.html

You can add repositories to download packages by editing this file:
vim /etc/apt/sources.list

For Eclipse C++ CDT installation please refer to this blog post.

Script:
A very useful command to record your command-line session actions for later use and replay later:
        script -a -f -t  2> timingfile  mydemo.log     #append, flush-writes, timestamp)

       scriptreplay timingfile mydemo.log    #replays with timing as per timestamps captured above
For further info see:

     man script
     man scriptreplay

Links:  
http://www-users.cs.umn.edu/~skim/cs1901/script.html
http://devdaily.com/blog/post/linux-unix/use-unix-linux-script-command-record-your-command-line-i-o

Killing Processes on Ubuntu with 3 Finger salute(How-to-Geek)

gpm: A command-line text mode clipboard manager with mouse integration.
Use left mouse button to select and copy text and paste with middle mouse button.
Super useful installation/usage howto from Gentoo: http://www.gentoo.org/doc/en/gpm.xml#doc_chap4

gpm options: http://www.oreillynet.com/linux/cmd/cmd.csp?path=g/gpm


apt-get install -y gpm #General Purpose Mouse server (clipboard manager for command line
NOTE: You can set the system default packages on Ubuntu using debconf-set-selections and debconf-get-selections

Glipper: Clipboard manager for Gnome. It retains all your text copies.
#a) Installs Glipper the multiple copies clipboard manager into Ubuntu Gnome Desktop
apt-get install glipper
#b) Right click on the "Windows Quicklaunch Bar" and Add to Panel->Clipboard manager
#c) Click on any of the previous copies to get the copy-paste you need.
I went through the entire list of packages available and selected a few below from the Development (meta-package?) of Ubuntu (10.04.1 ubuntu desktop 32 bit)

[COMMON-DEV]
apt-get update #updates your packages

apt-get install -y gcc #GNU C compiler

apt-get install -y gcc-mingw32 #Cross Compilation
apt-get install -y g++


apt-get install -y make
apt-get install -y linux-headers-`uname -r` build-essential  #C/C++ header files for building C/C++ programs 
apt-get update eclipse eclipse-pde #Update Eclipse and plugin-dev-env for connect to p2 repository of CDT for installing the CDT

apt-get install -y vim-gtk

apt-get install -y c++-annotations
apt-get install -y c-cpp-reference #Reference for C/C++
apt-get install -y cpp-doc #C++ documentation
apt-get install -y manpages-dev glibc-doc


[PACKAGE-MGMT]
apt-get install -y apt-build #GUI for apt-get
apt-get install -y ceve #Parse linux package dependencies

[PYTHON]
apt-get install -y python-dev
apt-get install -y idle2.6
apt-get install -y cableswig #Python/Tcl wrappers for C++ code
apt-get install -y boa-constructor #RAD tool for python/wxwidgets

[PROFILERS]
apt-get install -y gprof
apt-get install -y kprof #KDE util for gprof analysis/viewing
apt-get install -y valgrind
apt-get install -y cachegrind
apt-get install -y alleyoop #Valgrind related
apt-get install -y valkyrie #GUI for valgrind
apt-get install -y kcachegrind #KDE Gui for valgrind
apt-get install -y sysprof #System-wide CPU profiling
apt-get install -y systemtap systemtap-client systemtap-grapher #Collect data on a live running linux box
apt-get install -y syslog syslog-ng #Interprocess application logging facility


[TESTING]
apt-get install -y libcppunit-doc libcppunit-dev  #C++ unit-testing lib

apt-get install -y subunit #Run and Save remote test results for later use
apt-get install -y stress #Stress test your computer with heavy load
apt-get install -y httest #Simulate client/server actions with pattern-matching for test validation
apt-get install -y fuzz #Stress test s/w with random inputs

[DEBUG]
apt-get install -y gdb gdb-doc #Debug programs
apt-get install -y gdbserver #Remotely debug from another system where GDB is installed.
apt-get install -y ddd #GUI for gdb
apt-get install -y bashdb #Bash debugger
apt-get install -y zshdb zsh-dbg #debugger and debug symbols for ZSH
apt-get install -y gcov #Source code coverage analyzer
apt-get install -y ggcov #GUI for gcov Source code coverage analyzer
apt-get install -y lcov #HTML and directory-wise view of gcov output

apt-get install -y collectd-dbg #statistics collection and monitoring daemon
apt-get install -y wireshark wireshark-dev #Network debugging
apt-get install tack-dbg #Diagnostic tool for correctness of terminfos.
apt-get install -y tau tau-racy tau-examples #Multithread/multiproc tuning and analysis with GUI
apt-get install -y leaktracer #Simple C++ leak tracer
apt-get install -y electric-fence #malloc debugger
apt-get install -y duma #Fork of electric-fence library with added features
apt-get install -y winpdb #GUI debugger for Python
apt-get install -y gtkparasite #Python interactive debugger for running GTK+ code.
apt-get install -y happycoders-libdbg-dev #allows modern debugging paradigms for Large codebase
apt-get install -y happycoders-libsocket-dev #
apt-get install -y bless-hex-editor #Hex Editor
apt-get install -y hexdiff #Visual hex difference analyzer
apt-get install kompare #diff and merge util for files and dirs.
apt-get install -y tkdiff #diff/merge util in TK
apt-get install -y ht #Viewer/Editor/Analyser for all kinds of executables.
apt-get install -y abi-compliance-checker #C++ sharedlib binary compatibility checker
apt-get install -y tesq #Decoding of terminal escape sequences used by Unix Terminals

Wednesday, July 11, 2007

BookReview: The Mythical Man Month by Frederick P. Brooke

Huge insight into what makes a very big software project successful or Not....

"There is no silver bullet...." except for good management of the design and implementation.
"12 doctors can (only) deliver a baby in 10 months..." are some quotes to remember from this book everytime your manager asks you to do yoga with the project plan.

BookReview: The Practice of Programming by Brian Kernighan and Pike

"Good design is what happens when you can take out no more from your creation without breaking it". This is the philosophy of this book.

It teaches you the simple principles of (as shown on the cover)
1) Clarity
2) Simplicity
3) Generality

Thank you Kernighan and Pike for teaching these Zen like aspects of programming.