Thursday, July 28, 2011

Good Articles on Internals of C++ and Unix

A) C as a glorified assembler:

"C a glorified assembler" or "C a generalized assembler"

Actually I only really started understanding C when I understood 2 things about it.
1) Anything you code in C finally translates to assembly language.
2) How your compiler interacts with the O.S. loading/linking mechanism.

You can easily test this.
Use the 'gcc --save-temps ' feature of your gcc compiler to see the assembly language output.

C program to assembly code translation
>>>>>>>>
C program
>>>>>>>>
extern int printf(const char *, ...)
int main()
{
 printf( "Hello, World!\n");
 return 0;
}

#Retain temporary intermediate files generated during preproc, compiling, assembling etc.
gcc --save-temps test.c

>>>>>>>>
Pseudo-Assembly (see link for more accurate example)
>>>>>>>>
.data                              # data section starts at say 0x100
msg ds "Hello, World!\n"           # define string with label as msg and content as "Hello, World!\n"

.text
push sp                            # save the current stack pointer
push msg                           # push the function parameter onto stack
call _printf                       # make the library call
ret   0                            # essentially return 0 as errorvalue to bash/cmd.exe

>>>>>>>>

Since the C language is portable from one platform to another it kind-of acts like a virtual machine (without byte-code generation of-course).
The same C program will compile on different machine-compiler pairs to the specific assembly language and O.S. of the platform in question.

This makes for  chip/hardware specific assembly(Intel x86 or Motorola or ARM ...) portability.

Write Great Code :

What happens when you run your program?
The dynamics from Compilers to O.S. to Assembly language to Memory to CPU.
This book covers it all

You have 2 choices:
1) You can choose to spend years garnering small nuggets of internals of your Program and how it runs on your machine. You'd need a few books each on Operating System, Linker Loader, Assembly Language, Compiler Design, Hardware.
Next would be loads of patience to connect all this into a chimera/frankestein-of-sorts (horribly put together)
OR
2) You could read it all here - Integrated into a beautiful and coherent whole written by a master.

Write Great Code, Volume 1: Understanding the Machine
Write Great Code, Volume 2: Thinking Low-Level, Writing High-Level
The Art of Assembly Language


Refer:
  1. C to Assembly Translation Article from EventHelix
  2. Inside the C++ Object Model : http://techtalkies.blogspot.in/2007/07/stub-bookreview-c-object-model-by.html
  3. X86 Calling (and stack cleanup) conventions: http://en.wikipedia.org/wiki/X86_calling_conventions
    X86 Disassembly Wikibook (download as pdf): http://en.wikibooks.org/wiki/X86_Disassembly
==================================================================

B) Interaction with O.S. loader/linker/virtual-memory:

The specific C compiler would also link the O.S. specific C runtime properly.

So gcc which is a portable compiler would link in.
a) mingw C runtime on Win32
b) linux C runtime on Linux.

The same C program would also convert to correct assembly/runtime on
Watcom C++, Visual C++ compiler, AIX, HP/UX compilers etc.

Note:
a) printf() comes from the C Standard Library
b) C runtime library contains the startup code which executes before and after main

Refer:
1) C++ runtime environments on HP-UX
2) Internals of C/C++ compiler implementations
3) How to use debug C runtime to debug your application
4) How to write a minimal Kernel in C (a very simple Hello world program which also describes the libc C runtime library)
5) Understanding System Calls
Note:
The C standard library calls like open(), close(), putc(), getc() etc actually just forward the call to O.S. system calls in the linux kernel. The C runtime actually maps a stdlibrarycall to an interrupt vector i.e. system call index number in a lookup table. Then it uses a software interrupt to call the system call (transfers from user mode to kernel mode and back)

==================================================================

These are a set of articles that take a look at what happens inside C, C++, Unix internals as we compile and run our programs. The info here is invaluable to debug compile errors, runtime errors and memory issues. You can go through these in your spare time. Take a print out to read in your spare time and go through these articles.

  1. What Happens When You Compile and Link a Program
  2. What a Compiler Turns Your C Code Into
  3. Virtual Base Classes Implementation
  4. How the C++ compiler mangles/decorates function namesUnix And C-C++ Runtime Memory Management For Programmers
  5. Under The Hood Look At Operating Systems Internals with Windows and Linux : http://techtalkies.blogspot.in/2010/08/operating-systems-and-linux.html
  6. mmap is not the territory (Part 1): http://techtalkies.blogspot.com/2010/09/mmap-is-not-territory-or-mapfail-sigbus.html
  7. mmap is not the territory (Part 2): http://techtalkies.blogspot.com/2010/09/mmap-is-not-territory-part-2.html

==================================================================

Wednesday, July 13, 2011

Remotely editing files with ssh, sshmenu, bcvi and vim

No need to install your favourite vim settings etc on the remote server just use bcvi to start gvim on your workstation.
It uses scp internally to create an editing session from workstation to server

Transparent multi-hop ssh
sshmenu setup howto
Remotely editing files with bcvi and vim
Editing remote files in vim via scp

Browsing XSLT with Vim by adding a Custom Language to Ctags

XSLT Source Code Browsing with Vim and Ctags

Creating a tags file

1) Test out the regular-expressions for your custom language:
$> egrep 'pattern' *.xsl
2) Copy-paste custom language with the above regular-expressions into ~/.ctags.
$> vi ~/.ctags
--langdef=EXSLT
--langmap=EXSLT:.xsl
--regex-EXSLT=/--regex-EXSLT=/--regex-EXSLT=/$> cd srcdir
$> ctags -R *
$> vi ./tags        #check for tag-entries containing the search patterns
$> vi test.xsl     #Use Ctrl-] to jump from a pattern usage to its definition, Ctrl-T to jump back
Note:
1) Above solution for templates only uses xpath of match="xyz".
The search-key for the template (displays taglist for templates with same xpath).
Potentially a template is uniquely identifiable using match AND mode:

Monday, July 11, 2011

Apprenticeship Patterns: Patterns and Anti-Patterns of Problem-Solving

The Intellectual Side of Problem-Solving:

(Borrowing heavily from AI terminology)
Imagine a 3D Cube containing the start-state, the goal-state, the problem-space and the solution-paths.
You're essentially scanning a problem space for a solution i.e. a combination of sub-steps which finally lead to goal.
1) Explore the problem space first i.e. try out things
2) Identify what worked or is promising and extend the branches into the sub-problem spaces
3) Recurse over the sub-branches until you've covered enough of the problem-space.
4) If the volume of the problem-space is vast it may help to BackTrack from Goal towards the problem state.

This reduces the volume of problem-space to be probed for possible solution-paths. This is known an "Alph-Beta" Pruning of the (Decision??) Tree.
//TODO: Add Diagram of 3D Problem Space and Decision(??) Tree.


In Nature, Lightning follows a very similar method of Back-Tracking.
It seeks out the easiest path between cloud and ground through an insulating volume of air.
Actually there is an initial weak backtracking path from ground to cloud, followed later by a strong return path.
//TODO: Add picture of Lightning fingers

The Emotional Side of Problem-Solving:

Basicially we must first ask "Why" is problem-solving so important?
Not all problems we face are life-threatening to require immediate attention.
So then what is it that makes solving them so important in the first place.

Refer:
1) Psychology of Computer Programming by Gerald Weinberg
2) Emotional Intelligence

Experience - coming to terms with "immediate failures" vs. "future pay-offs"

1) Try out many things to get a better solution or a solution which appeals to your way of doing things.
2) All the things you try won't/can't succeed at least immediately.
   Mostly many of these "dead-end branches" are just waiting for the right time to flower. You're actually building up a fund of ideas and efforts which will help you overcome some other obstacle.
   Example:
   Unix was a result of dissatisfaction with implementing Multics which itself was a replacement for some other O.S.
   Ken Thompson went on to implement ideas for distributed O.S in CODA, Plan9 etc. 30-40 years later these lessons learned and 'the-roads-tried-but-not-taken' have resulted in writing a 'go' language.
   It's used in Google which puts those same ideas refined over time into concrete working programs.

3) Watching my son getting frustrated when his cycle got stuck in a corner.
   He'd start screaming and banging the cycle into the wall.
   Beating it with red-face and small fists.
   The tears of frustration and incomprehension as to why the universe conspired against his cycling.

4) This felt so much like my own frustration with my own efforts.
   The perceived low return-on-investment of the "meagre" results.
   So many ideas half-implemented, unimplemented and ALL the wonderful things not even attempted i.e. 'the-road-not-taken'.
   The universe almost conspiring to thwart each hope and attempt at magically succeeding at all things.

5) Frustrations comes from other people's expectations.
   And more so our own awareness of not "measuring-up" to their and our own expectations.
   Perfectionism adds heavily to an already sticky situation.
   A feeling that time is running out.
   The impending joining of ranks with washed-out oldies.
   A feeling of helplessness and hopelessness that things can't/won't change no matter what.  
   Unable to leave coding and unwilling to change to management (manipulating people not machines).
   Fear of becoming obsolete, of not living up-to the dreams, hopes and self-expectation built up over the years.

----

Patterns: The Middle Path between the Emotional and Intellectual Sides of Problem-Solving

"Why patterns work?!!" AKA "All I needed to know about Patterns I learned in Kindergarten"

Patterns force you to Think and NOT concentrate so much on the Feeling part (which is the cause of frustration)
a) Read, understand, see how the Pattern fits the Problem.
b) Re-evaluate the problem itself.
c) Try carrying out/coding the Pattern.
d) Customizing the Pattern to the situation.
e) Utilize your Feeling side by reflecting on aesthetics as pattern application is a very qualitative activity.

Patterns are primarily a set of ready-made formulas (cookbook-style) for successful problem-solving.

Raise Flagging Hopes:

These are also sufficiently magic-endowed by hearing good things from others etc. This helps you get over the inertia of hopelessness or feeling stuck at a problem. "Who knows this just might work - I've nothing to lose."

Abstract the problem solving process.

De-personalise the problem takes the bite out of it.
If the trial fails it's just the Pattern failing NOT you.
You're fore-warned to try out different patterns for different circumstances.
So you just try some other Pattern or just do something else.
Since the Pattern comes from outside, you give yourself time to understand it.
This is the crucial part where you become accepting of time and effort to customize the Pattern to your particular situation.

Conclusion:
Patterns are very effective Devices for Distraction from Result and Focus onto Process instead.

  1. Failure Modes of Hacker Schoolers : https://www.hackerschool.com/blog/66-four-failure-modes-of-hacker-schoolers

Friday, July 1, 2011

How-to Repair an Oracle Database using Putty, SQL Developer GUI and some trial-and-error

I had a problem earlier when I made some changes to the test/dev DB and didn't know how to revert back to correct state.

Issue:
I couldn't delete a row in a table containing incorrect data as the row's primary key was acting as a foreign key in some other table. If I deleted this row it would violate the integrity of the other table(s).

Process:
So I took a backup of the entire DB (just in case I needed it  and I did need it!!).
Dropped the constraints, deleted all the related rows (or something like that)
Recreated the constraints and the DB was restored to correct state.
In the process I learned quite a bit about DB's.

Tried out individual commands in the GUI.
Then copied multiple commands into a script.
Ran the scripts, corrected any errors till I got a working script.
Liberally used transaction and rollback to help in the trial-and-error process.

----
I don't remember the specific details but basically
I used a GUI DB browser (SQL Developer) to connect to a Remote DB by port forwarding from my Unix machine.

Working with command-line client is ok if you're really good at it and know all the command syntax etc.
But using a GUI browser really speeds things up as it can manage things like transactions, rollback etc all at the click of a button.

Database GUI SQL Developer allows you to
+ Try out individual commands (with rollback) to see if you're getting the right results.
+ Copy paste small commands into a bigger sql script (with start transaction at start and commit or rollback at script end)
+ Browse a remote DB in GUI mode using a DB connection (using ssh tunnel),
+ backup the entire database (for safety)
+ maybe try out things on a local database first, before attempting the same on the actual DB.

You can export the original DB. Import into a local DB. Try out the operations to check it works.
It may also help to create try out your modifications on a duplicate DB on remote machine instead of touching the problematic DB directly.

1) SQL Developer Setup and Tunneling (port forwarding) instructions are available here.
http://www.cs.tau.ac.il/~boim/courses/databases2009/slides/moreinfo/connection-guide.htm#_Toc150420090
http://www.madirish.net/?article=152
2) You may need to tweak a few things on your Unix machine to get the tunnel working though.
3) Finding somebody in your/other teams who's good at Databases will be a big help.
    Local person who can see what the problem is and give quick suggestion is always better.
4) Searching on error messages would help find howto's for fixing the problem.
5) Search error messages in forums like stackoverflow.com will help to quickly find the info or relevant links.

HTH.
----
Oracle SQL Developer

PDF - Oracle SQL Developer User Guide
Oracle SQL Developer at wikipedia
Oracle SQL Developer Tutorial
Connecting Oracle SQL Developer to MS SQL Server
How-to Export Tables/Entire DB with Oracle SQL Developer

Oracle SQL*Plus command line interface (CLI)

SQL*Plus CLI FAQ
SQL*Plus CLI command reference
SQL*Plus Tutorial