Misguided Agile - "Agile" != "Faster"

Yes, "Agile" does not look same as "Faster", it does not even spell same and it does not have same meaning. If you see a project on Agile causing lots of pain to everybody around then a question should be asked - why are we doing Agile?

And if in the answer, project owner mentions word "Faster" then chances are high this is misguided agile, a guided missile that is going to land on your project and team. This is true even if Faster is the last word of the fifth reason for choosing Agile. It is like hero word, that takes the whole screen of mind even when other words exist.

In my view Agile is about getting software right. Consider a team building a software in isolation and bringing it out to customer after a year only to be denied acceptance. Customer says, "that's not what i wanted ...". There was a time when this was not only happening but was happening a lot. And at that time we wanted a way to collaborate with Customer, to create right software, a software that customer wants and will use.

Fast forward few years and now we have a situation where even if you don't have a customer in sight, even when things can be completely defined beforehand, we still need Agile because it has started meaning Faster. It should be ability to respond to changes but it is about getting more quickly from same resources. While Faster might be and generally is a side effect of Agile, the basics have to be kept in mind. There are subtle changes to Agile that make this process lose its beauty and make this a ugly daily reporting tool.

Come Home (Software) Engineer

Everything that comes our way, changes us... many a times we do not see the change. Mind is sticky - particularly when its money or company. Money changes our beliefs, make us risk averse, puts on a track that we did not want to be in. Equally powerful influence is of people we work with, the company we keep. I realize that i picked some of the habits i actually hated. The sum of all this is that one is on wrong track, wrong role at job and is behaving differently from one wanted to. Lost Home. Hmmmm ... time to go back home.

So, why did i like my job of software engineer? Here is the list to check and rank,

Solving hard problems
Understanding things
Making Money, financial security
Being powerful
Build word class products
Being most intelligent in group
Be a leader, create leaders
Business alignment
Build big architectures - like big bridges in civil world

To me this quote shows the spirit of engineer:
Scientists investigate that which already is; Engineers create that which has never been.

To me creativity, knowledge and solving problem come at the top. And i like solving problems that take some time - not the ones that are just memory recall away. And my day job should be filled with these.

It is to recall that we software engineers have a pact with GOD, He doesn't write code and we don't do miracles. From being a super-hero it is to get more humble, low on ego. As Dijkstra says, "The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague."

Hiring people to write code to sell is not the same as hiring people to design and build durable, usable, dependable software.
--Larry Constantine

It is to go back to be an engineer from a hire.

What is Architecture?

Architecture is being used with so many adjectives that i find it worthwhile to think about what is architecture. Business architecture, architecture of democracy etc give a feeling that architecture is about "essence". To begin with it looks like executive summary and it even stretches to look similar to a theory of science or axiom of maths. What is a mere description of something (or say specification) vs an architecture is not very clear. Also it not clear whether it is the description or it is by the use of that description that an architecture should be demarcated by.

Gregory Chaitin in his book "META MATH" differentiates between description and theorem. In simple terms theorem somehow should take less amount of information encoding for representation than the description. And as we find a more concise theorem the elaborate one is not needed.

Do we call Einstein's theory of relativity as architecture of astronomy. Perhaps not. Next is the level where we group of theories and facts together and call it architecture. Do we mean the principles when we say architecture of democracy?

Or as in a formal system, the alphabet, production rules and axioms which together describe the system constitute the architecture of that system.

Or do we talk of architecture only when we build systems? That is, if there is nothing to architect, there is no architecture. I can't think of an example where this is not true.

Architecture is a set of principles and techniques, as written in the book "Art of Systems Architecting"

Anyway when we discuss of this in Information Technology/Software Engineering field the definition is little narrower.

Here is one of the definition:
Architecture- Orderly arrangement of parts; structure:

Another one from IEEE
Architecture. The organizational structure of a system or component
– IEEE Glossary of Software Engineering Terminology, 610.12–1990

Or a more precise one from IEEE 1471:

architecture: the fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution.

This means we need,
A System
Environment
System having components
Components interacting with each other

And we are talking about design and evolution of this system.

GC performance

On a single cpu, regular laptop,


New GC reclaim : 115MB in 0.0285765 -> 4024MB/sec
New GC scan: 940MB in 0.0285765 -> 32 GB/sec
Full GC: 860MB in 6.2 sec -> 138MB/sec

Groovy Bug: Anonymous class

Human Genome sequencing: computational aspect

Business,
… Knome.com, currently provides genome sequencing services but the cost is about $99,500 per genome
… At the end of February 2009, Complete Genomics released a full sequence of a human genome … will contain approximately 80,000-100,000 false positive errors in each genome
… In June 2009, Illumina announced that they were launching their own Personal Full Genome Sequencing Service at a depth of 30X for $48,000 per genome
… In November 2009, Complete Genomics announced that they are now able to sequence a full genome for $1,700… Complete Genomics has previously released statements that it was unable to follow through on
… In March 2010, Pacific Biosciences said they have raised more than $256 million USD in venture capital money and that they will be shipping their first 10 full genome sequencing machines by the end of 2010. … by 2015 … $100 per genome


Technology and open source software
http://www.cbcb.umd.edu/research/assembly_primer.shtml
http://sourceforge.net/apps/mediawiki/amos/index.php?title=AMOS

Problem statement in computing terms
A long string of 4 characters (A, T, G and C) is broken down into many pieces. Some of the pieces may be missing too for biological limitations and human error. This has to be reassembled by looking at sub-sequences that look same. This is error prone, but here is good news.
Multiple copies of the long string were shredded and it is shown that 8-10 copies shredded is good to get a very good genome sequence done.

State of Art
Greedy Algorithms
Overlap-layout algorithm – Hamiltonian path
Eulerian path
Align-layout
BAC-by-BAC (hierarchial sequencing)

… June 2008 the quantity of purity-filtered sequence data generated by our Genome Analyzer (Illumina) platforms reached 1 terabase, and our average weekly Illumina production output is currently 64 gigabases…


Looks inspiring?

-bala

Bug in java's System.out.write with large byte array

Here is a program that just writes lot of "a"s. Compile it and run it with
java B 60000
It doesn't write. Now try,
java B 50000
It does write

----------------------------------------------------------------------------------------
import java.io.*;

public class B {
public static void main(String[] args) throws IOException{
StringBuffer sb = new StringBuffer();
for(int i=0;i {
sb.append("a");
}
byte [] b= sb.toString().getBytes();
System.out.write(b);
System.out.println("Number of bytes: "+b.length);

}
}

---------------------------------------
My colleague Venkatesh pointed out that some error has occurred and because this method doesn't throw an exception, it sets the error,

System.out.println("Error=" + System.out.checkError());

J2ME jar size

If you build application, sooner or later you will hit J2ME jar size problem. if you use libraries like LWUIT, you start with 90KB. Then you add resources (images, strings) of 130KB. And then your app takes few more hundred KBs. This will hit you when you realize that some of the phones like MOTO RAZR will only support 400 or 500KBs app sizes and some phones will even restrict you to 200KB.

Statment level LOC, Method and Class are three variables that we played with. Keeping LOC and class constant we changed methods. Here are the numbers,

Methods	Jar Size (Bytes)	Delta for additional method
1	1301
2	1320	19
3	1339	19

Keeping LOC contant, we added classes (and in first three additions a new method also came about)

Methods	Jar Size (Bytes)	Delta for additional method
1	1301
2	1738	437
3	2167	429
4	2605	438
5	2913	308	Empty class added

So, for every method add a 19B and every class add a 308-400B hit is there.

Here is the class that we were playing with,

public class HelloWorld {
public void say()
{
System.out.println("H");
System.out.println("e");
System.out.println("l");
System.out.println("l");
System.out.println("o");
}

}

For methods we had,


public class HelloWorld {
public void say()
{
System.out.println("H");
System.out.println("e");
System.out.println("l");
System.out.println("l");
System.out.println("o");
}
public void say1()
{
System.out.println("W");
System.out.println("o");
System.out.println("r");
}
public void say2()
{
System.out.println("l");
System.out.println("d");
System.out.println("!");
}

}

Is Resource view (REST) as powerful as Object view?

We are comfortable with object view of the world. Everything is a class and each class has data members and method members and has relationships to other classes. In plain English we have nouns and verbs. If you are user of a object API, you get the right noun instance and call the verb on it - get the duck instance and say "fly". We can model bank domain to rocket domain using this.
Modeling this as a service as in SOA or method call as in RPC is also straightforward. Calling a method on a remote object is possible (CORBA). And that in turn is equivalent to a method call which has object identity as an argument.
Is Resource view equally powerful? I thought a service "SendEmail" might be tough to be seen as a Resource that has a universal interface (HTTP method). http://www.prescod.net/rest/restmail/ shows that this is possible. Does it look contrived?
Well, we could analyze this with two approaches. By making the whole world look like resources we are converting a set of "nouns and verbs" to fixed few verbs and lots of nouns. I as a human can do many things - run, sleep, think etc. How do i reduce it to nouns? One way is state model approach - i see things as states and each state is addressable by URI. So i have a running man, sleeping man etc. Is a state a resource? Not really.
Another way to look is that we are only interested in representation and transfer of representations between client and server. So you get representation of a resource - which is a man. And you update the resource's representation on client, say you change it from running to sleeping and then you call update onto the resource again. It is like you take a toy look at it and turn the switch on and put it back - the toy starts playing. So any object model can have an equivalent resource model where object are resources, data members are part of representation of object and method calls are triggered on server side by client modifying the representation of object and updating the server. This has an integral part, serialization of object into an representation that can be shipped to far away place and a change could be applied with a returning representation.
On a broader view REST looks like shopping experience. You go to a shopping mall and all shops are resources. You hop from one to another - your state transfers, resource stays where they are and you only get a representation of resource (godown is the actual resource).
It also looks as if REST is remote object protocol - except that methods are not called directly. They however could be effected by state change. How does document style services differ - well, they look more closer to REST than RPC style services. The equivalence is when there are seven kind of services - Get, Post, Head, Put, Delete, Trace and Option.
This again takes me to LISP and turing machine. Anything can be modeled by a simple list or a set of read/write/mark instructions. My faith in higher level ideas like REST is strengthened.

REST is primarily for remote communication. One will not implement their business logic entirey using just resources - it will be object model. REST representations are photographs of these object model - one object at a time.

FREE stack

Flex, REST and Enterprise backEnd

EE could be JEE or Jersey based REST provider. Back-End programming language is Java.

What one should be aware of:

1. Flex doesn't support PUT, DELETE. So if u try hooking xmldb as backend and Flex front-end it will not be out of box.
2. Flex doesn't give u status code back in HTTP response
3. Your major concerns would be how to integrate new app built on this stack with existing infrastructure - especially identity management.

What are some of the pluses that stand out

1. With flex you no longer worry about lower level UI aspects. It is far more faster to develop UI with flex. And it is building bridges with other technologies for backend.
2. With REST you are ready for a developer platform right from the begining.
3. With REST you are web ready. You can take on scale, performance, mashup etc easily.
4. With Java you have a proven platform for doing enterprise application backend.

Sudoku and Architecture creation

Here is how i solve Sudoku puzzles. I look at the whole problem and try to see from my memory if i have ever seen a pattern like this or even a sub-pattern. If i find one then i try to see if whole or part of or learning from the old solution could be used. In either case i proceed as below. Let me call it pattern approach.

The simplest of sudokus have a weak part of the problem. If you get that part and solve it, suddenly the whole sudoku solution comes up on its own like a thread that has been pulled from a woven cloth. Sometimes it is multiple threads, so when u take out one thread, you have to find the start of another thread and so on. Same might be true when u are solving a huge set of simultaneous equations with pen and paper. You look for equation with least variables or some peculiar form that has clue in it. Looks like a detective approach. Let me call it weakest part of problem approach.

In some of the harder sudokus author removes all the starting points. One can however find a starting point that needs assumptions of a small part of initial solution. One starts, tries it and backtracks on findind deadend. Then starts again with a different assumptions. Let me call it trial and backtrack.

Given the problem specification (i.e. all requirement and context analysis done), an architect tries a similar approach:

1. Pattern approach
2. Weakest part of problem approach
3. Trial and backtrack approach

Is this online site alive

I have tried lots of social networking sites. But it was only today that i tried plaxo. Here is the first experience on this site. The first wow on a social networking site is finding people you know who are there. Connecting them brings the site alive. What differentiates an online site from a remote data store and remote application delivery site is that it connects with other's data.
There was one more thing that brought more life to plaxo. It collected my internet footprint - it searched blogs and got entries authored by me. Ofcourse a different UI adds to all this.

Capabilities that an Architect picks up

One starts of as a learner of technology
One masters programming
One masters problem solving
One gets hang of technologies - to know trends
One if able to apply technologies to solve problem
One is able to conceptualize big (in scale) solutions
One then learns to develop a plan for achieve solutions - what is the order of building pieces
One learns to bring in business into picture - buy vs build
One learns to adjust plan as business situations change
One learns how to lead execution of an architecture
One learns how to sell architectural proposals
One learns how to teach
One learns to lead and facilitate - let everybody realize their potential

My belief is that there are more of such learning steps... rest of mountains in the journey would be visible once one crosses these ... i am still stuck in one of the mountains ...A lifetime is not enough to master all this ... that will keep the excitement in the journey!

Troubleshooting JBoss HA

JBoss HA relies on JGroups for communication. It is very easy to work with, but if it doesn't it gets little difficult to troubleshoot. This is what i realized last week when i tried to set it up on two machines. Here are the steps you can follow if you are stuck:

1. Check whether multicast works - http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Clustering_Guide/4/html-single/index.html#id3016498
Want to read some theory on multicast - http://www.ibiblio.org/pub/Linux/docs/HOWTO/other-formats/html_single/Multicast-HOWTO.html

2. Check if JGroups is able to establish communication channels - http://myt.ag/URLWeb.aspx?email=steve%40fooworks.com&url=http%3a%2f%2fwiki.jboss.org%2fwiki%2fWiki.jsp%3fpage%3dTestingJBoss&sn=

3. Enable logging in JBoss to see more info - http://www.jboss.org/community/docs/DOC-9164

4. Make sure that you are not booting JBoss with -b 0.0.0.0. Give IP address of machine instead of 0.0.0.0

Three types of architects

Enterprise Architect
Infrastructure Architect
Software Architect

from : http://msdn.microsoft.com/en-us/library/cc431351.aspx

Another one of Architecture

Architecture is user experience
Architecture is negotiating and bargaining
Architecture is technology
Architecture is form
Architecture is communication
Architecture is artful
Architecture is agile
Architecture is the creation of a better world

from : http://msdn.microsoft.com/en-us/library/cc304371.aspx

Architecture of a process

Evolution is a process that makes lot of us wonder. In the beginning there was a condition created ,that included lightening, that resulted in amino acids. These then got together causing proteins and cells to be formed. And from then on two steps followed continuously - mutation and reproduction. Huge number of repetitions of this caused such a variety of species, human being one of them. If God designed the process, He probably did not design it with Human Being or any other specie as the culmination. He designed a process that would go on and improve. Wonder what would happen if any one of this steps was missing the process would not yield. Lot of repetitions of this step - much like fractals - causes results that make us wonder. It would simply be not possible to design human specie as an independent design.

Assume that God was in the process of designing this. In the first release of design He would make reproduction as primary requirement. Once cells were designed, He would have then thought of bringing change in the cell to make evolution possible. Mutation is also very carefully designed, because any mutation we try artificially results in certain death. As a software engineer i am inclined to think of God writing a program (in assembly language with UAGC blocks, a DNA being a module). Every time we muck with the code the program fails. In a computer program as long as i write syntactically correct program, the program will run. It may not achieve anything meaningful, it may not terminate but it will run till an 'exit' is called. God's designed the virtual machine (that runs the program) that terminates syntactically correct programs. This deviation from randomness to get optimal results is also evident in "scale free design - http://www-news.uchicago.edu/releases/06/060807.networks.shtml.

Architecture is not just a structure that one sets in the beginning and then it stands touched. It is more like a process. Once the principles or say fundamental steps are followed over years product gets better and better. If Amazon asks its engineers about what happens millions of people access a program, it is an example of architecture of a process that results in scalable systems. I would say REST is also another example of same. SOA Choreography gives a base wherein one can create such an architecture.

Choreography and Orchestration

... I thought, described the same concepts. Here is what people are talking about,

Orchestration always represents control from one party's perspective. This distinguishes it from choreography, which is more collaborative and allows each involved party to describe its part in the interaction.

In orchestration, there’s someone — the conductor — who tells everybody in the orchestra what to do and makes sure they all play in sync.

In choreography, every dancer follows a pre-defined plan — everyone independently of the others.

Orchestration defines procedure and Choreography defines protocol.

Orchestration is generally a full-on execution mechanism for recursive composition of services (aka WS-BPEL). Choreography describes the observable behavior that makes up a contract between a set of peers. It doesn’t say how it is to be achieved just what can be observed. Orchestration is much imperative and choreography declarative. They can work hand-in-hand.

What is the relationship between choreography and web services composition? ... A composite service is a system that arranges existing services in a workflow of some sort (or process if you want) and deploys this arrangement as a service of it’s own. The added value of this system therefore is a previously non-existing functional construct. The key point is that the composite service is indeed a composition and not a simple collection of existing services and that within the composition the service invocations must! be coordinated, e.g. through choreography where the control is centralized. So choreography is a mechanism for coordination (control) in a composite service.

What is Web Services Flow Language? WSFL supports two types of composition and choreography: Flow models: describes business processes; Global models: describe overall partner interactions. Flow models - Describe how to choreograph the functionality provided by a collection of Web services to achieve a particular business need Global models - Describe how a set of Web services interact with each other. ... it was mentioned somewhere that WSFL is superseded by BPEL.

WebService composition can be achieved view two patterns - hierarchical and peer-to-peer. Hierarchical interactions are often found in more stable, long-term relationships between partners, while peer-to-peer interactions reflect relationships that are often established dynamically on a per-instance basis...

Beauty in Software Architecture

Some of the architectures look beautiful to me. I wonder what are some of the properties which make a Software Architecture beautiful and even if not beautiful ... very attractive. To be attractive something should be easily understood. Fewer concepts, fewer new concepts and simpler relationships between those concepts would constitute simplicity. If something is complicated you have to force your mind to understand it and chances are high that it will not appear beautiful unless the payoff of effort is higher. Later brings us to "surprise" aspect - core of Application Server like JBoss has nothing to do with applications or JEE, it is JMX, interested in only starting/stopping services.

Then something that brings a wow is, how much thinner the software can get. A big application server can have a microkernel only 50K size. That is beauty. For something flexible enough to grow very very big and shrink very very small, the core pattern has to be simple. And that pattern has to be repeatedly applied. And with zero exceptions. That catches attention.

And finally, recurrence. That always is attractive. The services of JBoss are MBeans loaded by core, but core itself is MBean. All this takes me to Mandelbrot ... aren't principles of what is beautiful same?

Model Execution Platform - build vs assemble

As one decides to embark on the model driven development journey, one has to answer how a platform would be created that would execute the model. By executing the model i mean, converting it into a runnable code or interpreting it. Performance is something that i have not personally seen examples of in MDD. Till we really get a hand on performance it is difficult to decide which way one should go - build or assemble. Given that performance goals are honored in both approaches, what would be next deciding factor.

Building whole framework from scratch is just impractical given time to market. Assembling open source and commercial softwares doesn't work as these things do not work seemlessly. There are MDD platforms being built, which i still see organizations will not bet on.

The approach then some cautious organizations take is to take open source / commercial products and write code to create the platform. And slowly one goes overboard developing their own versions of what is available, just because as implementation gets deeper the stakes get higher and requirements from new component increase.

There is one bottleneck which is increasingly becoming important and it is learning curve. Learning curve needs to be managed as one builds proprietory stuff. Learning curve is important for following reasons,

1. Small learning curve means higher productivity. There is more room left in ur RAM to look into other things than managing your platform.
2. Faster ramp up of new hires means tolerance to churn.
3. Ramp up experience also ends up positioning the platform. An easier platform gives an impression of well thought out and well designed platform.
4. Smaller learning curve might be the only way as systems become complex.

Now smaller learning curve can be achieved by either hiding details from people or using well documented, well designed components. Hiding details and presenting a good front to system take lots of effort not only in development but maintenance too. Well documented and well designed compnents that confirm to standards shorten the learning curve. Open source components have the benefit of good documentation and almost always adherance to standards. But then if u get a similar component for money, chances are higher that it would be good quality. But then documentation is always more for open source.