Lean and Agile Development, the Big Picture

Lean and Agile Development has been around for many years now and numerous practices have evolved around various phases of software development. There has been a proliferation of consulting services around agile and lean. Fortunately or unfortunately, not many methodologies have evolved - apart from XP and Scrum.

Most articles or books on Agile development start with a reference to the Agile Manifesto as a precursor to the practices followed. But to obtain a better understanding why agile practices are followed, we need to understand why individuals and interactions are valued over processes and tools, why working software is preferred over comprehensive documentation, why responding to change is more important than following a plan and why customer collaboration is considered paramount.

It is common sense that businesses exist to maximize profit, but what is uncommon about some of the software methods of the past (Six Sigma, ISO 9000) is how following those would result in better business profitability. ISO and Six-Sigma touted that improving quality and reducing defects would result in better customer satisfaction and higher profit. But a defect-free product that arrived years late does not improve profit.

Lean thinking and TPS is adopted to make sure products are delivered with reduced cycle times, but the base metrics are derived from the time-cost-quality triangle and optimizing one usually results in sub-par performance in another. To run an optimum business, we need a metric that follows the principle of optimal-substructure, that can be applied to all sub-components of the business and optimizing this metric in each area should help us optimize the metric for the whole business.

It is a fundamental fact that ‘asymmetric information’ leads to profitability. Insider trading in stocks, counting in card games, testing in drug research, A/B testing of web products, price comparison in online shopping, and many more such instances try to improve our ability to have better information and ultimately profit from it. Better information results in better decision making and higher profit. Information that is available to us much before it is available to the rest of the world can be considered asymmetric and enhances competitive advantage. In the software development process, the ‘product’ or outcome of various activities is in fact information in various forms - requirement documents, design documents, code, test specifications, reports, and so on. In the context of a business unit developing software one can think of better information delivered faster to the customer as an optimal metric that can be used to measure each individual activity We can call this metric Information Rate.

When we think of the software development process as an entire system, the best form of information one can deliver to the customer is actual working software. During the process of developing working software (information), there may be opportunities to improve the quality of this information by continuously interacting (customer collaboration) with the customer. If the customer thinks there is better opportunity for generating better information by introducing new inputs, one must respond to changes. We can now understand how meaningless it would be to deliver information that is outdated (working software that requires change). Lastly delivering software requires a set of steps where information is handed over by individuals from one step to next. For each such step, the individuals from the previous step act as customers and hence to improve customer collaboration, one needs to improve on individuals and interactions.

Lean Development

One can thus see how the principles of the agile manifesto directly lead to the generation of better, faster asymmetric information and has a direct impact on business profitability. This informational view of software is a paradigm shift in the way we approach software quality. It is no more necessary for working software in production to be perfect from day one, what is more important is for the software to be usable and functional, and most importantly to deliver the best information of how it needs to change to meet future requirements !!

Concurrent Testing in Grails With GPars and Spock

Last month I did a major backend refactoring of on of our applications and was feeling happy about how much the code has been simplified and easier to understand. All was well until suddenly, we started getting strange errors in production. Our set of unit and integration tests had passed and I was quite confident that nothing had broken.

With a few hours of debugging and head scratching, I nailed the problem down to some mutable variables that were being shared across multiple threads and sessions. In order to avoid initiating and creating multiple objects of a particular service, we had converted it to a Singleton, and the resulting change caused some variables ending up shared between multiple threads and break down the complete application. We did not have any unit or integration tests developed to test for concurrency and hence could not catch the issues earlier

This experience led me to write test cases for concurrency and subsequent evaluation of technologies appropriate for developing concurrent applications. A few hours of reading opened my eyes to how much the world of concurrent programming for the JVM had changed in the last few years. My experience is core java were from Java 2.0 years and with constructs like Thread, Runnable and Synchronized methods.

Java 5.0 has apparently changed the space completely with introduction of the new util.concurrent package. Here are some of the things are now possible now when using the modern JDK

* Manage your own thread pool using Executor Services
* Use the concept of callable to asynchronously call and receive responses from thread
* Coordinate threads using Countdown latches (similar to Semaphores in C++)
* Better data structures for data exchange between threads - BlockingQueue
* Collection data structures - ConcurrentQueue, ConcurrentHashmap for better performance
* Better granularity in synchronizing blocks of code (rather than using object level 'synchronized' methods) using Locks

Further exploring the topic I came across new libraries that were available to implement concurrency using implementation patterns like Memory Transactions and Actors. The prominent ones for the JVM were Akka and GPars.

Akka is implemented in Scala and supports both of the patterns mentioned - it supports software transactional memory (STM) by using specialized transactional primitives that can be modified only within defined transactions. It also supports the concept of defining Actors - Objects that extended an Actor Object would always have their methods executed in separate threads. The Actor pattern of concurrency is also supported by the Groovy GPars library.

Since Grails obviously played best with Groovy, and also since the implementation of concurrent did not involve complex synchronizations, I chose to implement concurrent Spock tests using GPars. The combination of Spock’s highly expressive specification language and the clean design of GPars makes writing tests quite enjoyable.

For example Spock allows Data Driven testing and allows you to wite tests like

class MathSpec extends Specification {
    def "maximum of two numbers"() {
        // exercise math method for a few different inputs
        Math.max(1, 3) == 3
        Math.max(7, 4) == 7
        Math.max(0, 0) == 0

GPars allows you to implement concurrent execution with a pool of threads and groovy collections like so:

withPool {
    def selfPortraits = images
        .findAllParallel{it.contains me}
        .collectParallel {it.resize()}

Combining these two results is extremely intuitive parallel test cases. I have included one of our example test cases below:

package com.squareprism.xyz

import grails.test.mixin.TestMixin
import grails.test.mixin.support.GrailsUnitTestMixin
import spock.lang.Specification
import static groovyx.gpars.GParsPool.*

 * See the API for {@link grails.test.mixin.support.GrailsUnitTestMixin} for usage instructions
class ParallelXYZGeneratorSpec extends Specification {

    // data array for test data
    def dataArray = [
            [11.26, 75.78, 25, 11, 1973, 4, 55],
            [11.26, 75.78, 25, 11, 1973, 4, 55],
            [11.26, 75.78, 25, 11, 1973, 4, 55],
            [11.26, 75.78, 25, 11, 1973, 4, 55],
            [11.26, 75.78, 25, 11, 1973, 4, 55],
            [11.26, 75.78, 25, 11, 1973, 4, 55],
            [11.26, 75.78, 25, 11, 1973, 4, 55],
            [11.26, 75.78, 25, 11, 1973, 4, 55]
    def results // collect results here

    def setup() {

    def cleanup() {

    void "test Parallel XYZ Generation"() {
        withPool(8) { // use 8 parallel threads
            results = dataArray.collectParallel{
                return XYZGenerator.calculate(it[0], it[1], it[2], it[3], it[4], it[5], it[6])

            results.size() == 8


Be Careful With That FAQ

It could kill your SEO rankings

During the last few weeks we were experimenting with how we can improve our SEO rankings for Jaatakam.

We had some pretty basic stuff in our pages as content as far as SEO keywords were concerned. Nothing targeted, but just enough content to inform the user about the website. We had found some pretty good improvement with impressions and views after we optimized our Terms and Privacy policy pages earlier.

Having had some decent success, we went about adding more content to our page to have a better coverage of keywords (Yes. we wanted to be ranked better for those top 100 keywords!). Added some interesting questions and answers that were relevant to the site. Created a separate FAQ page. Added the page to the Sitemap. Let Google know about it. And waited.

The result. In two weeks our page ranking dropped and the number of impressions dropped by 60%.

On frantic analysis, found that the keyword density for those KWs that were helping us get so many impressions had dropped due to the new content added. And that resulted in far fewer hits than earlier.

Lesson learned. Be careful with adding new text content to your site as it could hurt SEO in a big way. One possible solution could be adding the new content to a separate blog page and convert users from that page to your site.

As of now, we have removed the added page and waiting for results. It may be a couple of weeks before we see any change, but we will update here when we see any improvements.

Optimizing TOC and Privacy Policy for Single Page Apps

One Line Advice = ‘Don’t Include in your Single Page’

Single Page Applications written in Javascript are very popular these days. All content loads in a single page and when the user clicks on links they see ‘blazing’ performance since pages don’t load from the server.

Almost all Apps today need to include pages around the Terms and Conditions (TOC) of use and Privacy Policy pages. Often times these pages have content that runs into many paragraphs and could be a significant portion of your site/apps content.

What we don’t realize when we add these monstrous pages of legal ramblings is how they impact our websites SEO in a negative fashion. When we have 100 words or relevant content in our app and 2000 words of legalese all included in a single page, the relevant content becomes 5% of overall site from a search engine perspective. Important keywords will have a much lower density and this has a significant effect on your page rankings.

The solution. Get rid of TOC and Privacy from your single page and add them as separate page URLs. Use your robot.txt to disallow the terms and privacy pages from getting indexed. Soon you will start seeing much more relevant content keywords within your webmaster console and better impressions overall.

See below a sample robots.txt on how you should configure your website. Reach out to your webmaster now.

Sample Robots Txt