The ETL from Hell – Diagnosing Batch System Performance Issues

by Nigel Rivett Too often, the batch systems that underlie a lot of database processing just grow without conscious design. When runs start to extend beyond their allotted time, and tuning no longer solves the problem, it is often discovered that batches are run in series, with draconian error handling. It is time to impose some rational design, and Nigel is a seasoned healer of batch processes. Overview Batch systems, which perform housekeeping jobs without human intervention, are often used with databases, usually for the population of data warehouses but more generally for any regular backend processing such as accounting processes. In this article, I’ll be discussing the typical problems in batch processing, showing how to determine their cause, and describing how to resolve them. We will concentrate on an overnight batch run because this is such a common way to populate a data warehouse, but the same principles will apply to any batch system, whenever it is run. Systems that are designed for high availability have additional challenges, and processing will already be designed so that maintenance can be carried out while the system is available. These systems can still benefit from the principles outlined in this article because control of the process can still be an issue. >> Go to Source...

Are athletes really getting faster, better, stronger?

Watch this TED Talk from David Epstein. When you look at sporting achievements over the last decades, it seems like humans have gotten faster, better and stronger in nearly every way. Yet as David Epstein points out in this delightfully counter-intuitive talk, we might want to lay off the self-congratulation. Many factors are at play in shattering athletic records, and the development of our natural talents is just one of them. >> Watch...

Peter Ustinov

What a guy ! Would have loved to cook for him my favourite dish … Avocado Mango Prawn Spicy Salad on slices of grilled pineapple with cinnamon sugar …

La Tour Eiffel

I know it’s a bit late but 125 years and still standing : La Tour Eiffel is one of my favourites! And that is not only because it helped me through my marathon in Paris in 2010 ,,, it was just so energising. Find out all the details from the wiki … as usual!

Sofadeve – Mailer Response Tracker Module

This module went live just in time for the Meeting of german academics in Edinburgh in April 2014 and has claimed to be great success. The module uses the ground breaking webFRAME from Sofadeve’s product suite and provides a very simple but nevertheless appealing response page which is referenced into the mailer inviting to the upcoming event. Interested ? Want to know more ? Have a look yourself ? In case of one or two or three yes answers to the above use our contact page to get in touch...

Design Patterns

Nice and neat overview including some criticism … isn’t that refreshing? In software engineering, a design pattern is a general repeatable solution to a commonly occurring problem in software design. A design pattern isn’t a finished design that can be transformed directly into code. It is a description or template for how to solve a problem that can be used in many different situations… >> Go to Source...

To create for the ages, let’s combine art and engineering

When Bran Ferren was just 9, his parents took him to see the Pantheon in Rome — and it changed everything. In that moment, he began to understand how the tools of science and engineering become more powerful when combined with art, with design and beauty. Ever since, he’s been searching for a convincing modern-day equivalent to Rome’s masterpiece. Stay tuned to the end of the talk for his unexpected suggestion. Watch...

NoSQL

Now that we just get used to SQL, lets go to NoSQL. Have a look at this prominent introduction … good for perspective . Introduction to NoSQL by Martin Fowler So, nothing lost...

Prime Numbers – Visual Patterns ?

Will these pictures give you some idea of what their secret is ? Maybe there is no secret ! The Ulam spiral is a way of visualizing the distribution of prime numbers (black dots in the image and blue in the movie below). This pattern is one the great unsolved mysteries in mathematics and has important consequences in Cryptography. You create the pattern on an “infinite” grid of squares where you start somewhere and label the square with 1 then move one to the right and label the square 2 then move one up and label it 3 and keep going in a spiral fashion. This is your base grid. Now color or highlight the prime numbers in that grid plane and you get the picture if you use black … Most interesting is the appearance of diagonal straights and other patterns in this … Watch...

Samuel Barber

Samuel Osmond Barber II (March 9, 1910 – January 23, 1981) was an American composer of orchestral, opera, choral, and piano music. He is one of the most celebrated composers of the 20th century: music critic Donal Henahan stated that “Probably no other American composer has ever enjoyed such early, such persistent and such long-lasting acclaim.”[1] His Adagio for Strings (1936) has earned a permanent place in the concert repertory of orchestras. He was twice awarded the Pulitzer Prize for music, for his opera Vanessa (1956–57) and his Concerto for Piano and Orchestra (1962). Also widely performed is his Knoxville: Summer of 1915 (1947), a work for soprano and orchestra, which sets a prose text by James Agee. Unusual among contemporary composers, nearly all of his compositions have been recorded. >> Go to Source...

The Taming of The Finest Bubbles

An initial burst of effervescence occurs when the Champagne contacts the dry glass on pouring. These bubbles form on imperfections in the glass that facilitate nucleation or, to a lesser extent, on cellulose fibres left over from the wiping/drying process as shown with a high-speed video camera. However, after the initial rush, these naturally occurring imperfections are typically too small to consistently act as nucleation points as the surface tension of the liquid smooths out these minute irregularities. The nucleation sites that act as a source for the ongoing effervescence are not natural imperfections in the glass, but actually occur where the glass has been etched by the manufacturer or the customer. This etching is typically done with acid, a laser, or a glass etching tool from a craft shop to provide nucleation sites for continuous bubble formation (note that not all glasses are etched in this way). In 1662 this method was developed in England, as records from the Royal Society show. Dom Pérignon was originally charged by his superiors at the Abbey of Hautvillers to get rid of the bubbles since the pressure in the bottles caused many of them to burst in the cellar. As sparkling wine production increased in the early 18th century, cellar workers had to wear a heavy iron mask to prevent injury from spontaneously bursting bottles. The disturbance caused by one bottle exploding could cause a chain reaction, with it being routine for cellars to lose 20–90% of their bottles this way. The mysterious circumstance surrounding the then unknown process of fermentation and carbonic gas caused some critics to call the sparkling...

Eiffel

Not the tower but the programming language. I am surprised how few people know about it and how even fewer dare to use it. But then we love C# and that’s the fashion and it is funky and sexy and does it all … ??? However, have a look and think … Eiffel is an ISO-standardized, object-oriented programming language designed by Bertrand Meyer (an object-orientation proponent and author of Object-Oriented Software Construction) and Eiffel Software. The design of the language is closely connected with the Eiffel programming method. Both are based on a set of principles, including design by contract, command-query separation, the uniform-access principle, the single-choice principle, the open-closed principle, and option-operand separation. Many concepts initially introduced by Eiffel later found their way into Java, C#, and other languages. New language design ideas, particularly through the Ecma/ISO standardization process, continue to be incorporated into the Eiffel language. >> Go to Source...

Architecture at home in its community

When TED Fellow Xavier Vilalta was commissioned to create a multistory shopping mall in Addis Ababa, he panicked. Other centers represented everything he hated about contemporary architecture: wasteful, glass towers requiring tons of energy whose design had absolutely nothing to do with Africa. In this charming talk, Vilalta shows how he champions an alternative approach: to harness nature, reference design tradition and create beautiful, modern, iconic buildings fit for a community. Barcelona-based architect Xavier Vilalta works in Europe, Africa and the Middle East. He adopts and updates traditional design principles to construct modern buildings that truly suit their environment. >> Play...

Steven Paul “Steve” Jobs

Steven Paul “Steve” Jobs (February 24, 1955 – October 5, 2011) was an American entrepreneur,[7] marketer,[8] and inventor,[9] who was the co-founder (along with Steve Wozniak and Ronald Wayne), chairman, and CEO of Apple Inc. Through Apple, he is widely recognized as a charismatic pioneer of the personal computer revolution[10][11] and for his influential career in the computer and consumer electronics fields, transforming “one industry after another, from computers and smartphones to music and movies”.[12] Jobs also co-founded and served as chief executive of Pixar Animation Studios; he became a member of the board of directors of The Walt Disney Company in 2006, when Disney acquired Pixar. Jobs was among the first to see the commercial potential of Xerox PARC’s mouse-driven graphical user interface, which led to the creation of the Apple Lisa and, one year later, the Macintosh. He also played a role in introducing the LaserWriter, one of the first widely available laser printers, to the market.[13] After a power struggle with the board of directors in 1985, Jobs left Apple and founded NeXT, a computer platform development company specializing in the higher-education and business markets. In 1986, he acquired the computer graphics division of Lucasfilm, which was spun off as Pixar.[14] He was credited in Toy Story (1995) as an executive producer. He served as CEO and majority shareholder until Disney’s purchase of Pixar in 2006.[15] In 1996, after Apple had failed to deliver its operating system, Copland, Gil Amelio turned to NeXT Computer, and the NeXTSTEP platform became the foundation for the Mac OS X.[16] Jobs returned to Apple as an advisor, and took control...

How An Arcane Coding Method From 1970s Banking Software Could Save The Sanity Of Web Developers Everywhere

Forty years ago, a Canadian bank pioneered a brand new computer system that allowed non-programmers to help write code. The paradigm was so disruptive that it was ignored by computer scientists for decades. But as web apps get increasingly complex, and web devs become increasingly stressed out, “flow-based programming” may be raging back to life. >> Go to...

Douglas Hofstadter

Douglas Richard Hofstadter (February 15, 1945 – …) is an American professor of cognitive science whose research focuses on the sense of “I”,[2][3] consciousness, analogy-making, artistic creation, literary translation, and discovery in mathematics and physics. He is best known for his book Gödel, Escher, Bach: an Eternal Golden Braid, first published in 1979. It won both the Pulitzer Prize for general non-fiction[4][5] and a National Book Award (at that time called The American Book Award) for Science.[6][a] His 2007 book I Am a Strange Loop won the Los Angeles Times Book Prize for Science and Technology.[7][8][9] … >> Go to Source...

Paul Jackson Pollock

Paul Jackson Pollock (January 28, 1912 – August 11, 1956), known as Jackson Pollock, was an influential American painter and a major figure in the abstract expressionist movement. He was well known for his unique style of drip painting. During his lifetime, Pollock enjoyed considerable fame and notoriety, a major artist of his generation. Regarded as reclusive, he had a volatile personality, and struggled with alcoholism for most of his life. In 1945, he married the artist Lee Krasner, who became an important influence on his career and on his legacy.[1] Pollock died at the age of 44 in an alcohol-related, single-car accident; he was driving. In December 1956, several months after his death, Pollock was given a memorial retrospective exhibition at the Museum of Modern Art (MoMA) in New York City. A larger, more comprehensive exhibition of his work was held there in 1967. In 1998 and 1999, his work was honored with large-scale retrospective exhibitions at MoMA and at The Tate in London.[2][3] In 2000, Pollock was the subject of the film Pollock, directed by and starring Ed Harris, which won an Academy Award. >> Go to Source...

Mozart

Hello hello hello … it’s Mozart day ! Don Giovanni, String Quintetts, Requiem, Rondo alla turca, Piano Sonata in F Major, Clarinet Concerto in A Major, … who cares … there is no favorite … he is just a genius … and that’s because 99% of his work is simply sublime . Below is the top of the wiki page … Wolfgang Amadeus Mozart (German: [ˈvɔlfɡaŋ amaˈdeus ˈmoːtsaʁt], English see fn.;[1] 27 January 1756 – 5 December 1791), baptised as Johannes Chrysostomus Wolfgangus Theophilus Mozart,[2] was a prolific and influential composer of the Classical era. Mozart showed prodigious ability from his earliest childhood. Already competent on keyboard and violin, he composed from the age of five and performed before European royalty. At 17, he was engaged as a court musician in Salzburg, but grew restless and travelled in search of a better position, always composing abundantly. While visiting Vienna in 1781, he was dismissed from his Salzburg position. He chose to stay in the capital, where he achieved fame but little financial security. During his final years in Vienna, he composed many of his best-known symphonies, concertos, and operas, and portions of the Requiem, which was largely unfinished at the time of his death. The circumstances of his early death have been much mythologized. He was survived by his wife Constanze and two sons. He composed over 600 works, many acknowledged as pinnacles of symphonic, concertante, chamber, operatic, andchoral music. He is among the most enduringly popular of classical composers, and his influence on subsequent Western art music is profound; Beethoven composed his own early works in the shadow of Mozart, and Joseph Haydnwrote that “posterity will not see such a talent again in 100 years.”[3] >> Go To Source...

Zooming the Zone

We all know that the best way to get something done is to write down a list of what to do. Well, here for you need of course some idea of what it is you are doing and that usually implies a little of analysing or broadly spoken “cutting into pieces”. In doing so it is sometimes necessary to keep cutting the things again and again until they are small enough to be digested. Parallels to actual eating food? Yes, naturally we wouldn’t eat the whole salmon in one go – even if it smells delicious and we can’t wait to have it down our throats. We would separate it into smaller pieces and then start pleasing our taste buds … but what happens when our cutting results in pieces which have the same behavior or structure, lets say pattern as one of the previously found? Deadlock? Frustration? Maybe initially some excitement of recognizing something and some pleasure of presumptuous “Oh, I know that!”. but that will die away quickly when we realize that we actually didn’t understand anything. Point of depression is reached quickly. However, no need to be disappointed. Just change the zoom from in to out and look at it from the whole. Embrace the thing. Let it simmer in your brain and find its own way. Remember the fractals and Mandelbrot set pictures? How certain pattern popped up again and again and the recursiveness of the whole thing became more and more obvious to that extent that we actually do understand now what this is about. At least some of us. Almost not necessary as...

What’s Up