I blogged about uncertainty facing Java (and Sun’s software stack) now that is being owned by Oracle. It appears that my fears are about to come true….:(
Archive for the ‘Systems Architecture’ Category
QED: a glimpse of the future for Java
Wednesday, August 25th, 2010Continuous integration with SQLCMD and Hudson
Thursday, June 3rd, 2010If you are not doing continuous integration, you should; and if you are – then you ought to consider database install as integral a part of your build process.
Most CI servers out there would allow you to execute batch or shell commands, and virtually every RDBMS provides a command line utility (and creating one on your own – if needed – is rather trivial).
Installing a database as part of your build process, and populating it with data could play role in your unit testing strategy, and should definitely be considered integral part of functional and regression testing procedures.
The following gives but an example of how to make MS SQL Server database install a part of your build process utilizing Microsoft command line utility SQLCMD and open source continuous integration server Hudson. This could be applied to any other RDBMS package – MySQL, PostgreSQL, Oracle, DB2 or Sybase – with minor adjustments.
The command line utility can be downloaded separately, or installed as part of SQL Server 200X installation. If your unit tests require database support, it might be a good idea to install free SQL Server Express Edition which could be started as part of the build process and shut down afterwards.
“The sqlcmd utilitylets you enter Transact-SQL statements, system procedures, and script files at the command prompt, in Query Editor in SQLCMD mode, in a Windows script file or in an operating system (Cmd.exe) job step of a SQL Server Agent job. This utility uses OLE DB to execute Transact-SQL batches.”
This provides an opportunity to make creation of a database and all dependent database objects a part in your continuous integration build process with Hudson – an open source continuous integration serverthrough executing scripts – either integrated with your build management utility such as Maven, Ant or MSBuild – depending on your platform, or just plain batch or shell commands.
A very basic Windows batch command in Hudson installing database through SQLCMD might look like this:
sqlcmd –S<IP address>,[port] -U<user> -P<password> -dmaster -i%WORKSPACE% \exec.sql
- -S indicates IP of the SQL Server instance to connect to
- - U and –P - user ID and password, respectively (this example uses SQL server Authentication)
- -d specifies the default database to connect to, and [master] database is the one you would want if creating a database is part of your build process.
NB: for complete commands list see documentation. Keep in mind that UserID/Password are in clear text, and will be sent over the network as such (unless you are using DAC). To minimize amount of hard-coded use include files in your script.
Here is an example as SQL code could be organized, in order of execution (I will link script files soon):
| 1 | exec.sql | main controller of the database installation process |
| 2 | constants.config | contains declaration of all variables to be used in the script; note that file extension is irrelevant for execution |
| 3 | backupDB.sql | backup existing database (if present); note that backup directory must exist on remote computer |
| 4 | createDB.sql | create new database; note that all the paths must exist on the remote computer |
| 5 | createTables.sql | creates all tables in the database; it might include creation of indices and constraints as part of the script but I would advise against it because of the potential dependencies conflicts |
| 6 | createFunctions.sql | creates all the user-defined functions for the database; the order in which objects are created in the database is important, placing functions before [views] and [stored procedures] reflects common dependency pattern as both could use the functions. |
| 7 | createViews.sql | creates all views |
| 8 | createProcedures.sql | creates all procedures |
| 9 | createConstraints.sql | adds constraints to the objects: primary keys, foreign keys, indices etc. |
| 10 | importData.sql | if your database has static data this could be used to add it at creation time; you may want to switch 9 and 10 as your data might potentially violate constraints (e.g. orphaned records); this also could be used in unit testing strategies |
| 11 | createUsers.sql | add all users; this script assumes that logins are already created (if not, add script to create logins first) |
| 12 | grantPrivileges.sql | grant privileges to the objects (e.g. EXECUTE) |
Gotchas:
It is important to understand that GO command completes the batch execution and flushes the buffer; it makes SQLCMD “forget” everything you might have declared prior to executing the command. In the above example, all variables declared in [constants.config ] are no longer part of the script once the GO command was issued.
When creating scripts, keep in mind differences between local (Hudson) directories and remote (SQL Server) ones. The former refer to location of the SQL script files checked out by Hudson from your source control, understood by SQLCMD and Hudson only; the latter specifies directories that SQL Server understands – backup and database locations.
SQLCMD takes in arguments in clear text which constitutes potential security breach; use it in fully trusted environment. Alternative would be implement workaround such as local batch files in secure directories with hard-coded userID/Passwords, and rely on Hudson security matrix; only users with access to the server would be able to see it. This does increase maintenance butb is relatively easy to implement.
If you want SQLCMD generated messages to be displayed in Hudson console output do not specify output file. Alternatively, I could envision a plugin that would parse the output file, and present it nicely in Hudson environment; I might take a stab at it, time permitting.
The successful execution of the scripts relies on correct order of creation – you must figure out object dependencies, and factor it in your scripts. Unfortunately, this is classical Catch 22 – the reliable way to determine dependencies is to query SQL Server after the objects has been created… Which means that you ‘d have to run all the script manually first, and adjust your scripts accordingly.
Here are some clever scripts that allow for discovery of the correct sequence for stored procedures and functions.
A bite of Apple
Saturday, May 1st, 2010It is official – Steve Jobs does not like Flash. The reasons explaned in a long essay cite everything under the sun – from open standards to battery life to lack of H.264 standard support… Job’s assertion that “Flash is the number one reason Macs crash” is all but certain to anger many on both sides of the dispute..
While I am not discounting Apple’s anal urge to control the straw that saved it from near death experience in the beginning of 2000, I think Steve Jobs got a point: Flash is a crutch obliterated by nascent open standards (same probably goes for Microsoft’s Silverlight) and will be obsolete withing few short years. The pendulum has swung the other direction – the hardware processing.
[update] Microsoft decided to side with Apple on this, probably given up its hopes for Silverlight to upend Flash. This eerily reminds of the tactics applied to Java rivalry when C# was released as an ECMA standard
Forest behind trees: Story of Enterprise Architecture
Thursday, April 29th, 2010I was listening to Story of Human Language audio course the other day. Dr. John McWhorter was explaining how European languages came up with the idea of gender for inanimate objects. The example he was using was silverware in German, with spoon being “he” (der Löffel), fork being “she” (die Gabel), and knife being of neuter gender (das Messer). The current theory maintains that this is the result of gradual changes, small steps taken one at the time that lead to the situation as we see it now. And each of the steps made perfect sense to the people at the time. Yet, the notion of gender in a language, left alone attribution of a specific gender to an object, appears manifestly arbitrary to non-native speakers.
It had occurred to me that this could be a perfect metaphor for ad-hoc Enterprise Architecture without roadmaps: a series of decisions that were a good idea at the time leading to a sorry state of chaos because there was no life-line stretching from “as-is” into the future state
The second distinction made by the professor was that of a language complexity inversely reflecting advancement of a society. Despite popular notion that the more evolved society would have more complex languages, in fact the opposite is true. A language used in a fast paced society loses many accoutrements considered necessary in less advanced societies (e.g. compare the etiquette of a French Royal court of Louis XIV with that of modern France) .
Applied to Enterprise Architecture this would imply that in the organizations with evolved EA programs the IT systems landscape will be less – not more – complex, and more efficient at the same time.
Design vs. Architecture
Friday, April 16th, 2010Some time ago I came across a discussion posted in a LinkedIN Enterprise Architecture group; it was about how (if) design is different from architecture. The question kept bugging me for a while – I felt that there is difference but could not quite formulate my thought. Until today, that is 
The break trough came while reading Software Systems Architecture by Nick Rozanski and Eoin Woods
Design refers to functional composition of the system, while architecture is also concerned with behavioral aspects of it. Therefore, architecture includes the design component, and extends it by adding quality attributes.
Time to Move on: James Gosling leaves Oracle
Wednesday, April 14th, 2010As of April 2, 2010 the ”Father of Java” is no longer with Oracle. This follows departures of Monty Widenius (2009) and Ken Jacobs (2010) Oracle might have acquired the body but the soul is gone…
A cute slide presentation from eWeek: The Life and Times of Java and James Gosling
CSM Feature to Price Comparison
Monday, April 12th, 2010If there is a single criterion distinguishing professional software developer from a hobbyist this would be Version Control system use. And one does not have to break a bank to do it, there are systems to fit every budget…
The following chart put together by Reliable Software plots VC/CSM systems’ features against licensing costs in a very persuasive way (take a look at an exhaustive breakdown of features by product)
Another in-depth resource:
http://better-scm.berlios.de/comparison/comparison.html
Also you may want to take a look at the Wikipedia article for broader spectrum of available products.
Thinking software into existence
Saturday, April 10th, 2010I remember a quote that I think belongs to Michelangelo, though I was unable to find it on the Internet. Once asked about his approach to creating beautiful statues he replied: “I take a piece of marble, and cut off all unnecessary parts”. This resonates with the quote that I was able to find – “I saw the angel in the marble and carved until I set him free”.
Now, what does it have to do with software creation? It seems to me, quite a lot. A software system is a manifestation of a thought process. Just as a statue that could be carved from marble, granite or cast in bronze, there are many different technologies that software could be implemented in, each with their own advantages and drawbacks.
Knowing these allows for a series of trade-off decisions that lead towards a particular implementation – functionality and features as envisioned by an architect, constructed by an engineer, and implemented by a developer; at every one of these stages creative thinking is at the core of the process. An idea, a vision leads the process of discarding unnecessary parts up until a coherent system emerges from alphabet soup of JMS, Ajax, SSO, MOSS, AD, DOM, SAX, SSIS and other incomprehensible acronyms…
A software system is thought into existence, everything else is implementation details.
Do system architects dream of business analysts?
Tuesday, March 16th, 2010The following represents my answers to the questions posed by a Ph.D. aspirant in search of the thesis
1. Q: Do you agree that architecture design and requirements elicitation are usually in parallel or have a big time overlap? In other words, Architectural design usually starts before requirements elicitation ends, and usually a big time overlap exists between these two activities.
A: No, requirements’ gathering starts first. Even a simple sentence stating “I’d like to have a system that..” constitutes a requirement in my opinion. There are aspects such as existing Enterprise Architecture that require Solution Architecture to align with; it might exist prior to requirements gathering for a specific software-intensive solution.
2. Q: Do you agree that when performing architectural design, making some decisions depends on the outcome of some other decisions? I other words, a sequence/order exists for the set of architectural decisions need to be made.
A: Yes. Some of the decisions predicated upon the others, therefore there is a hierarchical structure, a decision tree.
3. Q: Do you agree that, in most cases, for a particular architectural decision, only a [s]mall portion of the whole requirements actually influences the decision making? In other words, there is mapping exist from the architectural decisions you need to make and the requirements required when you make those decisions.
A: No. There are architecturally significant requirements, requirements that affect the decision only marginally, and everything in-between. Moreover, they affect the solution architecture on different levels: system architecture, software architecture, information architecture, infrastructure architecture etc.
4. Q: Do you agree that requirements engineers usually do not know clearly what items/aspects of requirements are architectural significant (i.e., it is not easy to distinguish architectural significant requirements from normal non architectural significant requirements)? Thus, they may ignore/miss some items of requirements (some of them may just look like trivial details) that are actually architectural significant during their requirements elicitation.
A: Yes, to certain extent. Not being trained to architect a system, business analysts (the “requirements engineers” in your question) cannot decide which requirements are architecturally significant, and which are not. It is up to solution architect to reconstruct the missing links from the requirements gathered like a paleontologist might reconstruct a dinosaur’s skeleton by just a few broken bones
5. Q: Do you agree that requirements engineers usually are not aware of which items/aspects of requirements are required urgently by architects and which items/aspects of requirements can be scheduled a bit later to elicit? Thus, they may first elicit the requirements required for making the decisions in the tail of the decision making sequence, and schedule the elicitation of requirements required for the architectural decisions in the front of the decision sequence to the end of the requirement elicitation phase.
A: Ditto. See answer to Q.4
6. Q: Do you agree the following statement: If the requirements engineers are informed of what items/aspects of requirements are required and when they are required by the architects for making architectural decisions, the requirements engineers will be able to properly schedule their requirements elicitation activities and elicit the required requirements in enough detail and precision. So they will have higher chance to provide the required requirements to the architects before the architects make those architectural decisions.
A: Yes. Business Analysts (aka “requirements engineers”) need to work with all stakeholders, including Solution Architects, to solicit and refine requirements for a software intensive system