DJ’s Weblog

Well over 10 years ago I bought a box set of Magic The Gathering (MTG) cards. I wasn’t really sure what they were, but they looked fascinating. Unfortunately, they gathered dust after a while, mostly because there wasn’t anyone else to play against, and I didn’t understand the rules properly anyway. Fast forward to 2009. My son Joseph, having discovered and dusted off the box set, taught me how to play, at at Starbucks cafe over a few hot chocolates and cappuccinos on a cold winter afternoon.

Hooked!

Since then we’ve been fans of the game of endless possibilities and ever changing scope and interest, and almost regulars at our local MTG store, Fan Boy Three on Newton St in Manchester.

So to educate myself in all thing Magic, I turned to MTG’s official website, Wizards of the Coast’s The Multiverse, and in particular, to their incredibly prolific set of column authors on the Daily MTG. More articles on design, deck construction, strategy and match reports than you could shake a Planeswalker card at.

But while I read a lot, the majority of it is on paper, in the bath, on the train, and soaking up the countless minutes lost at the start of every meeting, while you wait for people to get started, fail to get the projector working, fetch coffees or fiddle with the air conditioning. And on paper, the MTG articles are good, but for a novice like me, there’s something missing. The articles make lots of references to cards by name, and when reading online, there’s a nice popup of the card details so you can see what the author is talking about. But on paper?

So I had an itch to scratch. What I wanted was an accompanying printout of the cards mentioned in any given Daily MTG article. So when the author referred to Hedron Crab,  Baloth Woodcrasher or Oran-Rief, the Vastwood I would know what they were talking about.

I decided to use Google App Engine, and have my Python HTTP responder in the cloud. I created a very simple app “mtgcardinfo”, part of my Github-hosted scratchpad area gae-qmacro. Given the URL of an MTG article, the app uses urlfetch() to go and get it, parses out the card names, and produces an HTML response with a whole load of image references. Luckily the card detail popups in the articles are powered by Javascript and are great indicators of card names for anyone who cares to wield a regex to look for them.

And of course to glue it all together, I used a bookmarklet, so I could jump to the list of cards while directly in the article.

So if you’re interested, have a go: http://qmacro.appspot.com/mtgcardinfo.

The combination of App Engine, Python, HTTP and Javascript is rapidly becoming my new Swiss Army Knife of choice in the web-based online world. And the best thing? I’m teaching Joseph this stuff, and not only is he incredibly good at it, he *loves* it!

Cloud computing, virtual machines. It’s big business. Amazon has its Elastic Compute Cloud (EC2) which provides “resizable compute capacity in the cloud“, Microsoft has Azure, providing “on-demand compute and storage to host, scale, and manage Web applications on the Internet” and Google’s offering is App Engine which offers “the ability to build and host web applications on Google’s infrastructure“. As you might know, I’m personally very taken with App Engine.

The offerings are slightly different - for example, while EC2 is bare virtual hardware, App Engine is a web application platform in the cloud. But they all have similar pricing arrangements, based generally on uptime or CPU time, I/O  and storage.

Does this seem familiar to you? It does to me, but then again, I did just turn 0×2B this month. In 1988 I was working in the Database Support Group at a major energy company in London, looking after the SAP R/2 databases, which were powered by IMS DB/DC, on MVS - yes, IBM big iron mainframes. I still look back on those days with fond memories.

In reviewing some 3rd party software, I wrote a document entitled “BMC Software’s Image Copy Plus: An Evaluation“. BMC’s Image Copy Plus was a product which offered faster image copies of our IMS DB (VSAM) databases. (Image Copy Plus, as well as IMS, is still around, over 20 years on! But that has to be the subject of another post).

One of the sections of the evaluation was to compare costs, as well as time — by how much would the backup costs be reduced using BMC’s offering?

And have a guess on what the cost comparison was based? Yes. CPU time, I/O (disk and tape EXCPs) and actual tapes.

Everything old is new again.

It’s been pretty much six years to the day since I last wrote here about Dashboard, Nat Friedman’s project and implementation of a realtime contextual information system. So I thought it fitting to make a short demo showing integration between Google Wave and SAP, inspired by the cluepacket-driven style shown so nicely with Dashboard.

I got my Wave Sandbox account a week or so ago, and have had a bit of time to have a look at how robots and gadgets work — the two main Wave extension mechanisms. To get my feet wet, I built a robot, which is hosted in the cloud using Google App Engine, another area of interest to me, and the subject of this weblog entry. I used Python, but there’s also a Java client library available too. You can get more info in the API Overview.

What this robot does is listen to conversations in a Wave, automatically recognising SAP entities and augmenting the conversation by inserting extra contextual information directly into the flow. In this example, the robot can recognise transport requests, and will insert the request’s description into the conversation, lending a bit more information to what’s being discussed.

The robot recognises transport requests by looking for a pattern:

trkorr_match = re.search(' (SAPK\w{6}|[A-Z0-9]{3}K\d{6}) ', text)

In other words, it’s looking for something starting SAPK followed by six further characters, or something starting with 3 characters, followed by a K and six digits (the more traditional customer-orientated request format). In either case, there must be a space before and a space following, to be more sure of it being a ‘word’.

How does it retrieve the description for a recognised transport request? Via a simple REST-orientated interface, of course I use the excellent Internet Communication Framework (ICF) to build and host HTTP handlers so I can expose SAP functionality and data as resources in a uniform and controlled way. Each piece of data worth talking about is a first class citizen on the web; that is, each piece of data is a resource, and has a URL.

So the robot simply fetches the default representation of the recognised request’s ‘description’ resource. If the request was NSPK900115, the description resource’s URL would be something like:

http://hostname:port/transport/request/NSPK900115/description

Once fetched, the description is inserted into the conversation flow.

http://www.youtube.com/watch?v=G7W2M6H3OQo

(Originally written on SDN but republished here because of portal access issues)

A short while ago I was sent a review copy of “SAP Business ONE Implementation”, by Wolfgang Niefert, published by Packt Publishing. On receipt, I skimmed through it, and my first impressions were very favourable. I’m now reading through it a second time as I sit with a cup of tea and a slice of cake in North Wales on a Bank Holiday weekend, and I’m happy to say that my opinion hasn’t changed.

To give you a bit of background, I’m an SAP veteran of 22 years – starting out with R/2 version 4.1d in 1987, moving through R/3 in the mid-90’s and on to Enterprise and beyond. But this is the first time I’ve studied SAP Business ONE in any detail. So while I have a lot of experience of SAP’s traditional products, I’m approaching SAP Business ONE, and “SAP Business ONE Implementation” more as the potential owner of a small business.

I certainly haven’t been disappointed. “SAP Business ONE Implementation” is written “for technically savvy business owners, entrepreneurs and departmental managers”. And I think by and large the book does a great job of reaching out to and connecting with exactly that audience. I was expecting the book to be a fairly technically orientated implementation how-to. But it is more than that. It takes you from business first principles, connecting well at the level of sales, delivery, inventory, warehousing, manufacturing and other business challenges. It explains how SAP Business ONE is designed to address those challenges, and guides you through installation, implementation and some configuration of the system. Once the basics have been established, it moves further to cover project planning, reporting and analysis, business process analysis, customer relationship management, logistics & supply chain management, contract management, and ends up addressing, albeit briefly, more complex reporting tools and topics, data migration, and electronic commerce.

The book has fewer than 300 pages. A book that addresses the areas that this book does could easily be twice that size. But that’s where this book does well. It’s an approachable, undaunting and really rather good introduction to running your business with SAP Business ONE. The writing style is very easygoing, and informative without being patronising. There are plenty of examples, and all the screenshots you’d need. It doesn’t try to be a reference book. It does try to be a sort of hybrid guide to solving the business and technical challenges of running a small or medium sized company using SAP software, and I would say that it succeeds.

If you’re a small business owner considering stepping up and taking control of your business with SAP Business ONE, if you’ve already got SAP Business ONE and want to explore more application features at a high level, or if even if (like me) you’re an SAP hacker wanting to learn about what SAP Business ONE can do, then you could do a lot worse than grab a copy of this book.

Recently it has come to people’s attention that there is a website www.sap-abap4.com out there with a lot of very interesting content … which seems to have been completely “lifted” from the SAP Developer Network (SDN) and reproduced verbatim, except that in each case the original author name has been removed!

Lots of discussion is taking place how best to deal with this. One way (and I’m posting it as a blog entry as much for my memory’s sake as anything else) is to conditionally rewrite requests for images. I’m using Apache and therefore the mod_rewrite extension is my tool of choice.

It just so happens that there are a couple of screenshots in a recent SDN blog entry of mine “A return to the SDN community, and a touch of Javascript” and these images are hosted on my own server at www.pipetree.com.

So as a little test, I can control the requests for these images, rewriting those requests so that a different image is served depending on the request’s referrer — the URL of the page that the images are referenced on with an <img /> tag.

So with some mod_rewrite voodoo in a local .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_REFERER} ^http://www\.sap-abap4\.com
RewriteBase /qmacro/x
RewriteRule ^SdnPageTitle(Fixed|Broken)_small\.jpg$ StolenContent.png [L]

I can send a ‘StolenContent.png’ image, if the referrer is from the rogue site.

The result of the rewrite is that when viewed on SDN, the blog entry looks fine, and the screenshot images look as they’re supposed to:

Images appear as they're supposed to

But when the images are used on www.sap-abap4.com, they will appears differently:

Image appears differently

So there you have it. It’s not a complete solution to the problem by any means, but it at least will alert unsuspecting readers of that website to what’s happening (if you’re testing yourself, you might have to refresh the pages in your browser, as it will probably have cached the first version of each image). Perhaps the SAP community network team can apply this technique for the images hosted on SDN.

This is a bit of a hand-wavy post, but I wanted to get my thoughts down. Recently there’s been a spate of interest around interaction with devices, applications and systems … via a chat-style interface. This is nothing new, of course. Bots have existed on the IRC networks for a long time. The venerable Purl, an infobot, was a particular favourite of mine. When instant messaging (IM) came along, we had a new chat interface - which took the form of one-on-one or conference (’group’) chat. With Jabber (XMPP), ‘Chatbot’ was a favourite in the various conference rooms. Back in late 2002, I even wrote about Chatbot in a 2-part series “The Command Line of the Future” (”Is Jabber’s Chatbot the Command Line of the Future?” and The “Command Line of the Future Part 2: Expanding Chatbot’s Repertoire“):

Consider for a moment what this command line of the future might look like. More and more people are online. More and more people are permanently connected, whether it be through DSL, cable, or 802.11 technology. And more and more of these people are communicating. Talking. Having conversations. In addition to email and Internet Relay Chat, or IRC, the (relatively) new kid on the block, Instant messaging (IM), is playing a huge part in facilitating these conversations. And in the same way that it’s common for us to have a command prompt or three sitting on our graphical desktop, it’s also becoming common to have chat windows more or less permanently open on the desktop too.

But when thinking of IM, why stop at conversations with people? The person-to-application (P2A) world isn’t the exclusive domain of the Web. Bots, applications or utilities that have their interface as a projection of a persona into the online chat world, are a great and fun way to bring people and applications together in a conversational way.

Interacting with a bot is the same as interacting with a person: type something to it and it replies. And what’s more, because of the similarities between a classic command-line prompt and that of a chat window, where you’re talking with a bot — both scenarios are text-based — interaction with a bot is scriptable.

Forward to the present.

Just the other day, @davemee and @technicalfault alerted me to @manairport, Manchester Airport’s online persona on Twitter, obviously yet another chat-style interface. You can interact with it via direct messages (DMs). You follow it, it will follow you back, and you’re away.

me: d manairport be7217

manairport: Received request for information: be7217
manairport: Status of 17:40 flight BE7217 to Dusseldorf
            departing T3: Scheduled 17:40

Nice and useful!

And then just this morning, I read a weblog post on SDN entitled “SAP Enterprise Service and Google Wave“. In it, the author talks about connecting Google Wave (you guessed it, yet another chat-style interface, amongst other things) with SAP, in particular enterprise ’services’. In the short demo, order information from an SAP system is retrieved in a conversational way. The concept is great. The obvious issue with what’s shown in the demo (and I know it’s only a proof of concept) is that the bot responds with a data structure dump of information. What we’re looking for is something more, well, consumable by humans. Smaller, more distinct and addressable pieces of information that can be returned and be useful.

But what was more telling, at least to me, were the difficulties he described in connecting to the complex Enterprise Service backend in SAP:

“… find the webservice … create a proxy … I did have some problems with calling the ES … On Appengine there are some limitations on what you can call of Java classes … From an architectural point I’m not real proved of the solution…”

Hmm. Why does architecture have to be complex? Using Enterprise Services, using SOA, is more complex than it needs to be. There’s a reason why the web works. There’s a reason why Google designed App Engine’s backend infrastructure (including asynchronous task queues) in a simple HTTP-orientated way. There’s a reason why the Wave robot protocol is based on simple HTTP mechanisms. There’s a reason why mechanisms like PubSubHubBub and Webhooks are based on HTTP as an application protocol. Because simple works, and it works well.

Let’s come back to the “smaller, more distinct and addressable” issue. If we let ourselves be guided by a Resource Orientated Architecture (ROA) approach, rather than a Service Orientated Architecture (SOA) approach, we end up with simpler application protocols, flexible, reliable and transparent integration, and pieces of information that are addressable — and usable — first class citizens on the web. This is Twitter’s killer feature.

Enterprises suffer enough with complexity paralysis. We should endeavour to embrace the design of HTTP as an application protocol (which is what I’m doing with Coffeeshop), rather than fight against it.

After yesterday’s screencast showing the use of coffeeshop from the command line, here’s one that expands upon the direction I’m taking the implementation, following the REST/HTTP philosophy. It shows, I hope, that embracing REST-orientated HTTP features, such as content negotiation (”conneg”), and the concepts of resources and representations, gives you a fantastically flexible and straightforward application protocol to work with and be guided by. (I’m not doing full-blown conneg, that will come later. But what I am doing works well for me).

http://www.youtube.com/watch?v=NhAWH2-Quuk

In this shorter screencast, I continue on from where I left off — viewing the message detail resource in the web browser. I use conneg to request that same resource in JSON instead of HTML, and show how the JSON representation can be easily parsed, and the data reused, further along the pipeline.

To follow on from the first coffeeshop demo screencast, I thought I’d make another. This time it’s to highlight the fact that coffeeshop is fundamentally a REST-orientated, HTTP-based pubsub application at the core, and not just a web-based application. Hopefully this comes across through the use of command-line HTTP tools to manipulate Channel, Subscriber and Message resources.

This time, the coffeeshop instance I’m using is one running on Google’s App Engine cloud infrastructure — on appspot.com.

http://www.youtube.com/watch?v=TI48cdpWOBg

In the screencast, I also make use of Jeff Lindsay’s great Postbin tool for creating the recipient resources for the Subscribers. It was originally created to help debug Webhooks, but of course, a Subscriber is a sort of Webhook as well. (Postbin runs on App Engine too!).

Anil Dash provides food for thought in his post “What Works: The Web Way vs The Wave Way“. While I agree with him on the importance of the incremental approach to technology progression on the web (”The Web Way”), I do profess to have an intense interest in the pollination of XMPP into the HTTP space.

However, I must call him on this small statement:

“XMPP is way too complicated for any normal human to deploy”

Compared to what? I’m getting the idea that he’s referring to ’simpler’ mechanisms such as HTTP or SMTP servers. Simpler? Has Anil modified a sendmail config file recently?

These days setting up an XMPP server is pretty straightforward. Then again, I am perhaps somewhat biased

There’s an interesting article “HTTP PubSub: Webhooks and PubSubHubBub” that covers working with webhooks and points to a great HTTP / webhook developer utility “PostBin“, which:

“lets you debug web lets you debug web hooks by capturing and logging the asynchronous requests made when events happen. Make a PostBin and register the URL with a web hook provider. All POST requests to the URL are logged for you to see when you browse to that URL.”

The article also shows a very simple pubsub “Hello, World” script, postbin.rb, that nicely demonstrates the basic features of Watercoolr — another HTTP-based pubsub mechanism.

So I thought I’d write the equivalent to postbin.rb, this time demonstrating the same features in Coffeeshop. This way, we can see how things compare. It’s in Python, but that’s neither here nor there.

import httplib, urllib, sys

hubconn = httplib.HTTPConnection('localhost:8888')

hubconn.request("POST", "/channel/")
channel = hubconn.getresponse().getheader('Location')
print "Created channel %s" % channel

hubconn.request("POST", channel + "subscriber/",
  urllib.urlencode({'resource': sys.argv[1]}))
subscriber = hubconn.getresponse().getheader('Location')
print "Added subscriber %s" % subscriber

while True:
  print "Post message:"
  msg = sys.stdin.readline()
  hubconn.request("POST", channel, msg)
  message = hubconn.getresponse().getheader('Location')
  print "Message published: %s" % message

I’ve added some print statements to show what’s going on, and to highlight the HTTP resources created and utilised.

Here’s a sample execution:

> python postbin.py http://www.postbin.org/1a5m8w0
Created channel /channel/1/
Added subscriber /channel/1/subscriber/2/
Post message:
Hello, Webhooks World!
Message published: /channel/1/message/ahFxbWFjcm8tY[...]RgDDA
Post message:

This message appears in the PostBin bucket as expected. Nice!

As well as showing how useful PostBin is, I hope this demonstrates how the basic features of Coffeeshop work, and perhaps more importantly, shows you that the REST-orientated approach is straightforward and works well.