Testing complex Webapps with wget

How do you test web applications? You can use Selenium, but Selenium is very heavy.

I tried wget.

To find out what exactly to send to the app, I started Firebug inside Firefox, went to the “Network” section and recorded what I would do, to see, whether the application works. Then I took the accessed URLs together with the parameters and first tried to send them manually from the command line. Observe, that Firebug lets you copy the exact POST/GET parameter as was sent by the browser, when you right click on the POST/GET command in the Firebug Network log. Once I managed to redo all the steps manuall from the commandline I transferred what I did into a script and edited it until it was nice and would run through:

$ history > test_my_web_app

#!/bin/bash
#
# check availability of:
url="http://www.example.org/my-app" 

# exit immediately if something goes wrong -> test fails!
set -e

echo "Testing $url" 

# save an error log in case there's an error so that
# we can analyse it further
error_log=$( mktemp )
cookies=$(   mktemp )

echo "In case the tests fail, you'll find the error log under $error_log" 
# send everything that goes to STDERR to the log
exec 2>$error_log

echo "Getting a cookie..." 
wget --save-cookies $cookies "$url" -O /dev/null

# we could put username/password directly into the script but we're cautuous and don't
echo -n "please enter user for $url: ";     read    user
echo -n "please enter password for $url: "; read -s pass
echo

echo "Logging in..." 
# if we logged in successfully, then the webapp will send us an html page
# that contains the following string:
to_verify="You're logged in" 
# POST to the webapp:
wget --load-cookies $cookies \
     --post-data "login=Login&user_login=$user&user_password=$pass" \
     --save-cookies $cookies \
     --keep-session-cookies \
     "$url/login" \
     -O - \
| grep -q "$to_verify" 

# now we're logged in

echo "Query component DB..." 
# First search the DB (parameters need to be URI encoded...)
num_components_found=$(
  wget --load-cookies $cookies \
       --save-cookies $cookies \
       --post-data 'action=set_params&crit%5Bcolor%5D=red&crit%5Bmin_size%5D=0' \
       "$url/prepare-picture" \
       -O - \
  | grep 'Number of matching components:' \
  | sed 's/.*Number of matching components:<\/td><td>//; s/<.*//' )

echo "Make sure that the app found something..." 
[ $num_components_found -gt 0 ]

echo "Generate and get picture..." 
wget --load-cookies $cookies \
     --save-cookies $cookies \
     --post-data 'index=0' \
     "$url/get-pic" \
     -O /dev/null

echo "Hooray, $url successfully tested" 

rm $error_log
rm $cookies
<pre>

Surprisingly easy, isn't it?

Occassionaly we’ve been seeing messages like:

spam acl condition: error reading from spamd socket: Connection reset by peer

in /var/log/exim4/paniclog

But lately the problem has become much more persistent. I found out, that there’s some arsehole spammer sending us 500K spams every ca. 70 minutes. From /var/log/mail.log:

Jul 17 08:52:32 mail spamd[477]: spamd: identified spam (7.9/5.0) for spamd:1004 in 288.5 seconds, 505669 bytes.

Now since spamassassin would take that long (nearly 5 minutes!!!) to find out whether that mail is a spam or not, exim would simply timeout it’s connection to spamassassin and the result was the message seen above.

What I did was to use sa-compile to compile the SA rules to “native code” and to enable “Rule2XSBody”, which on Debian lives in /etc/spamassassin/v320.pre.

This seems to make SA quite a lot faster and to use less CPU.

If this won’t help we could also restrict the maximum size of messages to scan.

Yours Tomáš Pospíšek

Die internationale Open Source GIS-Konferenez FOSS4G findet dieses Jahr vom 6.-9. September 2010 in Barcelona statt.

Sourcepole ist mit drei Vorträgen vertreten:

• Comparison of Open Source Virtual Globes
• Performance and statistical analysis of WMS servers
• SpatiaLite, the Shapefile of the future?

Zusätzlich unterstützen wir den QGIS-Workshop und nehmen mit dem QGIS-Mapserver an WMS Performance Shootout teil.

Since I found Michel de Boer’s explanation of many of the fundamental SIP concepts in his 2005 dutch presentation of the Twinkle SIP phone very short and easy to follow - contrary to a lot of other material about SIP out there - I’ve translated it to english.

Here it is: Twinkle - a SIP softphone for Linux

Well, sometimes one wonders why people write tools that wheight a megabyte, when it’s easy to stick together one or two tools with a script and have the same thing.

Here’s a ruby script that generates a ping latency graph (target is google):

#!/usr/bin/ruby
#
# based on
#   http://oss.oetiker.ch/rrdtool/prog/rrdruby.en.html example
# with help of the tutorial
#   http://oss.oetiker.ch/rrdtool/tut/rrdtutorial.en.html
# and the docu
#   http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html
# 
# placed in the public domain by Tomáš Pospíšek

require "RRD"

$rrd    = `mktemp /tmp/tmp.XXXXXXXX`.chomp
$grafik = `mktemp /tmp/tmp.XXXXXXXX`.chomp
$start  = 0

def now()
  Time.now.to_i
end

def create()
  $start=now()
  RRD.create(
    $rrd,
    "--start", "#{$start - 1}",
    "--step", "#{1}",       # every 1s
    "DS:pinglatency:GAUGE:10:0:200", # 10s of missing data are OK, min=0, max=200 (ping time)
    "RRA:AVERAGE:0.5:1:#{60*30}") # 60 seconds * 30 minutes
  puts "RRD database created #{$rrd} at #{$start}"
  puts "Your graph is at #{$grafik}"
end

def ping_latency
   `ping -c 1 -n -w 1 google.com` =~ /.* time=(\d+)/
  # 64 bytes from 74.125.43.147: icmp_seq=1 ttl=52 time=31.6 ms
  return $1
end

def update()
  puts "updating #{$rrd} with #{now()}:#{ping_latency()}"
  RRD.update($rrd, "#{now()}:#{ping_latency()}")
end

def generate_gfx()
  # puts "generating graph #{$grafik}"
  RRD.graph(
   "#{$grafik}",
    "--title", " Ping latency to Google",
    "--x-grid", "SECOND:10:MINUTE:1:MINUTE:2:0:%X",
    "--start", $start.to_s,
#    "--end", # -> now
    "--interlace",
    "--imgformat", "PNG",
    "--width=450",
    "DEF:pinglatency=#{$rrd}:pinglatency:AVERAGE",
    "AREA:pinglatency#00b6e4:ms")
end

#main
create()
while(true)
  update()
  generate_gfx()
  sleep 1
end

It generates a graph like this:

Das erste deutschsprachige QGIS User Meeting in Bern war mit rund 80 Teilnehmern ein grosser Erfolg.

Die Folien der Präsentation des OpenLayers-Plugin sind hier erhältlich.

Die gesammelten Präsentation werden auf der QGIS-Homepage veröffentlicht.