Offline Programmatic Generation of Web Pages

Stephen B. Jenkins

About the Author

I am the senior programmer/analyst for the
Aerodynamics Laboratory of the
Institute for Aerospace Research,
National Research Council of Canada
Unix user since late 80s
Perl programmer since early 90s
This is the 6^th YAPC at which I've presented

About the Paper

Originally published in
;login: Magazine,
Vol. 29, No. 5, Oct. 2004 by
USENIX, The Advanced Computing Systems Association
These slides give a very brief visual overview of the concepts
The paper and the source code for Example #1 are available at: www.Erudil.com/hacker/publications.html#paper19

Three Ways to Provide Web Content

Static HTML pages
Dynamic CGI programs
Or a compromise between the two above:
Offline Programmatic Generation (OPG) of web pages

What is OPG?

Using programs to generate web pages at time and place of your choosing
Run programs manually or via cron

When to use OPG

When your data changes rarely
When your ISP doesn't allow CGI programs
When you (or your boss/client) are paranoid about security
When your web pages are:
- large
- slow to generate
- complex to modify

Data Sources

<DATA> segments
Text files
Server log files
CSV/Spreadsheet files

Four Examples of OPG

Music studio schedule
Hockey league statistics
Log file summary
Conference web site

Music Studio Schedule

Public web page showing available time slots
Private web page showing student info, etc.
Information in a <DATA> segment

Ex. 1 - Studio Schedule - Code

#!/usr/bin/perl
use strict;
use warnings;

my $title    = 'Teaching Schedule';
my $colwidth = 'width=75';
my %colorfor = ( 'bg'      => '"#D8E8D8"',
                 'hddark'  => '"#336666"',
                 'hdlight' => '"#FFFFFF"',
                 'choir'   => '"#FFCCCC"',
                 'student' => '"#CCFFCC"',
                 'avail'   => '"#FFE7CC"',
                );

my $html1 =<<EOF;
<html><head><title>$title</title></head>
<body bgcolor=$colorfor{'bg'} text=$colorfor{'hddark'}>
<table border=0>
<tr><th colspan=8 bgcolor=$colorfor{'hddark'}>
<font color=$colorfor{'hdlight'} size="+2">$title</font></th></tr>
<tr>
EOF

foreach ( qw( Time Mon. Tue. Wed. Thu. Fri. Sat. Sun. ) ) {
    $html1 .= "<th $colwidth bgcolor=$colorfor{'hdlight'}>$_</th>";
}
$html1 .= "</tr>\n";

my $pri = "";
my $pub = "";

while( <DATA> ) {
    next unless /^\d/;

my $time = substr($_, 0, 6, "");
    $pri .= "<tr><td bgcolor=$colorfor{'hdlight'} align=\"center\">$time</td>";
    $pub .= "<tr><td bgcolor=$colorfor{'hdlight'} align=\"center\">$time</td>";

my @days = /.{1,4}/g;
    @days = splice @days, 0, 7;
    foreach ( @days ) {
        my $bgc = $colorfor{'avail'};
        if( /\S/ ) {
            if( /CJ|off/ ) { $bgc = $colorfor{'bg'};      }
            elsif( /C\d/ ) { $bgc = $colorfor{'choir'};   }
            else           { $bgc = $colorfor{'student'}; }
            $pri .= "<td bgcolor=$bgc align=\"center\">$_</td>";
            $pub .= "<td></td>";
        } else {
            $pri .= "<td bgcolor=$bgc align=\"center\">available</td>";
            $pub .= "<td bgcolor=$bgc align=\"center\">available</td>";;
        }
    }
    $pri .= "</tr>\n";
    $pub .= "</tr>\n";
}

my $html2 = "</table></body></html>\n";

open  PRI, ">private.html" or die "Oops: $!";
print PRI  "$html1$pri$html2\n";
close PRI;
open  PUB, ">public.html" or die "Oops: $!";
print PUB  "$html1$pub$html2\n";
close PUB;

Ex. 1 - Studio Schedule - Data

Ex. 1 - Schedule - Public Page

Ex. 1 - Schedule - Private Page

Hockey League Statistics

Calculate and show statistics for players and goalies
Data in a text file

Ex. 2 - Hockey Stats - Data File

Ex. 2 - Hockey Stats - Players

Ex. 2 - Hockey Stats - Goalies

Log File Summary

Large eventlog files
Formatted like Apache logs
Summarise event numbers over previous day, week, month, quarter, and year
Article about this application will appear in Dr. Dobb's Journal, Nov., 2005

Ex. 3 - Event System - Log File

1088683200  34:75   jabba          modxxx.pl  updatenotify     0   Sent update message to 132.246.34.198:8450
1088683200  34:75   jabba          modxxx.pl  main             0   User on m2-vtdas.iar.nrc.ca has updated /usr2/people/wtdas/home/wtdas/wtdata/wt808/cfgdata/Ext808.cfg
1088683205  34:74   QNX            Dispatch   ProcessRequest   0   Data Button Process
1088683206  34:74   QNX            WTDAS      WTMessage        0   data - Requested Remote_Systems neff, vme
1088683207  34:74   QNX            WTDAS      WTMessage        3   data ERROR -  'neff' Didn't reply with DONE
1088683212  34:177  M2-SysDev4     wt803Das   wt805_HotWire.vi 1   TCP Open Connection in SendTCP.vi->wt805_HotWireSurvey.vi
1088683323  34:74   QNX            Dispatch   ProcessRequest   0   Data Button Process
1088683324  34:74   QNX            WTDAS      WTMessage        0   data - Requested Remote_Systems neff, vme
1088683325  34:74   QNX            WTDAS      WTMessage        3   data ERROR -  'neff' Didn't reply with DONE
1088683330  34:177  M2-SysDev4     wt803Das   wt805_HotWire.vi 1   TCP Open Connection in SendTCP.vi->wt805_HotWireSurvey.vi
1088683692  34:198  m2-wtdas-inf2  INF2       ControlObject.vi 1   Obtain Queue in Create Queue.vi->ControlObject.vi->Main.vi
1088683693  34:198  m2-wtdas-inf2  INF2       Main.vi          0   Started
1088683700  34:198  m2-wtdas-inf2  INF2       Client.vi        1   TCP Write in ClientSendPacketCopy2.vi->Client.vi
1088683708  34:198  m2-wtdas-inf2  INF2       LoadChildObj.vi  3   Invoke Node in LoadChildObject.vi->Temp.vi Error loading vi ReadTemperature.vi Path -> C:\My Documents\InstrumentRoot\te
1088684341  34:74   QNX            Dispatch   ProcessRequest   0   Data Button Process
1088684342  34:74   QNX            WTDAS      WTMessage        0   data - Requested Remote_Systems neff, vme
1088684355  34:74   QNX            WTDAS      WTMessage        0   data Notification -  Start of RUN 7 using Tare 6. This is 'yaw' run.
1088684617  34:74   QNX            Dispatch   ProcessRequest   0   Data Button Process
1088684618  34:74   QNX            WTDAS      WTMessage        0   data - Requested Remote_Systems neff, vme
1088684655  34:74   QNX            Dispatch   ProcessRequest   0   END RUN Button Process
1088684663  34:74   QNX            WTDAS      WTMessage        0   endrun - Requested Remote_Systems neff, vme
1088733604  34:75   jabba          mirror.pl  main             0   Command line equivalent: mirror.pl -v /usr/tmp/.webplot /usr/people/wtdas/home/wtdas/wtdata/wt808/webplot webplot$
1088733609  34:75   jabba          mirror.pl  main             0   Command line equivalent: mirror.pl -v /usr/people/wtdas/home/wtdas/wtdata/wt808 /usr/people/wtdas/m2-endormount/wt808 .

Ex. 3 - Event System - Daily Sumary

Ex. 3 - Event System - Index

Conference Web Site

Presentation list
Author list
Conference program
Data in an Excel file

Ex. 4 - Conference Web Site - Excel Data

Ex. 4 - Conference Presentation List

Ex. 4 - Conference Author List

Ex. 4 - Conference Program - Monday

Wrap Up

Laziness
- get others to do the grunt work of maintaining web sites
- if there are no others, get cron to do it
Impatience

they can do it whenever they want
no need to bug me, or wait for me

Hubris
- gives a consistent, professional look

Presented at YAPC::NA, Toronto, Canada, June, 2005