01/12/99 RMIUG Meeting Minutes - Measuring and Monitoring Web Sites

Dan Murray called the meeting to order at 7:00 and introduced the other members of the RMIUG Executive Committee in attendance: Art Smoot, Alek Komarnitsky (taking minutes), Bryan Buus, and Tom Bresnahan. About 90 people were in attendance. Some misc. announcements from the audience:

- Art Smoot announced the recent passing of Spence Schultz, an early and active member of the RMIUG and several other internet groups in the front range. At his memorial service, several people commented on his involvement with the internet, email, and the web.

- Cecilia Chin-Palles (adastra@csd.net) just relocated from Miami, Fl. to Boulder. She is currently looking for a position as a Network Technician. She has administrative experience with NetWare 3.x, 4.x, and GroupWise, experience with desktop/PC support (WinNT w/s, Win95, Win3.1), email clients (Outlook, Eudora, Lotus Notes, GroupWise), and some experience with Linux. She has trained users of all levels & interacts well with people. Please email her if you want to further discuss her skills, or if you know of an opportunity.

- Colson-Quinn, a Boulder law firm, is pleased to announce the launch of its web page at http://www.colsonquinn.com/. Joyce Colson and Rob Quinn, with over 38 years of collective legal experience, have represented major international corporations and small growing businesses. Colson-Quinn's practice is devoted to helping technology companies, especially small to medium sized, start and grow their business. Colson-Quinn's web page is designed to provide periodic updates of useful technology news, as well as a practical resource guide for Colorado technology businesses. If you have any questions or need further information, please contact Joyce Colson (joyce@colsonquinn.com) or Rob Quinn (rob@colsonquinn.com).

- Darren Powderly (dpowderl@teksystems.com) & Clarissa Ang (clang@teksystems.com) announced that TEKsystems, one of the top technical service providers in the United States, is interested in speaking with information technology professionals in the Denver Metro area. Due to their rapid growth, they are constantly in need of technical professionals with information systems skills at all levels -- including Help Desk specialists, field engineers, systems engineers, project managers and consultants. See their web page at http://www.teksystems.com/ or call them at 303-412-27{21,60}.

- Jun Akiyama (akiy@aikiweb.com), a recent addition to the Rocky Mountain area, has just moved here from the San Francisco Bay Area and is now looking for job openings. Jun was the Technical Manager at Walnut Creek CDROM, the company behind over a hundred CDROMs including the official FreeBSD CDROM and Slackware Linux CDROM as well as the popular FTP site ftp.cdrom.com. He has had plenty of experience in the field of Internet methodologies, project management, web design, and Unix administration. Jun's resume can be found at http://www.aikiweb.com/about/resume.html.

- Tom Smidt (tsmidt@microstaff.com) representing MicroStaff, an Internet technical recruiting company, announced he will make himself available for questions after the meeting for people interested in their contract or permanent opportunities. Tom said additional information and open positions are available by visiting the MicroStaff Web site at http://www.MicroStaff.com/.

Dan then asked the audience some misc. questions. Many people were from Boulder and one person had driven 162 miles from Pueblo (they won a Freshwater t-shirt!). In terms of where people work; a few work in government, a few in public sector, and most folks are employed in the commercial sector. There were about a dozen web developers, a dozen network/system admins, and a dozen software developers. About 1/4 knew of Freshwater and ServiceMetrics before the meeting.

Dan intro'd John Meier. John (john@freshtech.com) is Chief Technical Officer & Founder of Freshwater Software (http://www.freshtech.com/). As Freshwater Software's CTO, John Meier is responsible for guiding the company's technology vision and product engineering efforts. John co-founded Freshwater Software in 1996 and previously worked at US West on broadband applications and at Apple on Newton and Macintosh system software. He holds nine patents.

John started his talk with a story about how the previous night he was trying to get a copy of customers.com, a book about to improve customer service using technology (http://www.customers.com). He went to Amazon.com ... tried to order it ... and it failed! So he jumped over to barnesandnoble.com and ordered it there. Moral of the story: If you are on the Internet and doing business, and something is wrong, your customers won't complain, they will simply jump to another web site. Freshwater's business is providing tools to insure that your company's web site doesn't do this.

One product they sell is SiteScope. This runs local on your site and ensures that the various servers at your site are all working. They also have SiteSeer ... this runs on various spots on the Internet and attempts to connect to your servers (as a user would). It simulates ACTUAL customer inputs and lets you know how this well/fast this is working. I.e. are things starting to slow down and/or break? And if so, you would like to get an early warning about it. Besides "standard" tests (such as ping, SMTP, HTTPD), this has the capability of using customized queries, which can be quite useful to the customer in terms of identifying/isolating problems.

John showed how the path from the customer to your web site is a complex path through many routers, switches, and various servers (sharing functions) at your site ... these all have to work in order for the customer to use your site. But typically these are pretty reliable, and the most common mode of failure is the application itself on the server. Once a problem occurs, it can send a page, send an Email, do an event (fix the problem), and possibly hook into larger Enterprise Systems.

Freshwater often sells SiteScope to ISPs. The ISPs (such as MCI and Digex) provide SiteScope for their hosting customers and may also use the SiteSeer service to measure and enforce service level agreememtns.

SiteScope run as a server on NT, Sun, and SGI ... but can monitor pretty much any OS when it comes to web-stuff. SiteSeer is all web-based.... You sign up for it and get reports over the web.

The whole field of Internet performance/measurement/analysis is being researched - two places where you can find info are at http://www.merit.edu/ipma/ where Merit has various statistics on the "condition" of the Internet. Another place is http://www.w3.org/WCA/ which the W3C Web Group is looking into standards of characterizing how one measures performance.

Audience members asked about how Freshwater compares to Keynote Systems, which puts together a weekly report about performance metrics. Freshwater is more applicable to people who want IMMEDIATE notification of outages/problems ... although there stuff can be used to generate reports.

As John stepped down, Dan gave away more door prizes to those folks who could identify when Freshwater was started (1996) and who co-founded Freshwater with John (Donna Auguste).

Dan intro'd Steve Hultquist (ssh@servicemetrics.com) who is VP of Network Services at ServiceMetrics (http://www.servicemetrics.com/). Steve talked about the impact of design decisions on Web site performance. These design decisions are made by ISPs, hosting site networks, and site designers in application architecture, and information architecture. He also discussed the technology developed by Service Metrics for monitoring Web site performance from an end-user perspective. It allows owners to take clear actions and understand the impacts of design decisions on performance. Steve is responsible for the Professional Services line of business and the Service Metrics network infrastructure. Steve has been heavily involved in the technical nuts & bolts of the Internet since 1986 and appeared on the cover of Network World wearing a biker jacket for the Networld+Interop "Born To Network" campaign.

Steve had a number of slides each highlighting a key point:

Network impacts of performance/business: The faster/more available you are, then the more customer reach and exposure you have. Revenues go up, customer service expectations go up, branding and loyalty become important. This can provide a competitive advantage, and if you are aware of your performance needs, you can lower/control the costs of doing business (buy what you need IF you know what you need).

Issues of Web Performance Elements: Internet itself (backbones, ISP's), backbone link speed, switching vs. routing (former is often faster, but there are trade-offs). Note that if you are trying to assess ISP backbone speed, packet loss is typically a better characterization of "speed" than actual latency/ping time.

Peering issues are a BIG deal - both technically and politically. These are locations where ISP's exchange data - they can be public or private. Because there are many of these and each ISP wants the data off their network as soon as possible, the traffic paths are typically asymmetrical (i.e. two different paths). Public peering points such as MAE East (main Internet Exchange for East Coast) are almost scary to visit - just a "blockhouse" in a parking garage; pretty run-down place, stuff all over the place, everybody just crams stuff in there. The "MAEs" are by far the biggest problem in terms of connectivity on the Internet ... even though Sprint, Ameritech, PacBell, DEC, and others have also brought public exchange points online. This problem is recognized, so some of the big ISPs have decided to do private peering arrangements. BGP (Border Gateway Protocol) is the protocol that is used to determine how we send stuff ... typically 40,000 routes that describes where to go based on peering, etc. Another arrangement is Transit agreements where an ISP has agreed to carry someone else's "default" traffic that isn't handled by other arrangements. Some ISPs may not want to peer with smaller ISPs and force them to make transit arrangements with larger ISPs - this may get passed on as larger costs for customers.

Internet Connection: Speed of connection, diversity of connection (multiple routes), Quality of equipment used (can it hold all 40,000 routes?), possible Telco Issues (beware hungry backhoes! ;-). Your internal Web Site network has issues such as speed and type, using switching vs. routing, firewalls, traffic isolation & shaping (can do different things with different types of traffic - i.e. favor httpd traffic over ftp/mail?). Network monitoring and management is important, but be careful not to make it too intrusive (ex: at Interop, up to 20% of the traffic on the wire is SNMP management stuff!)

Web site servers: issues are numbers and types of hardware/software. You can use load balancing & redundancy. Architectural decisions include do you split everything among various servers, or separate out by function via division of labor (static vs. dynamic content, images, ads?)

Web application architecture can play a big factor in performance. Static vs. dynamic elements, internal vs. external sources (do you download in advance or on the fly?) languages (JavaScript, Java, ActiveX, ASP). What is your security policy and how does it affect performance?

Information Architecture: Amount of information displayed (show all 500 zillion hits, or just top 30 at a time), links between information. Investment Decisions: How do you identify where the problems are? What investments DO result in improvements? How do you keep partners accountable? How can I measure the investment impact? How success can be predicted and proven? How do I watch performance continually (what is normal/acceptable and/or abnormal).

How do you measure performance? Look at server itself (CPU, RAM, response), network (packet loss, ping time, etc), End-user (browser-based). Typically the last one is the most important, but hardest to do ... needs to be consistent, automated, and objective. Metrics include: DNS lookup, connection time, request time, response time, download time, connection teardown time. Can also look at a "per object" time ... i.e. break this down per individual object rather than an entire home page.

ServiceMetrics sells Subscription-based Web Performance Measurements (they assess how fast your pages are). They also have a Professional Services organization that helps analyze that data & your setup; and make recommendations for improvements.

They have distributed "collection" agents around the Internet (that are specifically located on "important" spots (i.e. exactly on the ISP's backbone)). They also have multi-homed agents that are connected directly to 100+ ISP's ... this removes peering arrangements out of the equation and allows you to assess when peering is an issue and when it is not. Steve gave a real-life example of a customer who's page typically loaded in about 5 seconds from various backbone vendors ... except one where it took 30+ seconds. This allowed them to identify that there was a bad peering arrangement with a certain ISP and communicate that to them.

They do polling three times/hour from 10 (soon to be 20) agents, and then store all that data in a Data Warehouse in Boulder where you can slice-n-dice it. For instance, they found sites where the entire performance problem was due to slow DNS lookups (this is typically due to slow DNS servers, slow network connection, DNS mis-configurations (one round-robin record out of sync, so every fifth time, real slow). He showed graphs that show Yahoo performance for front-page download= times during the day. They can also split this out by DNS, connect, request, response, teardown time, and isolate what is causing the slowness.

We brought both speakers up for some questions & answers ... in the text below, Q is the question, and responses are J-John, S-Steve:

Q: How much would it cost to monitor at one web page?
J: Software starts at $500 and higher. Service starts at $1,000/year.
S: Cost is based on volume, so hard to give an answer, but ServiceMetrics's pricing starts at $300/month. They typically target medium/large sites that depend on the Internet for their business.

Q: Can you define exactly what Networking means/is?
S: Networks are made up of Networks are made up of Networks, etc. The Internet is nothing more than a bunch of Networks that are connected together. Since the entire thing is really the connection between networks, they called it "The Internet".

Q: Pls give us a brief background on your company.
S: Service Metrics was started in July/1998 and backed by a couple of VC folks (Softbank and IDG Ventures). Grew from 3 people in July to 15 now.
J: Freshwater founded in 1996 by John & Donna and has VC funding from Mayfield Fund and Mohr Davidow. Knew they were going to build tools for WebMasters - that is their target audience. Went around and talked to various folks to find out what they needed to do their job, and based on that feedback, they develop products/services as needed.

Q: What is the granularity of the measurements:
J&S: Traceroute/response time is to the millisecond ... and if you want all the data, you can get it!
J: We poll however frequently you want because we are more operations oriented and want to tell you RIGHT away if there is an issue.
S: We do three times/hour because we are trying to show the trends.

Q: What is the impact on a "secure" httpd connection versus a "normal" one?
J&S: Significant ... at least two times (end-to-end ... could be more on server resources).

Q: Are any of your tests intrusive?
S: YES ... but the reality is that for any site of significance, they are in the noise. Their total number of hits is three times/hour times 10 sites ... so 30 hits an hour should not be bad.
J: YES ... but this is good ... because you HAVE to check every single piece in an end-to-end test ... testing the individual pieces no longer is good enough.

Q: How long does this suite of tests take? S: Typically spends 20-30 seconds per URL. They can even use a "test VISA" card that allows you to go all the way through purchase and credit card authorization ... but then it is not charged ... true test of the ordering process on the web site. J&S: They both check the validity of the returned HTML to insure the right stuff is returned.

Q: I'm interested in your products - how can I try 'em out?
J: You can try a free 10-day trial of SiteSeer and SiteScope; see our web page at http://www.freshtech.com/
S: Contact us via phone or e-mail (see http://www.servicemetrics.com/) and we'll be happy to talk with you about complimentary trials. We do offer a free Web performance audit.

The Meeting ended just after 9:00. A reminder about upcoming RMIUG events:
Mar: Building Brand Beyond the Web: Panel Discussions on Responsible Email Marketing.
May: TBD
Jul: All you wanted to know about domain names but were afraid to ask!
Sep: Tips and Tools for Web Site Development
Nov: Y2K Armageddon, the coming Internet/World Meltdown! ;-)

(To suggest a topic, pls send your idea to rmuig-comm@rmiug.org)

Minutes compiled, edited, and submitted by Alek Komarnitsky