levselector.com New York
home > usenet Email

Usenet

intro
books
nntp commands
perl

 
intro home - top of the page - email

NNTP (Network News Transfer Protocol, port 119, RFC-997) - is used for managing the notes posted on Usenet newsgroups. NNTP replaced the original Usenet protocol - UUCP (UNIX-to-UNIX Copy Protocol).

http://groups.google.com/googlegroups/basics.html - very good explanation of the basics of usenet
 
Before the Web and web browsers, and before email became ubiquitous, online communication meant posting text messages on electronic bulletin boards where others could read and reply to them. 

Usenet began as a collection of these bulletin boards (now called discussion forums or newsgroups) started in 1979 by Steve Bellovin, Jim Ellis, Tom Truscott, and Steve Daniel at Duke University.

Over the years, the number of such newsgroups has grown to the thousands, hosted all over the world and covering every conceivable topic about which humans converse. While there was incredibly valuable information available in the discussions taking place on newsgroups, finding that information could be an exercise in frustration and futility. Someone would start a topic with a posting on a bulletin board. Someone else would reply. This initial post and response now constituted a "thread" on the topic. The thread might grow to include dozens or even hundreds of individuals responding to the first post or any that came after it. They might start threads of their own as offshoots of the original discussion. Think of that initial post as a single cell dividing again and again, mutating and expanding geometrically with no predefined direction. The result is likely something you'd find in a low budget horror movie. Finding a specific bit of information in Usenet was an equally horrific task. To make it more complex, almost all newsservers expire messages after a few days or, at most, a few weeks. Expired messages are deleted from the live discussion forums and aren't viewable or searchable by users.

In 1995, Deja News was created to provide a user-friendly interface to Usenet. Deja began archiving and indexing messages so they could be searched and sorted, turning an ephemeral and unmanageable resource into a reference tool that was fairly easy to use. 

In February 2001, Google Inc., a company dedicated to providing access to all information online, acquired the Usenet discussion service from Deja.com, including its entire Usenet archive of more than 500 million messages -- over a terabyte of human conversation. Google has expanded accessibility to the Usenet database through deployment of improved search and browsing tools and integration of the full archive with more recent postings. The combined database of more than 650 million messages already constitutes the largest collection of Usenet data on the Web and is growing at a rapid pace. 

All messages are structured in a tree structure of groups. Group names follow directory structure with dots separators, for example:  alt.comics.batman. Usenet is a distributed system of servers exchanging messages - and clients reading and posting messages.

Servers:
The first implementation of NNTP Server Reference Software for Unix was done by Phil Lapsley for BSD Unix.  Since 1987 it was supported by Stan Barber.

There are many NNTP servers available for different platforms.  The most notable is INN (InterNetNews). It was originally developed by Rich Salz and now maintained by the ISC (Internet Software Consortium). INN includes an nntpd server to communicate with other news servers, and nnrpd - a newsreading server. Also C News - it is available on more platforms, requires less resources to run, consists of many smaller programms.

There are servers from Netscape, Microsoft, almost any major vendor. They may differ in many ways, for example, how they store the postings (as individual flat files ~3K average size, or some other way), how big is the archive (and is it searcheable and does it have a web interface, etc.).

http://groups.google.com/googlegroups/deja_announcement.html - Deja - www.deja.com - a major Usenet server installation and archive with a web interface to search through ~650 mln messages (over a terabyte of data) accumulated since 1995 (as of this writing - May 2001).  Deja was recently aquired by Google.

Clients (newsreaders):
NNTP clients are included as part of browsers - Netscape, Internet Explorer, Opera, and others. There are also separate client programs called newsreaders (here are some from www.tucows.com - Free Agent (Forte) , RMRNews,  Hogwasher, OUI (Offline Usenet Interface), Knews, News Force, Re: News, Paperboy, bMOBILE News, Pineapple News, Ink Spot).

Here are some FAQs and general information:
http://www.mibsoftware.com/userkt/userkt.html - Usenet Rapid Knowledge Transfer
http://www.faqs.org/faqs/usenet/software/ -
http://www.faqs.org/faqs/usenet/software/inn-faq/ -
http://www.academ.com/academ/nntp/ -
http://www.eyrie.org/~eagle/faqs/inn.html -
http://www.usenet.about.com/internet/netconference/cs/serversoftware/ -
http://www.cs.uu.nl/wais/html/na-dir/usenet/software/inn-faq/.html - INN
http://www.cs.uu.nl/wais/html/na-dir/usenet/software/inn-faq/part1.html - INN
http://www.cs.ruu.nl/wais/html/na-dir/usenet/hierarchies-config.html -
DNEWS News Server:
http://netwinsite.com/dnews.htm -
http://www.buller.se/DNews/ -
http://serverwatch.internet.com/reviews/news-dnews.html -

 
books home - top of the page - email

 
"Managing Usenet" - by Henry Spencer & David Lawrence, 1998, O'Reilly
"Managing Uucp and Usenet (Nutshell Handbook)--[Paperback] O'Reilly, Tim; Todino" - 10th edition, 1992
"Administering Usenet News Servers : A Comprehensive Guide to Planning, Building, and Managing Internet and Intranet News Services" , 1997.

 
NNTP commands home - top of the page - email

 
NNTP Command  -  What It Does
LIST  -  list available newgroups
NEWSGROUPS date time  -  list newsgroups created after date/time
NEWSNEWS grps date time  -  list articles after date/time in grps
GROUP grps  -  list available articles in given newgroup(s)
ARTICLE id  -  transmit article id
POST  -  have article that was posted here
IHAVE id  -  have article id here
STAT  358300  -  fetch article into server
HEAD  -  get headers of currently fetched article
BODY  FROM: reverse-path  -  get body of currently fetched article.
NEXT  -  fetch next article into server
QUIT  -  terminate NNTP session

 
telnet news-server 119
...
200 llyr NNTP server v1.5.11 ready at Wed Jan 19 10:40 2000
LIST
215 Newsgroups in form "group high low y/n/m".
...
soc.culture.china 0000358310 357649 y
...
.
GROUP soc.culture.china
211 175 357649 358310 soc.culture.china
STAT 358300
223 358300 <7952m0$4bv$1@pan.ip-plus.net> Article retrieved.
HEAD
221 358300 <7952m0$4bv$1@pan.ip-plus.net> Article retrieved; head follows.
Xref: hwcee soc.culture.taiwan:210593 soc.culture.china:358300
...
X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3
.
BODY
222 358300 <7952m0$4bv$1@pan.ip-plus.net> Article retrieved; body follows.
Answers from / Antworten von Chris Marx c/o PAF <pakaf@1.lol.li>
...
.
NEXT
223 358301 <36B60971.6964@webcom.com> Article retrieved.
QUIT
205 llyr closing connection. Goodbye.

 
Posting news articles by NNTP is straightforward: 

telnet news-server 119
200 llyr NNTP server v1.5.11 ready at Wed Jan 19 11:58 2000
POST
340 Continue posting; Period on a line by itself to end
Relay-Version: version newspost 1.00 2/2/99; site cee.hw.ac.uk
Posting-Version: version newspost 1.00 2/2/99; site cee.hw.ac.uk
Path: savitar
From: joe@cee.hw.ac.uk (Joe Bloggs)
Newsgroups: soc.culture.china
Subject: noodles
Date: Wednesday, 19-Jan-2000 12:00:00 GMT
Message-ID: <1431@cee.hw.ac.uk>
 ...
.
240 Article posted successfully.

News articles should be in RFC 850 format. 


 
perl code home - top of the page - email

Net:NNTP (libnet library)
News::NNTPClient

--------  under construction