What's New in AllNews Version 1.20 Build Date Feb 15, 1998 --------------------------------------------------------------------- Some vacation time, a new bigger, faster PC, and a growing irritation with spam has led to major upgrades in AllNews. I am now supporting Win31 and Win95, and expect to add WinNT soon, all with the same program version. Since AllNews uses only the 16-bit API and WinSock, it MAY be compatable with OS2/Warp. Notable improvements: Kill/Watch file Anti-spam/spew controls are now implemented Automatic decoding of attachments Enhanced long-filename support Individual group properties, including "From:" and signature files I hope to have a Win95 version of DCD out soon. Any tips on accessing the DUN and TAPI for detection of DCD loss and re-establishment of modem session would be appreciated. I want to put the indicator icon in the system tray. --------------------------------------------------------------------- bugs fixed: Detection of Win31/95/NT corrected (Win32S, OS/2 detection unknown) PANIC when opening too many groups fixed improved handling of articles with bad date improved handling of import article with missing post headers Famous GPF when closing AllNews fixed Accidental overwriting of posts when CONFIG file is replaced has been fixed handling of posts containing embedded ctl-Z (end of file marks) Horizontal scrolling of articles list in Win95 fixed Reserve Servers - improved handling of authentication mismatches improved access refusal handling - update server record retesting server info --------------------------------------------------------------------- Features now implemented ==> Config | Misc | TimeZone from dropdown list, +/-hhmm Config | Files Config | Filters (Reworked Config | Decoding (New set of options Config | Posting (moved to Groups | Posting Default properties for all groups, overridable per group --------------------------------------------------------------------- New features implemented KillFile is implemented Config|Filters|Spam Control Spam filter implemented for those servers supporting the Xref: optional XOVER field Groups properties VERY reworked - automatic upgrade for existing users new decoding features - by default existing users see same behaviour better - Long Filename - Truncate; preserve sequence Name Collision - skip decode; treat as error --------------------------------------------------------------------- Various changes "Cheats" menu item moved under "Files" dropdown menu item to reduce menu size --------------------------------------------------------------------- Note to existing users: When you run the new version of AllNews for the first time, it automatically upgrades your old files to a new format. If you are considering testing this version, and then backing off to an older version, back up all of the *.DAT files first, and restore them after the test. If you try to run an older version of AllNews with the new format files, you will observe bizarre behaviour, and may not be able to access some files. After the upgrade, AllNews will continue to operate the same as you had configured it in the previous version, by default. You must change settings to get the new operating modes that are set by default for new users. -------------------------------------------------------------- A word about file allocation As hard drive sizes passed the 512MB mark, the issue of "slack" or "hidden" disk space becomes significant. Files are allocated in increments called "clusters". The size of the cluster depends on the file allocation system and the size of the partition. FAT16, the most prevalent and the only system that can be shared between DOS, OS/2, Win31, Win95 and WinNT has a partiton size limit of 2GB per partition. At that partition size, the cluster size is 32KB. Thus a 100 byte file actually requires 32KB of disk storage. AllNews uses a file to represent each post that has been read. Some other newsreaders may use a single file to represent an entire newsgroup. Each approach has its advantages and disadvantages. AllNews has been optimized for dealing with large, multi-part binaries, where the slack is less significant. The text-based newsgroups have a high ratio of slack-to-post. If you are setting up a new PC, it would be reasonable to create a 512MB partition (logical drive), and direct AllNews to store its files on that partition. I am planning some enhancements that combine the two versions of storage for a future enhancement to AllNews. In my own use, I removed all files and programs from the hard drive letters D: and E:. Then I booted from the "Win95 Recovery Diskette" created during installation of Win95. I used FDISK to first delete all logical disks and partitions above C:. Then I partitioned a number of small (~512MB) logical drives: D:, E:, F:, G:, and H: (110MB). I set Configuration | Files | Temporary to "F:\FILES". I set Configuration | Files | Binaries to "G:\FILES". In Netscape, I selected Edit | Preferences | Advanced | Cache, and set the "Disk Cache Folder" to "E:\NETSCAPE\CACHE". These small files now incur only an 8KB overhead, instead of the default 32KB. On the desktop, I selected Start | Settings | Control Panel | System | Performance | Virtual Memory. I selected "Let me specify my own settings". I changed "Hard disk" to H: (which I partitioned to about 2xphysical memory). This put the windows swap file in its own reserved space, guaranteeint that there will be no fragmentation or allocation problems, and improving overall performance. FAT16 Disk partitions Cluster size <512M 8K 512M-1GB 16K 1-2GB 32K FAT32 (Win95 only NTFS (WinNT only HPFS (OS/2 only references: http://www.pcguide.com/ref/hdd/file/part.htm --------------------------------------------------------------------- Disaster Recovery This is when the power went off (or your 4-year-old pressed the reset button) while you were downloading news posts. AllNews loses, at most, the work in progress at the time of the failure. It cleans up any fragments automatically when it is restarted. This is a benefit of the file allocation and backup strategies used by AllNews. Some other newsreaders (I won't name names, but if it happens to you, you will know it) that maintain a single file, may potentially lose all of the posts ever retrieved in groups that were receiving download data. After rebooting, run SCANDISK (DOS/Win31) to reclaim "unallocated" space, and to adjust the affected file sizes. In those newsreaders that use the group/ file strategy, you MUST re-synchronize the groups before downloading any data. This is usually some mechanism similar to "Compress Databases". --------------------------------------------------------------------- The Config | Files and Config | Decoding settings are now fully implemented. Note that CONFIG.DAT and CONFIG.BAK are kept in the start-up directory with the AllNews.EXE program file, as the CONFIG file is where these settings are kept. The start-up directory is set in Win31 by highlighting the AllNews Icon, then Files | Properties | Working Directory:, and in Win95/WinNT by highlighting the AllNews Icon, then clicking the right mouse button - Properties | Shortcut | Start in:. --------------------------------------------------------------------- D)ecode X Automatically decode attachments after fetch Root directory +-- Decoding to directory ----------------------+ | o Decode to root/filename | | o Decode to root/news.group.name/filename | | o Decode to root/news/group/name/filename | | o Decode to root/ngn/filename | +-----------------------------------------------+ +-- Long filenames -------------+ +-- On name collision ------------+ | o Truncate to DOS 8.3 | | o Increment filename to unique | | o Truncate, preserve sequence | | o Increment extension to unique | | o Truncate to last 8.3 chars | | o Overwrite older file | | o Allow long filenames | | o Skip decode; treat as error | +-------------------------------+ +---------------------------------+ +-- After successful decode ----+ +-- On decoding error ------------+ | o Leave post as received | | o Save and leave in message | | o Remove attachment from post | | o Save and remove from msg | | o Delete post | | o Do not save; leave in msg | +-------------------------------+ | o Delete post | +---------------------------------+ New decoding options Accessed from the Config menu item | Decode dialog tab. Provides enhanced support for long filenames, both for file systems that do (Win95, WinNT, OS/2 HPFS) and for those that do not support long filenames (Win31, OS/2 FAT). If yours does not, the behaviour of long filenames is unpredictable, and selection of these options is not recommended. There are several "8.3" filename mangling options to support these systems. "Automatically decode attachments after fetch" When this option is checked, each item is submitted to the decoder after its download completes. Multi-post items will be decoded when all parts have been retrieved. "Root directory " The root directory name is controlled under the "Files" tab. This is the directory that all decoding is referenced to. "Decoding to directory" Allows the selection of a convention destination directory(s). You can decode all of them into a single directory, or into different directories whose name are derived from the newsgroup. The "root/news.group.name/filename" option is only recommended for filesystems that support long filenames. The other options mangle the newsgroup name to determine the name of the target directory (folder). If the directory does not exist, it is created automatically as files are decoded for it. If your disk navigator caches the directory tree, you may need to refresh your list to access the new files. "Long filenames" This option specifies how to handle the filename when the decoder attempts to decode a file with a long filename. "Allow long filenames" should only be used on those filesystems that support long filenames. "Truncate to DOS 8.3" specifies simple truncation. "Truncate to last 8.3 chars" is an alternate form. Many times a series of related posts differ only in a name by a sequence number at the end of the name. "Truncate, preserve sequence" specifies 8.3 truncation, where the sequence number is extracted from the long file name, and then appended to the truncated name. This preserves the sequence numbers for comparison with reposts. "On name collision" Specifies the action the decoder is to take when a decode operation will decode to a filename that already exists in the target directory. "Increment filename to unique" means that "aaaaa.JPG" would be promoted to "aaaaa1.JPG", "ABCDEFGH.JPG" to "ABCDEFG1.JPG", etc. This is what previous versions of AllNews would do. It can be confusing when combined with simple 8.3 truncation and applied to a sequenced list of posts with long names. The resulting filenames may have little relation to the original names of posts. "Increment extension to unique" applies the same logic to the extension, paying attention to long filename options selected. This may be a problem in some groups, such as the warez groups, where a series of posts may increment the extension to organize components of a large multipost series (x.ARJ, x.A01, X.A02). In this situation, the results may become confused and unusable to the program that recombines the parts. "Overwrite older file" I don't know why anyone would want to blindly overwrite the older file, but it was easy to implement. "Skip decode; treat as error" is my personal favorite. A name collision terminates decode of this file. Handling then proceeds as specified in "On decoding error". "After successful decode" This option tells the decoder what to do with the original posts once the attachments have been extracted. "Leave post as received" leaves the post(s) in the article list for the newsgroup, but the "X" indicator is changed to "d", showing that the post has been decoded. "Remove attachment from post" extracts the attachment information from the post and replaces it with a message indicating the filename where the decoded attachment was stored. Any accompanying message text is also left in this file. Note that this leaves a series of small text files, and see the section "A word about file allocation". "Delete post" deletes the post once all attachments are successfully decoded from the post. "On decoding error" This option specifies what the decoder is to do when an error is encountered while decoding the attachments. "Do not save; leave in msg" allows the greatest flexibility in recovering the attachment after other activity completes. "Delete post" also will delete any text posts, so be careful if you chose this option. General Notes I prefer "On name collision" - "Skip decode; treat as error", "After successful decode" - "Delete post"; "On decoding error" - "Do not save; leave in msg". This combination gives the best performance for unattended downloading for binary leaches, when each group decodes to a separate directory. After downloading completes, groups with similar interests are then combined into a different directory, allowing comparison of files with the same names for crossposting (this will automatically be taken care of in an upcoming version of AllNews). Then, another pass through the newsgroups, selecting "Decode" for the posts that did not decode during the download picks up the posts that had name collisions (reposts) within the group. Sometimes, I see that the same attachment is being attached twice in the same post, or that a post has multiple attachments with long filenames that resolve to the same filename because of the name mangling rules, or a series of posts that each include a common attachment. These types of posts will cause the decoder to report a failure (since that option is selected). This case is handled by selecting the post(s) in question, then pressing the "Manual Decode" button, and marking the "Increment extension to unique" box. The decode then proceeds and I can change the filenames later. If the decode still fails, see the section describing Manual Decoding". --------------------------------------------------------------------- Killing vs. Deleting Posts "Deleting" a post means marking the post deleted and removing it from the display. It remains in the article file and is processed with the rest of the list until the expiry rule for deleted posts is met. "Killing" a post means removing it from the Articles file. Its space is reclaimed, and it is not processed any further. A killed post may be re-introduced if it is later found in the list of posts on another server. A kill rule that can clearly define undesirable posts (such as a spew attack) as a set of posts that will NEVER be desired can improve performance and reduce disk and memory resources required by specifying a "Kill" action. These posts can then be recovered only by editing the Group | Servers | Highest Article Read, and re-fetching the list of posts after the rule has been deactivated, modified or deleted. Posts that have been "deleted" by kill rules can be recovered either by pressing the "Show Deleted" button or by pressing "Kill" and specifying the "Action: Select", "Apply to: All Posts". --------------------------------------------------------------------- Config | Filters (Global option) Kill/Delete cross-posted to more than n newsgroups If "n" is 0, this rule is ignored. See note above about Killing vs. deleting. 2 different rules may be specified. One for KILL, where obvious spamming is ocurring (e.g., more than 10 groups). One for DELETE where it is possible for the posts to be desirable, but not probable, that the posts are desirable (e.g. 4 or 5 related groups). other items grayed -- will be implemented in later versions --------------------------------------------------------------------- Edit Group | Filter spew - Kill/Delete (Option for newsgroup) Delete will allow parts 0 and 1 to appear in article list. Entire post can be recalled via Articles | Kill | Select, or by highlighting one part and clicking the "Parts" button. If Kill, other parts are not available, as they were not saved. --------------------------------------------------------------------- --------------------------------------------------------------------- KillFile KillFile can be accessed from Main Menu Item Main Toolbar right mouse button in groups menu (for a group) Kill Article in article list right mouse button in article read Each rule in the Killfile has a group number associated with it. Group 0 rules are applied to all groups. When a list of posts is retrieved from a server, the killrules for group 0 and the group being retrieved are applied in the order they are displayed in this list. In the Killfile screen, you may alter the order of killrules, add and delete rules, and edit existing rules. The screen used for editing and inserting rules is the same as that under the Articles | Kill Posts button. There are more options available now in this screen. Some options are not available in some modes, are grayed out. Some buttons operate differently in some modes, and have different text on them. All features displayed in each screen are fully operational. +- Action -+ +- Apply To --+ +- Kill Criteria ------+ +- Retain Kill Rule ---+ | | | | | | | | | o Kill | | o All posts | | o Exact Match | | o Single application | | o Delete | | o Shown | | o Article Parts | | o 1 day | | o Fetch | | o Selected | | o Begins with string | | o 1 week | | o Mark | | o New posts | | o Contains String | | o 1 month | | o Select | | | | o All selected | | o forever | | | | | | x Case sensitive | | x of inactivity | +----------+ +-------------+ | | | | +----------------------+ +----------------------+ o All Groups x Enabled o x Subject: ______________________________ x From: ______________________________ "Action" specifies the action to apply to articles which match the criteria of the kill rule. Kill and Delete operate as described in "Killing vs. Deleting Posts". "Fetch" schedules the post for retrieval, "Mark" causes the post to be tagged with the "M" mark. "Select" causes the post(s) to be selected in the articles list. "Fetch" all posts whose subject includes the word "server" is an interesting rule for those who are interested in open usenet servers. However, in the warez groups, posts often have subjects complaining about server problems. In those cases, undesired large binaries may be rtetrieved. An alternative is to "Mark" these posts. When encountered in the list, they may then be fetched or deleted, as desired. "Select" does not make sense as an action for lists of new posts, and is disabled for that option. "Apply To" is only useful in the Articles | Kill Posts selection. "All Posts" means that Deleted (but not Killed) posts should be considered, as well as those that are displayed in the list. "Shown" means only those posts that are displayed (not Deleted). "Selected" means those posts in the display that are selected. "New posts" means that posts that are in subsequently retrieved lists of posts should be considered. "Kill Criteria" specifies the selection decision to be applied to the posts selected by the "Apply To" criterion. "Article Parts" specifies that those posts that are parts of the selected multi-part post are to be operated upon. "All selected" means that the posts selected in the display are to be oparated upon. The remaining options interact with the "From:" and "Subject:" options. "Exact match" means that From and Subject (if checked) must exactly match those fields in the post. "Begins with string" and "Contains string" mean that the string in the From/Subject (if checked) must begin or be found in the corresponding fields of the post, respectively. "Case sensitive", if checked, means that the comparison must match upper/lower case. If not checked, either upper or lower case will match. Leading, trailing, and embedded blanks are significant. "Retain Kill Rule" specifies how long the kill rule should be retained in the killfile before being automatically removed. When Articles | Kill Posts has been selected, "Add to KillFile" causes the new kill rule to be added to the killfile for subsequent use, without being applied. If "Apply" is pressed, and "Single application" is selected, the kill rule is applied immediately, but not added to the kill file. If another retention is selected when "Apply" is pressed, the kill rule is applied immediately, AND added to the killfile. Selecting "Add to KillFile", with "Single application" is useful when the list of posts indicates that the poster made a post of interest that has not yet reached your server(s). The "Subject:" string may be edited to identify the missing post. If "Fetch" is specified, the post will be automatically retrieved (and perhaps decoded) when AllNews discovers a server carrying the post(s). "1 day", "1 week", and "1 month" indicate that the killrule should remain in the killfile until that amount of time has elapsed. If "of inactivity" is checked, the retention interval is restarted each time the killrule fires. "Forever" specifies that the killrule is to remain in the killfile until it is manually deleted. Each killrule may either be applied to "All Groups", or to the group currently active (the displayed in the killrule screen). In Articles | Kill Posts, if "All Groups" is specified, "Apply" will only apply the rule to the active newsgroup, but the killrule is applied to subsequent lists of posts for all groups. "Enabled" indicates whether the killrule is allowed to fire. It may be turned on and off, as desired, in KillFile | Edit. "Subject:" and "From:", when checked, indicate that these strings are to be compared to the corresponding fields in new lists of posts, as per the "Kill Crtiteria". If both are checked, BOTH strings must match. I am experimenting with a kill rule that is "All Groups" "Delete" posts which "contains string" in the "Subject:" "(0/1)". This deletes part 0/1 from posts. This part is normally not needed to successfully decode the part 1/1 attachment. Should it be necessary, I can click on the 1/1 post and select "Parts". I am then asked if I wish to undelete, and fetch, the deleted post.