From HECTOR SANTOS · 1996-04-05 13:36 · #73072 in FREE25
In reply to Re: orig MSGID -> new REPLY by HECTOR SANTOS
Hello Imb!
II> : IT WON'T WORK!!!!!! Do you know that word "WORK"?
II> Please don't tell the 35+ million internet systems around the globe
II> this as that's what they use - effectively, a simple sequential
II> counter and it most certainly does work. Admittedly, they also aren't
II> limited by character set or length as much but the basic priciple is
II> the same.
The principle is the same, but both technically and realistically it is not
the same. Not at all.
First, You have 8 digits for the fidonet msgid serial number. It is not the
SAME thing at all and if had more room, it alters the entire engineering
and thinking drastically of current dupe checking systems. No, it is not
the same thing.
Second, lets consider that the INTERNET is a somewhat of a controlled
distribution system and dramatically differs from the fidonet distribution
structure.
Your efforts (suggested dupe database structure) is valid and you bring up
many good points. But in practical terms and design, your suggestion would
kill fidonet tossers and not be accepted in the fidonet market place. The
fidonet market would not accept the overhead and tremendous slowdown
required in the specific suggestion you presented.
You must be in academia? <g> Well, I am an engineer. Here are the
following issues you havn't considered:
Lets get out of the way really quick that FTS-9 addressing does not limit
itself to just FIDONET. It can be other addresses since it does not
specifically exclude any other addressing scheme. But lets go ahead and
assume FIDONET addressing only.
1) First, the issue of across the board compliance.
You must understand FTS-9 is not a standard for all email systems that
make up the FIDONET network. In other words, the majority of email input
systems do not support FTS-9 nor generate MSGID. Fidonet is not the
internet where is a common working RFC standard exist for compliance.
This isn't nearly as big design problem because the Fido "Gate Keepers"
the gateways, the mail exporters can assume the responsibility of
generating "the missing" FTS-9 compliant information when they are
moved into the network.
However, this raises the issue of FTS-9 certification which states:
" Finally, a MSGID is generated only at the time of message
creation." (FTS-9, Jim Nutt)
So if a message does not have a MSGID, the mail exporter would
be voilating FTS-9 if it added one. Correct?
In reality, there are many systems which help the FTS-9 process by
compensating for other software lack of support. Hence they generate
a MSGID during exportation or importation into the BBS like it is
done with many off-line mail door systems.
An argument may follow which states no software should add a MSGID if he
didn't create the message in compliance with FTS-9.
Well, this then means that ALL mail which does not have MSGID
information are not candidates for FTS-9 based electronic mail duplicate
checking. They should be "skipped" if no MSGID is found. This will
exclude a very large percentage of mail which is created using QWK based
systems and other software that chose not to support FTS-9.
Practical Engineering Point #1: There is no across the board compliance
and not all input systems support FTS-9.
2) Second, FTS-9 states:
"The serial number may be any eight character hexadecimal number, as
long as it is unique - no two messages from a given system may have
the same serial number within a three years. The manner in which
this serial number is generated is left to the implementor." (FTS-9,
Jim Nutt)
The first sentence is important, it says "no two messages from a given
system" may be duplicated and it also says "within a three years"
period. The 3 years is an important issue in the capacity considertion
which I will bring out later.
But my main point here is that two messages within a given system can
not be the same. The logical extension is that the same two messages on
another system can not be the same just as well.
However, the last sentence says, "the manner in which this serial
numner is generated is left to the implementor".
You can now understand that FTS-9 invalidates the principle of a
standard being implemented across all systems in order to guarantee
uniqueness across systems.
In order to implement any standard MSGID function generator, the
FTS-9 specification must be changed. If it must be changed, then we
should alter all it flaws and not just use an 8 digit number. We
can also address the ADDRESS factor as well.
In reality, we all know FTS-9 is not going to change. We can write a
FSC (proposal), but none of this will guarantee the implementation by
software developers the suggested changes. We can atleast make an
attempt as you did. In the mean time, the "implementor" is allowed to
use whatever technique he feels is appropiate to guarantee no two
messags are the same on his OWN BBS.
Practical Engineering Point #2: We can only guarantee that two messages
are not the same for your OWN system. You can not guarantee this on
others.
3) Third, more direct to your technical suggestion is what are the primary
and/or secondary keys or factors in order to do searching in the table?
This plays a vital role in the logic since the overall hypothesis of
your design is that the message id can't be duplicated ACROSS all
FORUMS, if I understand you right.
So my first point is, why the other factors?
I am going to assume that you do allow for duplicity in the MESSAGE ID
and that the other factors are used in the consideration, after all, why
have them in the structure then?
Lets assume the current packet reading engine. Among the first things
extracted from the message are (assuning MSGID is present):
ECHO TAG
ADDRESS and SERIAL NO
Would you first look up ECHO TAG? or the SERIAL NO? or the Address?
Where would you get the DOMAIN from the message? Must FTS-9 change the
address have the domain in it? Of course, you also "LOAD the entire
message" and extract ORIGIN LINE information, if it is there.
Of course, the above doesn't take into account the realistic tossing
approach of checking for PATH address as well to check for dupes as
well. But lets just go with your dupe checking.
You have DATABASE lookup #1 to find a matching ECHO TAG. You have
database lookup #2 to find a matching DOMAIN. Do you know how many
Fidonet echos there are when you combine fidonet and all the different
private networks out there? How about NewsGroups gating info fidonet? I
have some customers with atleast 20,000 or more "newgroups echos" mixed
in with fidonet. Do you have any feel for the degradation of tossing
performance using the searching logic you suggest?
Also, since the engine is already reading "text", what value can be
placed on the tossing design to bring into consideration the parameters?
TO, FROM, SUBJECT, DATE for dupe checking? Why should they be ignored at
these stage? Is there any value?
Practical Engineering Point #3: Do not neglect performance issues in
your design consideration.
Btw, Are you familiar with the "Query Theory?"
4) What is the table capacity on a TOTAL BASIS and on PER ECHO BASIC?
The 3 year factor of the FTS-9 specification now comes into place
now.
Let me just ask the following. How many messages on a per conference
basis do most sysops allow in their BBS? Just an average? Is this
important? 100, 200, 300? 400? 500? Unlimited?
Lets just state for the sake of simplicity that the magic number is 500
messages per conference, after all, MOST BBS software place a default
value by default. 500 is high but this is just an example.
Lets also use an average of 100 echos that he carries. Lets also assume
that on average he tosses 100 messages per day per echo, but lets assume
that each echo has reached capacity.
This means that, the BBS stores
100 echos
x 500 maximum messages
======
50,000 messages
But for FTS-9 compliance, he must record,
100 echos
x 100 messages per echo per day
x 365 days in year
x 3 years
------
10950000 entries in your database
for a theoritical dupe file size of:
10950000
x 20 (size of entry)
---------
219 megs
Do you really believe a sysop will support a 219 meg dupe file for dupe
checking when he only carries 50,000 messages most? Sure, the file is
a growth file, but you need to take into account maximum capacity to
comply with FTS-9. Can you imagine the large hub carries just 800
fidonet echos and allows 300 messages per conference?
Practical Engineering Point #4: Do not assume the system will support
or the sysop "honor" the theoritical load requirements for FTS-9 dupe
checking requirements.
Lets bring out reality:
Many mail tossers will reject mail if its older than 30-45 days. Thats
definable by the sysop but alot are programmed to allow for this. This
nullifies FTS-9 3 year concept. So a MAIL TOSSER/EXPORTER who claims to be
compliant with FTS-9 is breaking the specs if he allows for AGE rejections
less than 3 years.
In any case, dupe checking starts with the MESSAGE ADDRESS, whether its
taken from the MSGID, ORIGIN LINE or whatever, if the MESSAGE ADDRESS
matches the home address, it can't be considered anything but a DUPE. Its
dumped.
The PATH statement is next. If your HOME ADDRESS is in there, its a DUPE,
period.
In my implementation, to compensate for systems that don't support MSGID, A
CRC is generated based on the concatentation of the following string:
TO+
FROM+
SUBJECT
ASCIIZ DATE
ECHO TAG
The result is then checked in my DUPE table. If a match is found, it
is a DUPE. If not, it is inserted.
One may say, what about the MSGID serial # that is already there?
What I do is check for the serial # and compare it in my dupe table. But
note this voilates my "Practical Engineering point #2" which says I am NOT
aware of the other guys serial number generation formula.
So by default, the check is enabled, but if there are complaints of
erroneous dupe dumps, it can be turned off.
One may also argue, "But Hector, thats the whole point of this thread, lets
develop that formula". I agree, but refer to my "Practical Engineer Point
#1". Unless there is across board compliance, you can never rely on the
"other guys" formula. And even if there was a decent PROPOSAL made, you
can not enforce it. So you will always be stuck with generating your own
number for dupe checking.
In any case, the odds are EXTREMELY good that the dupe checking would not
reach the DUPE TABLE check if the message is indeed a dupe.
Also, keep in mind that my UPLINK is also making sure "all messages are
unique to his system" so when you consider the "total filtering" process
performed across the system, dupe checking is pretty much reliable. You
only really see a dupe when a SYSTEM rescans a message base. The fact that
I am using an existing SERIAL # in the message, helps in this situation.
Now, I made a decent living interfacing with academia, researchers, etc, to
extract theory and move it into practical operations. So let me take your
theory, and make it useful. <g>
A design consideration would be to remove the ECHO word index and base the
dupe file storage on a per ECHO basis. This will give the sysop the added
benefit of being able "purge" unwanted files when a echo is unlinked from
his setup. Sysops are in constant echo turnovers.
I would also ignore the domain but since the domain database would be
relatively small, it wouldn't hurt. I would personally explore the use of
a 32 bit CRC of the address@domain.
The result would be:
*next 4 bytes
msgid 4 bytes
crcaddr 4 bytes
This allows you better manage the size factor based on usage. But in the
end, the maximum capacity will still be reached. The Max total size be
reduced to 131 megs. (Would you still support this size?)
But since the market reality would also tell you to ignore the impractical
3 year period and also base it on the 30-45 days based on "SYSOP DEMAND" or
use a TOTAL amount of dupes to be checked, your total would be drastically
reduced. Assume 16K dupe entries per ECHO
100 echos
x 16K msg dupe capacity
x 12 size of entry
-----
or 19.2 megs. More reasonable huh?
Now also remove your *next pointer and get it down to 8 bytes or 12.8 megs.
But more signficantly, you can now LOAD the 16K*8 or 128K "echo dupe" file
into memory and do very fast checking.
If you go by days, then can do even better:
100 echos
100 messages per echo per day
x 30 days
x 20 size of entry
-----
or 6000000 or 6 megs
Most setups keep a MAX per ECHO and also use the 30-45 AGE factor.
These are, as much as we don't want to hear it, the practical side of the
issue regardless of what is in the SERIAL NO field.
I won't debate with you that the CRC is not the "best" idea for calculating
a "serial number" for FTS-9, but in practical terms it is and it works. How
else would you take into account non FTS-9 supported mail?
I challenge you to use a CRC of "TO+From+Subject+AsciizDate+EchoTag" and
run it across ALL your current messages on your BBS right now, and tell me
how much of it is duplicated. If you don't find one, run it over a longer
period until you do find one.
But remember, EVEN if you do find one, it isn't a DUPE unless the ADDRESS
comparisions are made in the MSGID and PATH. In other words, if you find a
message in echo X and message in echo Z with the same CRC, you must also
systematic do the preliminaries comparisions before you conclude erroneous
duplicity.
Understand that you are not dealing with the Internet, but with fidonet,
there is no such thing of having a message with Multiple "echo tags" in it
and in my opinion, you should only be concern with the message of where it
is actually placed in FidoNet, in this case, - an echo. The fact, that a
SERIAL number matches one in another echo, shouldn't mean its a dupe.
I have said alot in the message, and whether anyone agrees or not, it is a
practical formula for success given the realities. It has for me. This
brings it down to what is the worth of FTS-9? I say it is WORTH alot and I
say the addressing is of extreme important "value" than the serial number
in the overall scheme of things. I hope more people will see that.
Finally, let me conclude by saying, we all know how mathmethics play a key
role in our theories and hence lives, but even Eienstien's E=MC2 is a
practical reduction of a more complex formula of theory, physic and quantum
mechanics.
Computer Scientist don't learn this stuff, do they? <g>.
See ya
--- Platinum Xpress/386/Wildcat! v1.2j
* Origin: Xpress Tech Support BBS * 305-248-7815 (1:135/382)