if you've been reading, you've seen my posts about cleaning up the R18 nodelist
segment... i wanted to share my script, such as it is, if others of you are interested in it...

the real meat is a long one-liner that looks only at the INA flags and eventually spits out any NXDOMAIN results it finds... if there's no output, everything's good... right now we have to manually match the output with the nodelist entry but that doesn't hurt too awful bad...

----->8 snip 8<-----
egrep -e "^(Zone|Region|Host|Pvt)?,.*,INA:.*,?" BLAHBLAH.SEG | egrep -o -e "INA:.*,?" | cut -d ',' -f 1 | cut -d ':' -f 2 | sort | uniq | nslookup | egrep
-e "[Cc]an't find" | egrep -v -e "\.in-addr\.arpa"
----->8 snip 8<-----


yeah, it is really long which is why i saved it to a file and for now, simply hard coded each of the net segs we're checking... just make sure you change BLAHBLAH.SEG to the file name of the segment you are testing...

so here's the breakdown of what we're doing in each step...

1. grep all lines that start with Zone, Region, Host, and Pvt or
an empty first field and also contain an INA flag. this allows
for skipping commented lines as you work on the list. we also
skip HOLD and DOWN nodes for now.

2. grep and output /only/ the INA flag and contents

3. using commas as the delimeter, grab the first field
this removes any possible trailing commas

4. using colons as the delimeter, grab the second field
this should give us only the listed domain and
leave off misplaced ports that shouldn't be in INA.

5. sort the list alphabetically

6. filter out duplicates

7. nslookup each of the domains found

8. only output "can't find" entries

9, don't output "can't find in-addr.arpa" entries from IP numbers


to explain #8 and #9...

#8 some lookups return "No answer"... i'm not sure what that's all about but traceroute won't/can't work on that domain so we'll take that "No answer" to be
the same as NXDOMAIN...

#9 we don't worry about entries that have IP numbers instead of domains... they may or may not return an in-addr.arpa entry... it isn't bad if they do not...


so here's a sample of the output from my actual script...

==== Begin "checkFTNdomains.log" ====
Checking n123.seg...
** server can't find aftershock.servebeer.com: NXDOMAIN
** server can't find armed.gamilitia.us: NXDOMAIN
** server can't find bbs.furmen.org: NXDOMAIN
*** Can't find bbs.techinvasion.net: No answer
** server can't find english.office-on-the.net: NXDOMAIN
** server can't find flashback-bbs.zapto.org: NXDOMAIN
** server can't find irex.hdcbbs.com: NXDOMAIN
** server can't find kia.zapto.org: NXDOMAIN
** server can't find montereybbs.ath.cx: NXDOMAIN
** server can't find orbitbbs.com: NXDOMAIN
** server can't find qlbbs.mooo.com: NXDOMAIN
** server can't find tffbbs.com: NXDOMAIN
====================

Checking net116.seg...
** server can't find am1620.servebbs.net: NXDOMAIN
** server can't find kj4tkv.org: NXDOMAIN
** server can't find magnum.synchro.net: NXDOMAIN
** server can't find opendoor.synchro.net: NXDOMAIN
====================

Checking net135.seg...
** server can't find bills.synchro.net: NXDOMAIN
====================

Checking net3634.seg...
====================

==== End "checkFTNdomains.log" ====


as i work on each segment, i comment out each of the lines returning NXDOMAIN... just like the nodelist, i comment the lines with a ';'... i also add ",NXDOMAIN" to the end of the flags. that lets me know why it is commented out when i look at the file later. as i eliminate batches of NXDOMAIN entries, i can run the check against my file again and see if i missed anything... that's why i have the weird first grep... the main thing was to skip commented lines but i realised that some desired ones would also be skipped so we came up
with that initial pattern...

FWIW: i'm considering that we could publish these entries in the nodelist as Down and with the trailing ,NXDOMAIN flag... since NXDOMAIN is currently an unofficial flag, putting it with User flags would be better...

eg: Down,sys,loc,op,...,INA:foo,U,NXDOMAIN

so why add NXDOMAIN as a flag on these invalid domain entries? personally speaking, it allows me to grep all the lines that contain NXDOMAIN so they can be used in posts asking if anyone has contact with the node... it is easier than trying to manually pick them out... especially if there's more than two or
three to work with...

why publish them in the nodelist instead of just removing them? so that others will know they are marked Down because of the NXDOMAIN... not sure if Down or Hold would be the best... both mean you cannot contact the node and NXDOMAIN reinforces that and tells why... as usual, Hold and Down should be removed from
the nodelist soonish so that aspect won't change... the idea is that listing the NXDOMAIN flag can only be done with the Down or Hold first field entry... the idea of doing it this way is to provide the chance for the node to have their entry fixed... being marked Hold or Down should catch their attention when the mail flow stops or their mailer complains about their entry in the nodelist... yeah, if they are even paying attention to their system...

so anyway... there it is... come at me, ya'll! :biggrin:

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it wrong...
... We never live; we are always in the expectation of living. - Voltaire
---
* Origin: (1:3634/12.73)

Good idea Mark, I'll add NXDOMAIN to my flags list.

Script looks good as well :)

if you've been reading, you've seen my posts about cleaning up the R18 nodelis
segment... i wanted to share my script, such as it is, if others of you are interested in it...

the real meat is a long one-liner that looks only at the INA flags and eventually spits out any NXDOMAIN results it finds... if there's no output, everything's good... right now we have to manually match the output with the nodelist entry but that doesn't hurt too awful bad...

----->8 snip 8<-----
egrep -e "^(Zone|Region|Host|Pvt)?,.*,INA:.*,?" BLAHBLAH.SEG | egrep -o -e
"INA:.*,?" | cut -d ',' -f 1 | cut -d ':' -f 2 | sort | uniq | nslookup | egre
-e "[Cc]an't find" | egrep -v -e "\.in-addr\.arpa"
----->8 snip 8<-----

yeah, it is really long which is why i saved it to a file and for now, simply hard coded each of the net segs we're checking... just make sure you change BLAHBLAH.SEG to the file name of the segment you are testing...

so here's the breakdown of what we're doing in each step...

1. grep all lines that start with Zone, Region, Host, and Pvt or
an empty first field and also contain an INA flag. this allows
for skipping commented lines as you work on the list. we also
skip HOLD and DOWN nodes for now.

2. grep and output /only/ the INA flag and contents

3. using commas as the delimeter, grab the first field
this removes any possible trailing commas

4. using colons as the delimeter, grab the second field
this should give us only the listed domain and
leave off misplaced ports that shouldn't be in INA.

5. sort the list alphabetically

6. filter out duplicates

7. nslookup each of the domains found

8. only output "can't find" entries

9, don't output "can't find in-addr.arpa" entries from IP numbers

to explain #8 and #9...

#8 some lookups return "No answer"... i'm not sure what that's all about but
traceroute won't/can't work on that domain so we'll take that "No answer" to b
the same as NXDOMAIN...

#9 we don't worry about entries that have IP numbers instead of domains... they may or may not return an in-addr.arpa entry... it isn't bad if they do not...

so here's a sample of the output from my actual script...

==== Begin "checkFTNdomains.log" ====
Checking n123.seg...
** server can't find aftershock.servebeer.com: NXDOMAIN
** server can't find armed.gamilitia.us: NXDOMAIN
** server can't find bbs.furmen.org: NXDOMAIN
*** Can't find bbs.techinvasion.net: No answer
** server can't find english.office-on-the.net: NXDOMAIN
** server can't find flashback-bbs.zapto.org: NXDOMAIN
** server can't find irex.hdcbbs.com: NXDOMAIN
** server can't find kia.zapto.org: NXDOMAIN
** server can't find montereybbs.ath.cx: NXDOMAIN
** server can't find orbitbbs.com: NXDOMAIN
** server can't find qlbbs.mooo.com: NXDOMAIN
** server can't find tffbbs.com: NXDOMAIN
====================

Checking net116.seg...
** server can't find am1620.servebbs.net: NXDOMAIN
** server can't find kj4tkv.org: NXDOMAIN
** server can't find magnum.synchro.net: NXDOMAIN
** server can't find opendoor.synchro.net: NXDOMAIN
====================

Checking net135.seg...
** server can't find bills.synchro.net: NXDOMAIN
====================

Checking net3634.seg...
====================

==== End "checkFTNdomains.log" ====

as i work on each segment, i comment out each of the lines returning NXDOMAIN... just like the nodelist, i comment the lines with a ';'... i also
add ",NXDOMAIN" to the end of the flags. that lets me know why it is commented >out when i look at the file later. as i eliminate batches of NXDOMAIN entries,
i can run the check against my file again and see if i missed anything...
that's why i have the weird first grep... the main thing was to skip commented >lines but i realised that some desired ones would also be skipped so we came u
with that initial pattern...

FWIW: i'm considering that we could publish these entries in the nodelist as Down and with the trailing ,NXDOMAIN flag... since NXDOMAIN is currently an unofficial flag, putting it with User flags would be better...

eg: Down,sys,loc,op,...,INA:foo,U,NXDOMAIN

so why add NXDOMAIN as a flag on these invalid domain entries? personally
speaking, it allows me to grep all the lines that contain NXDOMAIN so they can
be used in posts asking if anyone has contact with the node... it is easier
than trying to manually pick them out... especially if there's more than two o
three to work with...

why publish them in the nodelist instead of just removing them? so that others
will know they are marked Down because of the NXDOMAIN... not sure if Down or Hold would be the best... both mean you cannot contact the node and NXDOMAIN
reinforces that and tells why... as usual, Hold and Down should be removed fro
the nodelist soonish so that aspect won't change... the idea is that listing the NXDOMAIN flag can only be done with the Down or Hold first field entry... the idea of doing it this way is to provide the chance for the node to have their entry fixed... being marked Hold or Down should catch their attention when the mail flow stops or their mailer complains about their entry in the nodelist... yeah, if they are even paying attention to their system...

so anyway... there it is... come at me, ya'll! :biggrin:

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it
wrong...
... We never live; we are always in the expectation of living. - Voltaire

--- BBBS/Li6 v4.10 Toy-3
* Origin: Prism bbs (1:261/38)

On 2017 Nov 28 14:40:30, you wrote to me:

Good idea Mark, I'll add NXDOMAIN to my flags list.

i'll be sending them as U flags so that may not be necessary... the same with NOANSWER that i'm also using for nodes with valid domains that don't answer when polled with my mailer(s)...

Script looks good as well :)

thanks... it took a bit to work it all out... you should see what it does with the full nodelist :lol:

it finds three entries that have a badly formatted INA flag... one of those has
an email address in it!

Host,305,Southern_New_Mexico,Las_Cruces_NM,Tj_Barlow,-Unpublished-,9600,CM,IBN,INA:9600,ICM,IFC,INA:cscnet1.net,IBN:24554,ITN:60177
Host,317,Northern_New_Mexico,Albuquerque_NM,Tj_Barlow,-Unpublished-,9600,CM,IBN,INA:9600,ICM,IFC,INA:cscnet1.net,IBN:24554,ITN:60177

net 1:218 ,504,Stockdale_Computer,Bakersfield_CA,Lance_Cooper,-Unpublished-,300,CM,INA:transx@stockdalecomputer.com,ITX


they should look like this...

Host,305,Southern_New_Mexico,Las_Cruces_NM,Tj_Barlow,-Unpublished-,9600,ICM,IFC,INA:cscnet1.net,IBN:24554,ITN:60177
Host,317,Northern_New_Mexico,Albuquerque_NM,Tj_Barlow,-Unpublished-,9600,ICM,IFC,INA:cscnet1.net,IBN:24554,ITN:60177

net 1:218 ,504,Stockdale_Computer,Bakersfield_CA,Lance_Cooper,-Unpublished-,300,CM,ITX:transx@stockdalecomputer.com


on the first two, we removed "CM,IBN,INA:9600," because the rest of the entry covers it...
on the second one, we changed the INA flag to ITX and removed the empty ITX flag from the end...

my apologies if i'm stepping on anyone's toes... these are simple and easy changes... won't take more than 30 seconds to do them...


here's the output of the full nodelist check...

Checking nodelist.332...
** server can't find 12977.korpus1.ru: NXDOMAIN
** server can't find 2038.no-ip.org: NXDOMAIN
** server can't find 460n57.flnet.org: NXDOMAIN
** server can't find 50201907.ru: SERVFAIL
*** Can't find 9600: No answer <-- from the two bad INA:9600 flags above
** server can't find acme-anvil.darktech.org: NXDOMAIN
** server can't find aftershock.servebeer.com: NXDOMAIN
** server can't find alexkim.dyndns.org: NXDOMAIN
** server can't find am1620.servebbs.net: NXDOMAIN
** server can't find amachat.org: NXDOMAIN
** server can't find andreyf1.homelinux.org: NXDOMAIN
** server can't find armed.gamilitia.us: NXDOMAIN
** server can't find asgard-bbs.net: NXDOMAIN
** server can't find avkorop.dyndns.org: NXDOMAIN
** server can't find backtotheroots.it-awareness.ch: SERVFAIL
** server can't find bbs2.vampirebbs.org: NXDOMAIN
** server can't find bbs.altbbs.net: NXDOMAIN
** server can't find bbs.cyberchatnet.com: NXDOMAIN
** server can't find bbs.dbackbbs.com: NXDOMAIN
*** Can't find bbs.domotron.net: No answer
** server can't find bbs.furmen.org: NXDOMAIN
** server can't find bbs.openwifi-br.net: NXDOMAIN
** server can't find bbs.realms-online.org: NXDOMAIN
** server can't find bbs.spazoo.com: NXDOMAIN
*** Can't find bbs.techinvasion.net: No answer
** server can't find bbs.twixted.net: NXDOMAIN
** server can't find bills.synchro.net: NXDOMAIN
** server can't find blues.zapto.org: NXDOMAIN
** server can't find ddibbsystem.dk: SERVFAIL
** server can't find dk24.no-ip.org: NXDOMAIN
** server can't find drac.synchro.net: NXDOMAIN
** server can't find english.office-on-the.net: NXDOMAIN
** server can't find f120n5015z2.mooo.com: NXDOMAIN
** server can't find f1313n467.no-ip.org: NXDOMAIN
** server can't find f1.n4500.z2.fidonet.by: SERVFAIL
** server can't find f47.n461.z2.fidonet.net: NXDOMAIN
** server can't find f814.no-ip.org: NXDOMAIN
** server can't find fidocl.ddns.net: NXDOMAIN
** server can't find fido.gov.ivanovo.ru: NXDOMAIN
** server can't find fido.kapitan0v.com: NXDOMAIN
** server can't find fidonet.dyndns.info: NXDOMAIN
** server can't find fido.rostov.ru: NXDOMAIN
** server can't find fido.rostov.ru: NXDOMAIN
** server can't find fido.skuonline.ru: NXDOMAIN
** server can't find fido.wom.ru: NXDOMAIN
** server can't find flashback-bbs.zapto.org: NXDOMAIN
** server can't find FMLYNET.DYNDNS.ORG: NXDOMAIN
** server can't find irex.hdcbbs.com: NXDOMAIN
** server can't find juzy.no-ip.org: NXDOMAIN
** server can't find kellys.synchro.net: NXDOMAIN
** server can't find kia.zapto.org: NXDOMAIN
** server can't find kj4tkv.org: NXDOMAIN
** server can't find klonezonebbs.zapto.org: NXDOMAIN
*** Can't find landoftheloungelizards.com: No answer
** server can't find lostcryp.synchro.net: NXDOMAIN
** server can't find magnum.synchro.net: NXDOMAIN
** server can't find montereybbs.ath.cx: NXDOMAIN
*** Can't find msmacbbs.com: No answer
** server can't find novabbs.org: NXDOMAIN
** server can't find ns.sintez.nnov.ru: SERVFAIL
** server can't find oceanbbs.dyndns.org: NXDOMAIN
** server can't find opendoor.synchro.net: NXDOMAIN
** server can't find orbitbbs.com: NXDOMAIN
** server can't find qlbbs.mooo.com: NXDOMAIN
** server can't find rosastation.ddns.net: NXDOMAIN
** server can't find sib.r-style.ru: SERVFAIL
** server can't find sodin.homeip.net: NXDOMAIN
** server can't find solovyevka.ddns.net: NXDOMAIN
** server can't find talk.vrn.ru: SERVFAIL
** server can't find tbbs.homeip.net: NXDOMAIN
** server can't find tdmonline.org: NXDOMAIN
** server can't find tffbbs.com: NXDOMAIN
** server can't find theechozone.com: NXDOMAIN
** server can't find therealmbbs.com: NXDOMAIN
** server can't find thevoid.servebbs.org: NXDOMAIN
** server can't find torchwood-3.dyndns.org: NXDOMAIN
** server can't find transx@stockdalecomputer.com: NXDOMAIN <-- 3rd above
** server can't find ua2fga.homelinux.com: NXDOMAIN
** server can't find warriorbbs.dyndns.org: NXDOMAIN
** server can't find we6jbo.synchro.net: NXDOMAIN
** server can't find wizardos.synchonet.net: NXDOMAIN
** server can't find wolfpacbbs.no-ip.ca: NXDOMAIN
** server can't find x-file.bbsindex.com: NXDOMAIN
** server can't find yurykalinin.servemp3.com: NXDOMAIN
====================


so anyway, there it is :)

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it wrong...
... You're never too old to laugh at farts.
---
* Origin: (1:3634/12.73)

Hello mark,

Looks good mark. I have also recently felt it was time to gather up some initiative to verify the actual status of nodes in the region that I manage.

Based upon my own checking which included mailer polling, domain resolution, relevant port access status, a ping of the FQDN, DNS check of the FQDN, and telnet connect-ability.

I can verify that the node that the doamain drac.synchro.net is related to is not reachable.

** server can't find drac.synchro.net: NXDOMAIN

Jeff

--- BBBS/Li6 v4.10 Toy-3
* Origin: Region 14 Coordinator - bbs.ouijabrd,net (1:282/1031)

On 2017 Nov 28 18:41:04, you wrote to me:

Looks good mark. I have also recently felt it was time to gather up
some initiative to verify the actual status of nodes in the region
that I manage.

i hope that oneliner helps... it only provides one part of the overall check system, though... i've been trying to get hold of operators in my area and get some info from them but i can't really do much unless they are connected to my system... the main problem is not having some sort of infallible address to reach them at... a huge part is lack of transferrence where records are shared between *Cs so that new replacements can be given access to such data and the nodelist doesn't become fragmented by nodes just falling off the list when systems drop out without warning...

Based upon my own checking which included mailer polling, domain resolution, relevant port access status, a ping of the FQDN, DNS check
of the FQDN, and telnet connect-ability.

I can verify that the node that the doamain drac.synchro.net is related to is not reachable.

** server can't find drac.synchro.net: NXDOMAIN

i would mark that node as "Down" with a trailing flag set of ",U,NXDOMAIN" or ",U,NOANSWER" and given them the customary time period of two+ weeks to get hold of you for an update... the downside is to publish this request information somewhere that they or their contacts might see it and pass on the word to get hold of you... i've been dancing that dance the last two or three days but it is basically now closing since i've had no other responses with positive information...

yes, this can be rather painful... especially when you are as quiet as i am and
don't roam and range from your own system... i've had to start actually trying to connect to other systems that i've never heard from before! :lol: but seriously, that's one otoh that may be faced...

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it wrong...
... I've taken an oath of celibacy, so don't screw with me.
---
* Origin: (1:3634/12.73)