I need help, creating an open taxonomy for the Internet

[u/ki4jgt]

This was deleted from r/learnprogramming, so I'm reposting here.

---

I've been thinking about how cool it would be if we could categorize internet domains in the same way the Dewey Decimal System categorizes books. Such a system would allow independent search engines and directories to compete with Google again.

We live in a time where you can spin up as many VPSs as you need. Where data can be distributed in all sorts of ways, among those machines. And on the cusp of the fastest processing power in history (quantum).

I'm honestly looking for a discussion on extending this:

https://github.com/ki4jgt/Open-Catalog-System/blob/main/OCS.json

It's a taxonomy for classifying and grouping domains of similar topics and interests (Facebook, Google, programming, YouTube, etc). As I cannot possibly think of all the subjects contained on the web, I would love some open discussion, here on Reddit -- if that's OK, on how you guys think this should be structured, what subjects you think are missing, and your general feedback on what would make it better.

For those of you who don't know, taxonomies are how we used to find information before Google. To find an atlas at your local library, you'd go to the Information desk, then the reference area (things you can look at in the library but can't check out), then the particular reference type you were looking for.

So, INF.REF.DIC would give you a bunch of dictionaries. Since you're looking for information (a definition), and need a reference, I thought it only fitting to keep the convention. In web terms, Webster, Cambridge, and Urban Dictionary would be listed here.

RFCs are listed under INF.COM.RFC because RFCs extend beyond programming, and are used in a wide range of communication and data formats.

Feel free to share any suggestions you have in the comments below. Thanks. I am desperately seeking to expand this. The goal here is inductive, not deductive. We want to group domains together, with useful categories, instead of splitting them apart.

[u/Original-Active-6982]

I hope you are looking at using tags/attributes/properties to identify web site domains, not a strictly "one code per site." A site such as ibm.com can have multiple attributes such as "Hardware manufacturer", "Software As A Service", "Information Technology Research", "For Profit Corporation", ... Purely hierarchical organization (such as the DDS) doesn't give you flexibility for cross-linking and updating classifications.

[u/ki4jgt → u/Original-Active-6982]

If IBM.com were a book, where would it sit in a library? Probably in computer science, since that's their area of expertise.

[u/goldman60]

If you want a dewey decimal system for the internet why reinvent the wheel, just use the Dewey Decimal System. You can add a new Class for internet-isms like generic social media and search engines, it already covers basically everything else.

[u/ki4jgt → u/goldman60]

Because the OCLC owns the DDS (WorldCat), and is lawsuit happy. They even sued someone for using the DDS in a themed party.

Edit: That means an open system needs to be created. Lest we face the wrath of IP lawyers. Libraries aren't even allowed to use it without paying a licensing fee first.

Their reach isn't limited to the US. They have bodies all over the world, on every continent. And collect usage fees from all local libraries using WorldCat.

[u/ki4jgt → u/goldman60]

Furthermore, this is required, because I just Googled "OCLC lawsuit." The 12 pages it showed me didn't mention any lawsuits prior to 2022. If we're going to have an open web, we've got to transition away from Google. It's hiding information.

[u/nft420nft420 → u/goldman60]

The Dewey decimal system is not a perfect model and has a lot of choices made that are no longer the ideal way to categorize modern information. Many western idealogies have large sections while more international topics are often delegated to tiny sections. It's a bit confusing and I haven't read up in a while but it is time for an open source replacement to the whole Dewey decimal system.

[u/danja]

You might want to take a look at the Semantic Web/Linked Data way of representing taxonomies. Check for example : https://lod-cloud.net/

https://schema.org/docs/schemas.html

[u/danja → u/danja]

I just remembered there was DMOZ, which was a big catalogue of web sites. Dumps are still available online. You may be able to extract the taxonomy from that, although I'm sure there's a more recent one, maybe expressed in SKOS (another RDF vocab)

[u/lift_spin_d]

https://www.doi.org/

[u/dubsnipe]

The problem isn't building the taxonomy, since you can use the Dewey Decimal System. The big problem is conforming a web page to the taxonomy. Search engines originally focused on keyword appearances which is still a pretty useful approximation.

Having said this, I approached the problem by using transformer models: running a website's text through a local model (not an LLM chat, just the model) and compare the resulting calculations against a previously-defined vocabulary. It's not an original solution so you can find examples of this around.