And yet at the same time, we’ve also proven to ourselves that Many To Many Is Misunderstood. We’ve shown that the influence of our physical world, which is hierarchical, often spurs us to try to represent our thoughts in hierarchical structures. We are often dissatisfied with the results, and yet we can’t easily move beyond the frustration.

What can we conclude of this?

It’s definitely true that hierarchical frustrations are commonplace. One only needs to visit an unfamiliar grocery store and look for vegetable oil to find an example. Is vegetable oil a cooking oil, to be stored near the flour and sugar? Or is it a salad oil, to be stored near the vinegar? Should vegetable oil be stored near olive oil, or in a separate aisle? When you are rushing to the grocery store at the last minute to prepare for impending dinner visitors, these can be frustrating questions. And yet, there is not a way around them, as a grocery store aisle is rooted in the physical world, and thus hierarchical. (A grocery store could split their vegetable oil batches into two locations, but this would lead to the same maintenance challenges that we saw with the library card catalog in Many To Many Is Not Physical.)

To break this down – when visiting our hypothetical grocery store, there is an expectation that vegetable oil should be where we expect it. If we need it for a baking recipe, it should be in the baking aisle. If we need it for salad dressing, it should be alongside the vinegar. These competing expectations don’t make sense in a hierarchical sense, but they exist anyway. And when our expectations are not met, we get frustrated.

It is commonplace to have non-hierarchical thoughts in our mind and expect them to be reflected externally. It is also true that these expectations cannot always be met – grocery stores cannot, in this example, be expected to split their stocks to meet the contextual expectation of every customer.  But the reason they cannot be expected to do this is because they are rooted in the physical world, which is by definition hierarchical.  We accept this limitation, we accept the frustration, and learn to tolerate it.

But what of the non-physical world?

Let’s think back to our card catalog example. A card catalog seemingly represents a many-to-many relationship between library books, and the ways in which a customer might search for a book. And yet, we’ve shown that this is only an illusion. The card catalog is not actually one representation of a book stored in multiple places – it is multiple copies of the book’s representation, stored in multiple places in the catalog. And since there are multiple copies, this invites all the struggles and errors that come along with keeping the copies in synchronization.

But let’s take a closer look at what is actually required when keeping these representations in sync:

• As often as is reasonable, we must cycle through all the cards in the catalog, and make sure the books still exist and are in the right place.
• As often as possible, we must cycle through all the books in the library, and make sure they all have properly categorized cards.

These are both examples of highly repetitive, high-maintenance, easily defined tasks. In other words, perfect for software.

In fact, these routines are exactly how many-to-many relationships are represented in computer software. Software serves as the role of our librarians, instantly and continually doing all the checking to keep a many-to-many representation perfectly in sync. It’s as if you look up a library book in a card catalog by subject, but with the book then instantly appearing in your hand.

And we are used to dealing with many-to-many relationships all the time in the non-physical world. When browsing for a product on an ecommerce website, you can find the product you are interested in whether you are searching by price, by color, or by size. When searching for a book on amazon.com, you can easily find it by subject, by author, or by title. Online, we experience far less of the frustration we might feel in the grocery store salad dressing aisle when we realize that the store believes we’re actually looking for cooking oil.

And yet – even in the non-physical world, the hierarchical model creeps in where it shouldn’t. A perfect example is todo list software. Let’s look at this screenshot of this highly-used, but unnamed piece of productivity software:

There’s a lot that is good about this software – you can subcategorize your items.  You can even specify some items as sequential, and some as available to work on in parallel.  But there is clearly something missing.  You can start cooking the eggs before emptying the dishwasher, or buttering the toast.  You have to empty the dishwasher before putting the cereal in the bowl, but you can put the cereal in the bowl before cooking the eggs.  Would it not be clearer to represent the ordering of the tasks in the following manner?

It turns out that these more accurate dependency relationships are impossible to represent in a hierarchical manner.  In actuality, many of our plans and todo lists actually have many-to-many dependencies, and we struggle with how to represent these in our various checklist apps.  We accomplish this by compromising and making mental notes to ourselves, and we might not even be bothered by it except for once in a while.  This mimics our tolerance of hierarchies in the physical world.  Having learned to look past our frustration in the physical world, we look past it in the non-physical world as well, even though we don’t have to.

So even when we have the theoretical ability to use many-to-many to represent our thoughts, we do not, even though Hierarchies Are Overrated.  Again, what is the conclusion we can draw here?

We’ve shown that we have the ability and theoretical opportunity to use Many To Many.  We’ve shown that we often misunderstand it, and that we do not use it to the extent that we could.  It appears that we simply have not learned to use it to its capabilities.

It is also clear we have become accustomed to tolerating a frustration against hierarchical models, even when we don’t have to.  A solution is needed for this as well.

Finally, we can introduce our first value statement that will support many of our future conclusions.  If something has utility, it should be utilized.  This is at best a philosophical statement, but it is one we will adopt here, and it will reappear several times in the future.  And so we reason: if we have not learned to use Many To Many to its capabilities, and if Many To Many should be utilized, and if we are accustomed to putting up with a problem that Many To Many can solve, then the proper conclusion is that Many To Many Needs Advocacy.

Back in the olden days, every library had something called a card catalog.  You would go to the large wooden cabinet, pull out a thin wooden drawer, and look up a book.  A card inside would give you a location in the library, and then you’d go to that section of the library to find your book.

The shelves themselves were purely hierarchical.  The Dewey Decimal System laid books out generally by subject, and then alphabetically by author.  Each book was in only one area, each subject was in only one section of the library.

The problem with that hierarchy should be familiar by now – without the card catalog, you could only find a book if you knew what subject it was under, and what the author’s name was.  But with the card catalog, you could look up a book by Title, by Author, by Publisher, and even by any of several Subjects that the book might have been listed under.

It can be argued that the card catalog created a true Many To Many system, constructed in the physical world.  Each book could be looked up by multiple Authors, multiple (alternate) Titles, multiple Subjects, and they would all lead to the same book, at its Dewey Decimal location on the shelves.

But is it really Many To Many, or just an illusion?  The cards in a card catalog refer to representations of books – simple sets of information that include all of a book’s alternate titles, authors, publishers, and subjects.  And here is where the problem lies.  Ideally, each library book should have only one representation that can be looked up in multiple ways.  Instead, the card catalog’s cards are each copies of that representation.  If a book has twelve cards in the card catalog, those are twelve separate copies of the book’s representation, all displayed in slightly different ways.

And that’s where the synchronization problems can happen.  What if a book’s information needs to be corrected, or listed under a new subject?  What if a book is damaged or lost, or needs to be removed?  You can’t just edit or remove one card and expect all the other cards would automatically be adjusted.  You have to hunt through to find every single card.  For a large catalog, the librarians might even need a reverse catalog, where looking up a book would tell them the location of every related card  – and even the reverse catalog could go out of date.  In fact, librarians would regularly schedule card catalog cleanup periods, just to try and keep their heads above water.

So in fact, we’re back to a collection of hierarchical systems.  A book’s representation can’t actually be looked up in multiple ways because each representation (each card) is in only one drawer or location.  But since we’re trying to force a synchronized relationship between items in separate hierarchical locations, the maintenance it requires is costly and time-consuming.  The hierarchical system itself doesn’t capture this information that people impose onto it, because people are thinking about the system in non-hierarchical ways.  Therefore, the human thoughts aren’t completely externalized, the thoughts can be forgotten, and the outside structure can slowly become unsynchronized and corrupt.

It turns out that Many To Many can’t really be represented in a purely physical system.  It can be simulated, but only by introducing copies into hierarchical systems.  And this is because our physical world is purely hierarchical.  And because of all that, simulating a Many To Many system in the physical world is time-consuming, manual, tedious, and fraught with error.

In Many To Many Has Utility, we talked about how Many To Many better represents how people think about things.  Because of this, people have an instinctive bias towards Many To Many approaches.  However, because our physical surroundings are so hierarchical, we can rapidly run into problems.

Let’s take another look at my file cabinet, and that frustrating home escrow statement.  I want to put my statement in two different folders, and I just can’t.

To try and solve this, it’s been suggested to me a few times that I could make a copy of it, and store copies in both places.

By making copies, we have, in a sense, converted a Many To One relationship into a Many To Many relationship: a folder can contain Many files, but a file can also now be in Many folders.  Problem solved, right?

Well, what if I want to update one of the files?  Perhaps it is an invoice that I want to mark “PAID”.  If I pull out one of the invoices and mark it paid, it doesn’t automatically mark the other one.  It’s just a copy.  It’s not literally the same document in two places.  If I were to pull the other copy out later and think it wasn’t paid yet, I might double-pay.  This is of course a contrived example, but in the real world, it can lead to all sorts of larger problems.  Generally speaking, if you update one copy, you want the other copy to be updated as well, automatically.  Unfortunately, it’s impossible to do this in the physical world.

Another similar problem is if you want to remove the file.  You would have to remember every single location of the file and all its copies, to be sure it was fully “deleted” from your file cabinet.  Whereas if it were truly one file stored in multiple places, you would only have to delete it once.

These kinds of synchronization problems are common in the real world, and it’s because people are unwittingly trying to simulate Many To Many relationships, within a physical structure that doesn’t support it.  When we thought we were creating a Many To Many relationship in my file cabinet, we were actually just creating a new Many To One relationship – duplicating information and increasing maintenance costs.

As a result, people often abandon Many To Many as an organizational approach, concluding that it’s more hassle than it’s worth.  But in fact, the problem isn’t with Many To Many itself.  The problem comes from trying to graft it onto physical items.

As we discussed in Hierarchies Are Overrated, people are conditioned by the physical world and its hierarchical structures.  But because we are instinctively drawn towards Many-To-Many thinking, this can cause confusion.  We’re conflicted about how to represent our thoughts.  And since we can easily give up on Many To Many as an approach, a misunderstanding develops about Many To Many, leading to a lack of awareness about its benefits.

People tend to link concepts to other concepts in many directions.  I might see someone as both a friend and a co-worker.  Butter isn’t just a cooking ingredient; it’s also a bread spread.  A recent photograph of mine could be labeled both “holiday” and “family”.

These relationships are all impossible to represent in a strict hierarchy, and this is because – in the example of my friend/co-worker, while I do have many co-workers, my friend does not fill only one role in my life.

Note the words “many” and “one”.  Data relation experts talk about three main types of data relationships, and they are as follows:

One To One (1:1) is not particularly relevant to our discussions, but one example would be the relationship between a person and their social security number. A person has only one SSN (in theory), and an SSN has only one person (also in theory).

Many To One (N:1) is how to describe a hierarchical relationship.  Many files can be in a folder, but a file can only be in one folder.  The first example in Hierarchies Are Overrated also showed a hierarchy, but from a top-down perspective.

And finally…

Many To Many (N:N) shows how all elements can link to each other.  If the graphic were turned on its side, you could see that while each parent has multiple children, each child can also have multiple parents.  I can know many people that are “friends”, but someone who is a friend can also be a business partner, romantic partner, etc.

You can see that at least conceptually, Many To Many is a more appropriate model for how we organize thoughts in our heads.

I think it’s a given that we have a need to externalize the thoughts in our head – whether for todo lists, grocery lists, applying categories to things, etc.  Since we don’t organize our thoughts hierarchically, we often need an alternative to hierarchies.  Since Many to Many better represents how we think about things, it can be an effective alternative to hierarchies for how to externalize our thoughts.

Next, I’ll show some attempts at using Many To Many in the real world, and some of the common struggles it can cause.

Should I put it in my home insurance folder?  What if I later have an escrow question?  Will I remember where to look for it?  I have an escrow folder from when the escrow account was created, but the document seems pretty insurance-related.  I could combine it all into a home mortgage folder, but is that too general?

At that point, I tend to give up and put the document on an unsorted pile, in some corner of my office.

The basic problem is that I have a physical item – a paper statement – and even though I can think of multiple ways to categorize it, I can only put it in one folder.  I can group folders together into one category folder, but they can still only go into one category folder.

That’s basically what a hierarchy is.

In this sense of the word, hierarchy¹ means that an item can only have one container (or parent).  Containers can be nested, and many items can be in one container, but each item can only be in that one container.  One file cabinet; multiple folders.  One folder; multiple files.  Very structured, very orderly.

This is not how people think.  We do tend to think in terms of bundles of knowledge, but each item of knowledge we have might be contained in several bundles.  Our bits of knowledge and information link together in all kinds of directions.

Hierarchies, however, are common in the physical world.  A piece of paper can only be contained in one folder.  An article of clothing can only be put into one suitcase.  My heart is in only my body.  And this has conditioned us to try and apply hierarchical models onto other pieces of information.

There’s an old saying: “A place for everything, and everything in its place.”  It’s a comforting sentiment, isn’t it?  It makes me think of a matronly woman in a large house where everything is proper.  It’s kind of an archetypal fantasy, the Mary Poppins that knows just where everything goes and what everything should be.

Unfortunately, there’s a trick to the Mary Poppins illusion.  It is a result of a lot of very tedious work, and – more importantly – it assumes that every new thing that appears will fit comfortably with what is already there.

What really happens with knowledge?  It is learned or discovered.  It is analyzed, and synthesized.  It has to be reconciled with what came before, and sometimes it can completely change your existing understandings, busting apart your existing paradigm.

However, as a hierarchical structure grows, there is a growing cost to fitting new information into it.  As it gets larger, when a new element appears, it might require a new container or parent.  Sometimes, an entire branch might need to be moved or reorganized, and sometimes, the entire system has to be reworked.  It can get very unwieldy.

If you discover a new fact that doesn’t fit easily into the hierarchy, what do you do?  It is expensive and inefficient to reorder an entire hierarchy for every new piece of information.  And as more and more knowledge is discovered, it gets harder and harder (and slower and slower) to integrate.

What is the result?  People cope poorly.  They let the new knowledge pile up without synthesizing it, or they force it into the hierarchy ineffectively, or they simply stop gathering the new information.

And along the way, wisdom is lost.  People fall behind, sticking their heads in the sand, or they fail to realize implications of new knowledge.  Wisdom is lost, and evolution slows down.

What is to be learned from this?

Hierarchies have their place.  Hierarchies can be useful for bodies of information that are balanced, largely known and largely static, and when there are clear compositional units – parents and containers.  But hierarchies are a bad fit for any expanding or changing body of knowledge.  They’re not only a poor fit, but they can inhibit the growth of that knowledge.