8.10 Raw Counts and Weighted Counts

This section has two aims.

The first aim is to see if we can find a measure of how specialised a topic is. Intuitively, some topics are extremely specialised - only people working in quantum physics talk about Quantum Physics. But other topics cut across philosophy - you don’t have to be writing about arguments to talk about Arguments. Is there some way internal to the model to capture that notion?

The second is to cast some light on the relationship between the two ways of measuring topic size I’ve been using: raw count and weighted count. The raw count is the number of articles such that the probability of being in that topic is higher than the probability of being in any other topic. The weighted count is the sum, across all articles, of the probability of being in that topic. Mathematically, it is the expected number of articles in the topic.

I’ll start by looking at the relationship between the raw count and the weighted count for a single topic: Modality. And then I’ll turn to see what happens when the focus expands to the other 89 topics. Start with two variables:

  • \(r\), the raw count of articles in the topic. For Modality, this is 370. That is, there are 370 articles that the model thinks are more probably in Modality than in any other topic.
  • \(w\), the weighted count of articles in the topic. For Modality, this is 370.51. That is, across all the articles, the sum of the probabilities that they are in Modality is 370.51.

So these are fairly close, though as we’ll see, that’s not typical. (Indeed, I picked Modality to focus on because they were close, and I wanted to see how typical that was.) There is a third variable I’m going to spend a bit of time on. Focus on those 370 articles whose probablility of being in Modality is greatest. The model gives them very different probabilities of being in Modality. Here are two articles from the 370.

Table 8.20: Alexander Bird (2000) “Further Antidotes: A Response To Gundersen” The Philosophical Quarterly 50:229-233.
Subject Probability
Modality 0.8215
Causation 0.0840
Composition and Constitution 0.0554
Minds and Machines 0.0205
Table 8.21: David Keyt (1973) “The Philosophy Of C. I. Lewis” Philosophical Review 82:491-516.
Subject Probability
Modality 0.1373
Verification 0.1196
Value 0.0977
Idealism 0.0542
Analytic/Synthetic 0.0516
Universals and Particulars 0.0494
Chance 0.0469
Moral Conscience 0.0355
Definitions 0.0318
Time 0.0301
Perception 0.0290
Physicalism 0.0268
Promises and Imperatives 0.0247
Arguments 0.0240
Justification 0.0208
Dewey and Pragmatism 0.0206
Life and Value 0.0204

These were not picked at random. The Bird article is the one the model is most confident is in Modality, and the Keyt article is the one of the 370 that it is least confident about. (Honestly I think the model got this one wrong, and is confused by ‘Lewis’.) There is a range of probabilities between those though.

Among the 370 articles that make up the raw count for Modality, the average probability the model gives to them being in the topic is 0.3819917. (That’s a little under the midpoint between the Bird and the Keyt articles, but not absurdly so.) This is the value for Modality of the third variable I’m interested in:

  • \(p\), the average probability of being in the topic among articles that are more probably in that topic than any other.

Between \(r\), \(w\) and \(p\), there are three things that look like plausible specialisation measures.

Measure One
Ratio of raw count to weighted count, i.e., \(\frac{r}{w}\).
Measure Two
Average probability of being in the topic among articles that have maximal probability of being in the topic, i.e.,\(p\).
Measure Three
What proportion of the weighted count comes from articles that are ‘in’ the topic, i.e., \(\frac{rp}{w}\).

The first measure is intuitive because (as we’ll see) it places Arguments right at the bottom of the scale. And that’s intuitively our least specialised topic.

The second measure is intuitive because it measures how confident the model is in its placement for articles that get maximal probability of being in a topic. And specialised topics should, in general, do well on this. You won’t find the model thinking that this article is most probably a Quantum Physics article, but maybe jusy maybe it’s a Social Contract Theory article, and maybe it’s a Depiction article, so we should spread the probability around between those.

The third measure is intuitive for a similar reason. If a topic is specialised, it won’t pick up an extra 3% there or 5% there to its weighted count from articles in other topics. Most of \(w\) will come from articles ‘in’ the topic, i.e., from the part of \(w\) that \(rp\) measures.

The second and third are obviously related, since \(p = \frac{rp}{r}\). In some sense, they are just different ways of normalising \(rp\) to the size of the topic.

But intuitively all three should be related, since they all feel like measures of specialisation. It turns out this isn’t quite right.

Let’s start with measure one, the ratio of raw to weighted count. This is something I talked a bit about back in chapter 2 when discussing two ways of measuring topic size.

Ratio of raw count to weighted count by topic

Figure 8.14: Ratio of raw count to weighted count by topic

There is an enormous outlier here: Arguments. (Though Self-Consciousness and Concepts are also fairly low.) This makes sense - at least once the model decides that this will be a topic. There aren’t that many articles that are about arguments as such, but there are plenty of discussions of arguments in papers, so there are lots of ways to talk the model into thinking there’s a 2 or 3 percent chance that that’s the right topic for you.

To get a sense of how big the outliers are, it is useful to sort this graph by the ratio it represents.

Ratio of raw count to weighted count by topic (sorted)

Figure 8.15: Ratio of raw count to weighted count by topic (sorted)

I don’t quite understand why Beauty and Crime and Punishment are at the top of this graph. But I’ll keep a note of where Beauty appears in subsequent graphs, because it is an interesting case.

Now we’ll look at the second measure, i.e., \(p\). For Modality, the value of \(p\) is 0.3819917. How typical is that? Is the average maximal topic probability always around 0.4, or does it vary by topic? Let’s look at the data,, and note that the scale does not start at 0.

Average maximal probability by topic

Figure 8.16: Average maximal probability by topic

So it turns out Modality is reasonably typical, though there is a lot of variation here. Let’s look at the same graph but sorted by probability rather than topic number.

Average maximal probability by topic (sorted)

Figure 8.17: Average maximal probability by topic (sorted)

This does look like a measure of specialisation. If you’re talking about Quantum Physics or Evolutionary Biology, then you’re probably really talking about quantum physics or evolutionary biology. But lots of people use Ordinary Language and discuss Concepts. So lots of articles that are about lots of different topics can end up looking like Ordinary Language or Concepts articles.

Let’s turn to our last measure of specialisation, \(\frac{rp}{w}\). One way to think about how big that is, is to think about the average probability of being in Modality for the articles whose maximal probability is in one of the other 89 topics. Some of these probabilities are quite large, such as for this article.

Table 8.22: William Seager and Graeme Hunter (1982) “Reply To Forbes” Analysis 42:224-226.
Subject Probability
Radical Translation 0.4538
Modality 0.4407
Ontological Argument 0.0685

But most contributions aren’t that big. In fact, the average is under 1%. But there are nearly 32,000 of them, so they add up. In fact, they add up to about 61% of the weighted count for Modality. So the value of \(\frac{rp}{w}\) for Modality is about 39%, since that’s what is left once you take out the 61% that comes from the rest of the articles. Is that 39% typical? Well, we can again look at the graph for all 90 topics.

Proportion of weighted count that's from non-topic articles

Figure 8.18: Proportion of weighted count that’s from non-topic articles

And 39% is somewhat middling, though there’s so much variation that I’m not sure what I’d say is typical. Let’s look at that one arranged by order.

Proportion of weighted count that's from non-topic articles (sorted)

Figure 8.19: Proportion of weighted count that’s from non-topic articles (sorted)

And again, it looks like a measure of specialisation. The model doesn’t give Quantum Physics much credit for articles that aren’t squarely about quantum physics. But it gives Arguments lots of credit for articles that are not primarily about arguments.

If these three things are all sort of measures of specialisation, they should correlate reasonably well. But this turns out not quite to be right. The first and second, for example, are not particularly well correlated. Here is the graph of the two of them, i.e., \(\frac{r}{w}\) and \(p\), against each other.

Correlation between first and second specialisation measures

Figure 8.20: Correlation between first and second specialisation measures

There is a trend there, but it’s not a strong one. But the other two pairs are much more closely correlated. (We have here a real life example of how being closely correlated is not always transitive.) Here is the graph of the first against the third, i.e., \(\frac{r}{w}\) on one axis, and \(\frac{rp}{w}\) on the other.

Correlation between first and third specialisation measures

Figure 8.21: Correlation between first and third specialisation measures

Interestingly, although those two are correlated, part of what that means is that there are some cases that they both misclassify if construed as measures of specialisation. If we’re looking for a measure of specialisation, we don’t want Functions and Evolutionary Biology at opposite ends.

The last two measures, \(p\) and \(\frac{rp}{w}\) are also well correlated, though with some outliers.

Correlation between second and third specialisation measures

Figure 8.22: Correlation between second and third specialisation measures

And our two measures of specialisation do sort of line up. There are some outliers - Functions is above the line and Beauty is below it. .

I think that what we learn from that is that the best measure of specialisation is \(p\), the average probability of being in the topic among articles whose maximal probability is being in just that topic. The other measures are roughly correlated with \(p\), but where they differ, \(p\) seems to do a better job of measuring specialisation. And they are (somewhat surprisingly) very well correlated with each other.

And we also learned something about the relationship between \(r\) and \(w\). It’s really well correlated with \(\frac{rp}{w}\). And the best way to understand \(\frac{rp}{w}\) is to think about the average probability of being in a topic when that topic isn’t maximal. So \(r\) is low relative to \(w\) when a topic is often the 2nd, 3rd or 4th highest topic, and high when it is not. This isn’t surprising, but I had thought that this would in turn line up with how specialised a topic is. And that didn’t really do that. What did line up with intuitive specialisation is the average probability of being in a topic among those papers where that topic probability is maximal.