It has taken a long time to develop the math and topologies to answer questions like this. Research is when you search for an answer, get it wrong, or not entirely right, and re-search and re-search and re-search … I’ve made and corrected a lot of mistakes.
For guitars like the PRS (Paul Reed Smith) 513 and the Ernie Ball MusicMan “Game Changer”, with two humbuckers and a single-coil pickup, I’ve come up with different answers at different times. Sorry about that. At last look, the sum of possible humbucking pairs, triples, quads, and quints comes up to about 72 potentially unique humbucking tones. I had missed a few possible pickup circuits.
Those circuits, BTW, are covered by the June 7, 2017 U.S. Patent Application, or at least all those not already embodied in the market.
One should bear in mind that a number of those 72 possible tones, particularly for quad and quint circuits, will likely be so close together in tone as to not count. It’s theory, and theory must be tested. And if someone tells you they can get 250,000 tones from 5 coils, you might recall the phrase, “too good to by true”. Unless you maybe count positions on the tone pot.
Patent US 9401134 B2 counted tones by using separate capacitors on a 12-way switch, which created separate resonant peaks and roll-offs with the pickup combinations. And on the prototype, the largest capacitor or two killed the 1-string. Re-search, re-search, re-search …
For one thing, the two coils of a humbucker see and reproduce essentially the same signal. Not only are they separated by the distance of about the 16th to 32 harmonic, depending on which fret, 12 to 0, is fingered, they also share the same magnetic field. Their magnetic entanglement could justify the term “transformer”. You can’t really combine the N1 and S2 coils, the north pole up and south pole up coils from humbuckers 1 and 2 respectively, and expect the output to be different from the same combination (series or parallel) of the S1 and N2 coils.
Furthermore, unless pickups are made to specifications not typically employed, it’s not likely that a single-coil pickup will have the same response to external hum fields as either coil of a humbucker. So humbucking combinations of the single-coil pickup with the coils of the humbuckers may be flawed.
Recall that the tones in the Fender Marauder guitar, with four single-coil pickups and 81 switch combinations (parallel circuits only), contained 50% duplicates. Furthermore, only about 16 of the 80 which had an output could have been humbucking, and half of those were duplicates. So 8 possibly unique humbucking tones out of 80 outputs, and no map to where they were. It reportedly failed in the market because it was considered “noisy”. And possibly hard to understand.
Perhaps you can see a trend.