29

7

From the pandas documentation, I've gathered that unique-valued indices make certain operations efficient, and that non-unique indices are occasionally tolerated.

From the outside, it doesn't look like non-unique indices are taken advantage of in any way. For example, the following `ix`

query is slow enough that it seems to be scanning the entire dataframe

```
In [23]: import numpy as np
In [24]: import pandas as pd
In [25]: x = np.random.randint(0, 10**7, 10**7)
In [26]: df1 = pd.DataFrame({'x':x})
In [27]: df2 = df1.set_index('x', drop=False)
In [28]: %timeit df2.ix[0]
1 loops, best of 3: 402 ms per loop
In [29]: %timeit df1.ix[0]
10000 loops, best of 3: 123 us per loop
```

(I realize the two `ix`

queries don't return the same thing -- it's just an example that calls to `ix`

on a non-unique index appear much slower)

Is there any way to coax pandas into using faster lookup methods like binary search on non-unique and/or sorted indices?

1Highly recommended Answer! Appreciate it. – Neerav – 2014-09-03T20:22:53.690

1I don't understand the timings at the end. df3 should be faster? – lucid_dreamer – 2018-08-21T08:44:46.317

@lucid_dreamer I was confused too, but df1 uses the default index which goes from 0 to len(df1) - 1 and is unique, so df1.loc[] uses a hashtable. df2 sets the index to 'x' which is not unique and not sorted, so it does a linear scan, O(N). df3 is the same as df2 but sorted and still non-unique, so it does a binary search. – Max Taggart – 2018-10-17T21:26:37.083

So why is the linear scan of df2 faster? – lucid_dreamer – 2018-10-18T09:55:03.807

1@lucid_dreamer please notice the unit of time. – HYRY – 2018-10-19T01:26:44.167

omg I'm sleeping – lucid_dreamer – 2018-10-25T18:03:28.303